Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

In-sensor optoelectronic computing using electrostatically doped silicon

Nature Electronics volume 5, pages 519–525 (2022)Cite this article

Subjects

Abstract

Complementary metal–oxide–semiconductor (CMOS) image sensors allow machines to interact with the visual world. In these sensors, image capture in front-end silicon photodiode arrays is separated from back-end image processing. To reduce the energy cost associated with transferring data between the sensing and computing units, in-sensor computing approaches are being developed where images are processed within the photodiode arrays. However, such methods require electrostatically doped photodiodes where photocurrents can be electrically modulated or programmed, and this is challenging in current CMOS image sensors that use chemically doped silicon photodiodes. Here we report in-sensor computing using electrostatically doped silicon photodiodes. We fabricate thousands of dual-gate silicon p–i–n photodiodes, which can be integrated into CMOS image sensors, at the wafer scale. With a 3 × 3 network of the electrostatically doped photodiodes, we demonstrate in-sensor image processing using seven different convolutional filters electrically programmed into the photodiode network.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Electrostatically doped silicon p–i–n photodiode.
Fig. 2: Programmable photoresponse of the dual-gate silicon p–i–n photodiode.
Fig. 3: Wafer-scale array of the dual-gate silicon p–i–n photodiodes.
Fig. 4: A 3 × 3 photodiode network for analogue multiply–accumulate computation.
Fig. 5: In-sensor image processing using the 3 × 3 dual-gate p–i–n photodiode network.

Similar content being viewed by others

Data availability

Source data are provided with this paper. The data that support the other findings of this study are available from the corresponding authors upon reasonable request.

Code availability

Experimental code is available from the corresponding authors on reasonable request.

References

  1. Kwon, M. et al. A low-power 65/14 nm stacked CMOS image sensor. In 2020 IEEE International Symposium on Circuits and Systems (ISCAS) 1–4 (IEEE, 2020).

  2. Park, J. et al. 7.9 1/2.74-inch 32Mpixel-prototype CMOS image sensor with 0.64 μm unit pixels separated by full-depth deep-trench isolation. In 2021 IEEE International Solid-State Circuits Conference (ISSCC) Vol. 64, 122–124 (IEEE, 2021).

  3. Chai, Y. In-sensor computing for machine vision. Nature 579, 32–33 (2020).

    Article  Google Scholar 

  4. Zhou, F. & Chai, Y. Near-sensor and in-sensor computing. Nat. Electron. 3, 664–671 (2020).

    Article  Google Scholar 

  5. Sonka, M., Hlavac, V. & Boyle, R. Image Processing, Analysis and Machine Vision 2nd edn (Springer, 2014).

  6. Zhan, C., Duan, X., Xu, S., Song, Z. & Luo, M. An improved moving object detection algorithm based on frame difference and edge detection. In Fourth International Conference on Image and Graphics (ICIG 2007) 519–523 (IEEE, 2007).

  7. Hussin, R., Juhari, M. R., Kang, N. W., Ismail, R. C. & Kamarudin, A. Digital image processing techniques for object detection from complex background image. Procedia Eng. 41, 340–344 (2012).

    Article  Google Scholar 

  8. Beresnev, P. et al. Automated driving system based on roadway and traffic conditions monitoring. In Proc. 4th International Conference on Vehicle Technology and Intelligent Transport Systems 363–370 (VEHITS, 2018).

  9. Gollisch, T. & Meister, M. Eye smarter than scientists believed: neural computations in circuits of the retina. Neuron 65, 150–164 (2010).

    Article  Google Scholar 

  10. Mennel, L. et al. Ultrafast machine vision with 2D material neural network image sensors. Nature 579, 62–66 (2020).

    Article  Google Scholar 

  11. Wang, C. Y. et al. Gate-tunable van der Waals heterostructure for reconfigurable neural network vision sensor. Sci. Adv. 6, eaba6173 (2020).

    Article  Google Scholar 

  12. Baugher, B. W. H., Churchill, H. O. H., Yang, Y. & Jarillo-Herrero, P. Optoelectronic devices based on electrically tunable p–n diodes in a monolayer dichalcogenide. Nat. Nanotechnol. 9, 262–267 (2014).

    Article  Google Scholar 

  13. Pospischil, A., Furchi, M. M. & Mueller, T. Solar-energy conversion and light emission in an atomic monolayer p–n diode. Nat. Nanotechnol. 9, 257–261 (2014).

    Article  Google Scholar 

  14. Lee, C.-H. et al. Atomically thin p–n junctions with van der Waals heterointerfaces. Nat. Nanotechnol. 9, 676–681 (2014).

    Article  Google Scholar 

  15. Dennis, V. C. et al. 2022 roadmap on neuromorphic computing and engineering. Neuromorph. Comput. Eng. 2, 022501 (2021).

  16. Liao, F., Zhou, F. & Chai, Y. Neuromorphic vision sensors: principle, progress and perspectives. J. Semicond. 42, 013105 (2021).

    Article  Google Scholar 

  17. Sze, S. M., Li, Y. & Ng, K. K. Physics of Semiconductor Devices (John Wiley & Sons, 2021).

  18. Xu, K. Silicon electro-optic micro-modulator fabricated in standard CMOS technology as components for all silicon monolithic integrated optoelectronic systems. J. Micromech. Microeng. 31, 054001 (2021).

    Article  Google Scholar 

  19. Wu, K., Zhang, H., Chen, Y., Luo, Q. & Xu, K. All-silicon microdisplay using efficient hot-carrier electroluminescence in standard 0.18 µm CMOS technology. IEEE Electron Device Lett. 42, 541–544 (2021).

    Article  Google Scholar 

  20. Signal and Image Processing Institute (USC-SIPI) Image Database (Univ. Southern California, accessed 22 April 2021); https://sipi.usc.edu/database/

  21. Yiyang, Z. The design of glass crack detection system based on image preprocessing technology. In 2014 IEEE 7th Joint International Information Technology and Artificial Intelligence Conference 39–42 (IEEE, 2014).

  22. Kanopoulos, N., Vasanthavada, N. & Baker, R. L. Design of an image edge detection filter using the Sobel operator. IEEE J. Solid-State Circuits 23, 358–367 (1988).

    Article  Google Scholar 

  23. Gedraite, E. S. & Hadad, M. Investigation on the effect of a Gaussian blur in image filtering and segmentation. In Proc. ELMAR-2011 393–396 (IEEE, 2011).

  24. Jang, H. et al. An atomically thin optoelectronic machine vision processor. Adv. Mater. 32, e2002431 (2020).

    Article  Google Scholar 

  25. Li, C. et al. Analogue signal and image processing with large memristor crossbars. Nat. Electron. 1, 52–59 (2018).

    Article  Google Scholar 

  26. Anila, S. & Devarajan, N. Preprocessing technique for face recognition applications under varying illumination conditions. Glob. J. Comput. Sci. Technol. XII, 12–18 (2012).

    Google Scholar 

  27. Zidan, M. A., Strachan, J. P. & Lu, W. D. The future of electronics based on memristive systems. Nat. Electron. 1, 22–29 (2018).

    Article  Google Scholar 

  28. Wang, Z. et al. Resistive switching materials for information processing. Nat. Rev. Mater. 5, 173–195 (2020).

    Article  Google Scholar 

  29. Ham, D., Park, H., Hwang, S. & Kim, K. Neuromorphic electronics based on copying and pasting the brain. Nat. Electron. 4, 635–644 (2021).

    Article  Google Scholar 

  30. Sun, L. et al. In-sensor reservoir computing for language learning via two-dimensional memristors. Sci. Adv. 7, eabg1455 (2021).

    Article  Google Scholar 

  31. Zhou, F. et al. Optoelectronic resistive random access memory for neuromorphic vision sensors. Nat. Nanotechnol. 14, 776–782 (2019).

    Article  Google Scholar 

  32. Liao, F. et al. Bioinspired in-sensor visual adaptation for accurate perception. Nat. Electron. 5, 84–91 (2022).

    Article  Google Scholar 

Download references

Acknowledgements

We acknowledge support by the Samsung Advanced Institute of Technology (SAIT) under Contract A30216 and by the National Science Foundation (NSF) Science and Technology Center for Integrated Quantum Materials under Contract DMR-1231319. Device fabrication was performed in part at the Harvard Center for Nanoscale Systems (CNS), which is supported by the NSF under Contract 1541959.

Author information

Author notes

  1. These authors contributed equally: Houk Jang, Henry Hinton.

Authors and Affiliations

  1. John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA

    Houk Jang, Henry Hinton, Woo-Bin Jung & Donhee Ham

  2. Center for Functional Nanomaterials, Brookhaven National Laboratory, Upton, NY, USA

    Houk Jang

  3. Samsung Advanced Institute of Technology, Samsung Electronics, Suwon, Republic of Korea

    Min-Hyun Lee, Changhyun Kim & Seongjun Park

  4. Institute of Advanced Composite Materials, Korea Institute of Science and Technology (KIST), Wanju-gun, Republic of Korea

    Min Park

  5. School of Materials Science and Engineering, Pusan National University, Busan, Republic of Korea

    Seoung-Ki Lee

Authors
  1. Houk Jang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Henry Hinton
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Woo-Bin Jung
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Min-Hyun Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Changhyun Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Min Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Seoung-Ki Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Seongjun Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Donhee Ham
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

H.J., H.H., S.P. and D.H. conceived and designed the experiments. H.J., M.P. and S.-K.L. designed and fabricated the electrostatically doped silicon photodiodes. H.H. designed the interface electronics. H.J., H.H., M.P. and S.-K.L. performed the measurements of individual dual-gate p–i–n photodiodes. H.J. and H.H. performed the in-sensor image processing. M.-H.L. and C.K. performed the wafer-scale measurements. W.-B.J. performed the COMSOL Multiphysics simulations on the dual-gate p–i–n photodiode. H.J., H.H., W.-B.J., M.-H.L., C.K., M.P., S.-K.L. and D.H. analysed the data. S.P. and D.H. supervised the project. H.J., H.H. and D.H. wrote the article. All the authors discussed the results and implications, and reviewed the article.

Corresponding authors

Correspondence to Seongjun Park or Donhee Ham.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Electronics thanks Yang Chai and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Figs. 1–10.

Source data

Source Data Fig. 1

Source data for Fig. 1b,c.

Source Data Fig. 2

Source data for Fig. 2b–d.

Source Data Fig. 3

Source data for Fig. 3b–d and raw data for Fig. 3d.

Source Data Fig. 4

Source data for Fig. 4b–d.

Source Data Fig. 5

Source data for Fig. 5a–d.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jang, H., Hinton, H., Jung, WB. et al. In-sensor optoelectronic computing using electrostatically doped silicon. Nat Electron 5, 519–525 (2022). https://doi.org/10.1038/s41928-022-00819-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41928-022-00819-6

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing