Skip to main content

Advertisement

SpringerLink
Log in

Bio-inspired smart vision sensor: toward a reconfigurable hardware modeling of the hierarchical processing in the brain

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Biological vision systems inspire processing methods in computer vision applications. This paper employs the insights of vision systems in hardware and presents a pixel-parallel, reconfigurable, and layer-based hierarchical architecture for smart image sensors. The architecture aims to bring computation close to the sensor to achieve high acceleration for different machine vision applications while consuming low power. We logically divide the image into multiple regions and perform pixel-level and region-level processing after removing spatiotemporal redundancy. Those processors use bio-inspired algorithms to activate the regions with region of interest of a scene. The hierarchical processing breaks the traditional sequential image processing and introduces parallelism for machine vision applications. Also, we make the hardware design reconfigurable even after fabrication to make the hardware reusable for different applications. Simulation results show that the area overhead and power penalty for adding reconfigurable features stay in an acceptable range. We emphasize to maximize the operating speed and obtain 800 MHz. Besides, the design saves 84.01% and 96.91% dynamic power at the first and second stages of the hierarchy by removing redundant information. Furthermore, the sequential deployment of high-level reasoning only on the selected regions of the image becomes computationally inexpensive to execute a complex task in real time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Kartheek Medathati, N.V., Neumann, H., Masson, G.S., Kornprobst, P.: Bio-inspired computer vision: towards a synergistic approach of artificial and biological vision. Comput. Vis. Image Underst. 150, 1–30 (2016)

    Google Scholar 

  2. Stroble, J.K., Stone, R.B., Watkins, S.E.: An overview of biomimetic sensor technology. Sens. Rev. 29(2), 112–119 (2009)

    Google Scholar 

  3. Koelling, T., Wang, J.: Advances in cmos image sensors. Secur. Deal. Integr. 30(10), 70–70,72,74, 10 (2008)

    Google Scholar 

  4. Zhu, H., Shibata, T.: A real-time motion-feature-extraction vlsi employing digital-pixel-sensor-based parallel architecture. IEEE Trans. Circuits Syst. Video Technol. 24(10), 1787–1799 (2014)

    Google Scholar 

  5. Tyrrell, B., Anderson, K., Baker, J., Berger, R., Brown, M., Colonero, C., Costa, J., Holford, B., Kelly, M., Ringdahl, E., Schultz, K., Wey, J.: Time delay integration and in-pixel spatiotemporal filtering using a nanoscale digital cmos focal plane readout. IEEE Trans. Electron Dev. 56(11), 2516–2523 (2009)

    Google Scholar 

  6. Gouveia, L.C.P., Choubey, B.: Advances on cmos image sensors. Sens. Rev. 36(3), 231–239 (2016)

    Google Scholar 

  7. Sakakibara, M., Ogawa, K., Sakai, S., Tochigi, Y., Honda, K., Kikuchi, H., Wada, T., Kamikubo, Y., Miura, T., Nakamizo, M., et al.: A 6.9-µm pixel-pitch back-illuminated global shutter cmos image sensor with pixel-parallel 14-bit subthreshold adc. IEEE J. Solid State Circuits 53(99), 1–9 (2018)

    Google Scholar 

  8. Delbrück, T., Linares-Barranco, B., Culurciello, E., Posch, C.: Activity-driven, event-based vision sensors. In: Proceedings of 2010 IEEE International Symposium on Circuits and Systems, pp 2426–2429. IEEE (2010)

  9. Atick, J.J., Norman Redlich, A.: Towards a theory of early visual processing. Neural Comput. 2(3), 308–320 (1990)

    Google Scholar 

  10. Bell, A.J., Sejnowski, T.J.: An information-maximization approach to blind separation and blind deconvolution. Neural Comput. 7(6), 1129–1159 (1995)

    Google Scholar 

  11. Foldiak, P.: Sparse coding in the primate cortex. The handbook of brain theory and neural networks (2003)

  12. Huang, Y., Rao, R.P.N.: Predictive coding. Wiley Interdiscip. Rev. Cognit. Sci. 2(5), 580–593 (2011)

    Google Scholar 

  13. Bhowmik, P., Pantho, M.J.H., Asadinia, M., Bobda, C.: Design of a reconfigurable 3d pixel-parallel neuromorphic architecture for smart image sensor. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 673–681 (2018)

  14. Nassi, J.J., Callaway, E.M.: Parallel processing strategies of the primate visual system. Nat. Rev. Neurosci. 10(5), 360 (2009)

    Google Scholar 

  15. Bhowmik, P., Pantho, M.J.H., Bobda, C.: Visual cortex inspired pixel-level re-configurable processors for smart image sensors. In: Proceedings of the 56th Annual Design Automation Conference 2019. ACM (2019)

  16. Pantho, M.J.H., Bhowmik, P., Bobda, C.: Pixel-parallel architecture for neuromorphic smart image sensor with visual attention. In: 2018 IEEE Computer Society Annual Symposium on VLSI (ISVLSI), pp. 245–250. IEEE (2018)

  17. Vahid, F., Givargis, T.D.: Embedded System Design: A Unified Hardware/Software Introduction, Chapter 1. Wiley, New York (2002)

    MATH  Google Scholar 

  18. Van Essen, D.C., Anderson, C.H., Felleman, D.J.: Information processing in the primate visual system: an integrated systems perspective. Science 255(5043), 419–423 (1992)

    Google Scholar 

  19. Rodieck, R.W., Rodieck, R.W.: The First Steps in Seeing, vol. 1. Sinauer Associates Sunderland, Sunderland (1998)

    Google Scholar 

  20. Kastner, D.B., Baccus, S.A.: Insights from the retina into the diverse and general computations of adaptation, detection, and prediction. Curr. Opin. Neurobiol. 25, 63–69 (2014)

    Google Scholar 

  21. Rieke, F., Rudd, M.E.: The challenges natural images pose for visual adaptation. Neuron 64(5), 605–616 (2009)

    Google Scholar 

  22. Koch, G., Oliveri, M., Caltagirone, C.: Neural networks engaged in milliseconds and seconds time processing: evidence from transcranial magnetic stimulation and patients with cortical or subcortical dysfunction. Philos. Trans. R. Soc. B Biol. Sci. 364(1525), 1907–1918 (2009)

    Google Scholar 

  23. Larkum, M.: A cellular mechanism for cortical associations: an organizing principle for the cerebral cortex. Trends Neurosci. 36(3), 141–151 (2013)

    Google Scholar 

  24. Motter, B.C.: Focal attention produces spatially selective processing in visual cortical areas v1, v2, and v4 in the presence of competing stimuli. J. Neurophysiol. 70(3), 909–919 (1993)

    Google Scholar 

  25. Michalareas, G., Vezoli, J., Van Pelt, S., Schoffelen, J.-M., Kennedy, H., Fries, P.: Alpha-beta and gamma rhythms subserve feedback and feedforward influences among human visual cortical areas. Neuron 89(2), 384–397 (2016)

    Google Scholar 

  26. Milner, P.M.: A model for visual shape recognition. Psychol. Rev. 81(6), 521 (1974)

    Google Scholar 

  27. Ahissar, M., Hochstein, S.: The reverse hierarchy theory of visual perceptual learning. Trends Cognit. Sci. 8(10), 457–464 (2004)

    Google Scholar 

  28. Gur, M.: Space reconstruction by primary visual cortex activity: a parallel, non-computational mechanism of object representation. Trends Neurosci. 38(4), 207–216 (2015)

    Google Scholar 

  29. Hochstein, S., Ahissar, M.: View from the top: hierarchies and reverse hierarchies in the visual system. Neuron 36(5), 791–804 (2002)

    Google Scholar 

  30. Lee, T.S., Mumford, D.: Hierarchical bayesian inference in the visual cortex. JOSA A 20(7), 1434–1448 (2003)

    Google Scholar 

  31. Reynolds, J.H., Pasternak, T., Desimone, R.: Attention increases sensitivity of v4 neurons. Neuron 26(3), 703–714 (2000)

    Google Scholar 

  32. Bylinskii, Z., DeGennaro, E.M., Rajalingham, R., Ruda, H., Zhang, J., Tsotsos, J.K.: Towards the quantitative evaluation of visual attention models. Vis. Res. 116, 258–268 (2015)

    Google Scholar 

  33. Itti, L., Koch, C.: Computational modelling of visual attention. Nat. Rev. Neurosci. 2(3), 194 (2001)

    Google Scholar 

  34. Tsotsos, J.K., Eckstein, M.P., Landy, M.S.: Computational models of visual attention. Vis. Res. 116(Pt B), 93 (2015)

    Google Scholar 

  35. Leon-Salas, W.D., Balkir, S., Sayood, K., Schemm, N., Hoffman, M.W.: A cmos imager with focal plane compression using predictive coding. IEEE J. Solid State Circuits 42(11), 2555–2572 (2007)

    Google Scholar 

  36. Miao, W., Lin, Q., Nanjian, W.: A novel vision chip for high-speed target tracking. Jpn. J. Appl. Phys. 46(4S), 2220 (2007)

    Google Scholar 

  37. Moini, A., Bouzerdoum, A., Eshraghian, K., Yakovleff, A., Nguyen, X.T., Blanksby, A., Beare, R., Abbott, D., Bogner, R.E.: An insect vision-based motion detection chip. IEEE J. Solid State Circuits 32(2), 279–284 (1997)

    Google Scholar 

  38. Oike, Y., Ikeda, M., Asada, K.: A 375/spl times/365 high-speed 3-d range-finding image sensor using row-parallel search architecture and multisampling technique. IEEE J Solid State Circuits 40(2), 444–453 (2005)

    Google Scholar 

  39. Zhang, W., Qiuyu, F., Nan-Jian, W.: A programmable vision chip based on multiple levels of parallel processors. IEEE J. Solid State Circuits 46(9), 2132–2147 (2011)

    Google Scholar 

  40. Culurciello, E., Etienne-Cummings, R., Boahen, K.A.: A biomorphic digital image sensor. IEEE J. Solid State Circuits 38(2), 281–294 (2003)

    Google Scholar 

  41. Shoushun, C., Bermak, A.: Arbitrated time-to-first spike cmos image sensor with on-chip histogram equalization. IEEE Trans. Very Large Scale Integr. (VLSI) Syst. 15(3), 346–357 (2007)

    Google Scholar 

  42. Serrano-Gotarredona, T., Linares-Barranco, B.: A 128x128 1.5% contrast sensitivity 0.9% fpn 3 us latency 4 mw asynchronous frame-free dynamic vision sensor using transimpedance preamplifiers. IEEE J. Solid State Circuits 48(3), 827–838 (2013)

    Google Scholar 

  43. Delbruck, T.: Frame-free dynamic digital vision. In: Proceedings of International Symposium on Secure-Life Electronics, Advanced Electronics for Quality Life and Society, pp. 21–26 (2008)

  44. Possa, P., Harb, N., Dokladalova, E., Valderrama, C.: P2ip: a novel low-latency programmable pipeline image processor. Microprocess. Microsyst. 39(7), 529–540 (2015)

    Google Scholar 

  45. Serrano-Gotarredona, R., Serrano-Gotarredona, T., Acosta-Jiménez, A., Serrano-Gotarredona, C., Pérez-Carrasco, J.A., Linares-Barranco, B., Linares-Barranco, A., Jiménez-Moreno, G., Civit-Ballcels, A.: On real-time aer 2-d convolutions hardware for neuromorphic spike-based cortical processing. IEEE Trans. Neural Netw. 19(7), 1196–1219 (2008)

    Google Scholar 

  46. Camunas-Mesa, L., Acosta-Jimenez, A., Zamarreño-Ramos, C., Serrano-Gotarredona, T., Linares-Barranco, B.: A 32x32 pixel convolution processor chip for address event vision sensors with 155 ns event latency and 20 meps throughput. IEEE Trans. Circuits Syst. I Regul. Pap. 58(4), 777–790 (2010)

    Google Scholar 

  47. Rao, R.P.N., Ballard, D.H.: Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat. Neurosci. 2(1), 79 (1999)

    Google Scholar 

  48. He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(9), 1904–1916 (2015)

    Google Scholar 

  49. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Google Scholar 

Download references

Acknowledgements

We acknowledge Sakakibara et al. [7] for the excellent work of pixel parallel image sensor. We are thankful to the National Science Foundation (NSF) for supporting us by Grant-1618606.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, University of Florida, Gainesville, USA

    Pankaj Bhowmik, Md Jubaer Hossain Pantho & Christophe Bobda

Authors
  1. Pankaj Bhowmik
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Md Jubaer Hossain Pantho
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christophe Bobda
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pankaj Bhowmik.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bhowmik, P., Pantho, M.J.H. & Bobda, C. Bio-inspired smart vision sensor: toward a reconfigurable hardware modeling of the hierarchical processing in the brain. J Real-Time Image Proc 18, 157–174 (2021). https://doi.org/10.1007/s11554-020-00960-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-020-00960-5

Keywords

Search

Navigation