We review the impact of deep-learning technologies on camera architecture. The function of a camera is first to capture visual information and second to form an image. Conventionally, both functions are implemented in physical optics. Throughout the digital age, however, joint design of physical sampling and electronic processing, e.g., computational imaging, has been increasingly applied to improve these functions. Over the past five years, deep learning has radically improved the capacity of computational imaging. Here we briefly review the development of artificial neural networks and their recent intersection with computational imaging. We then consider in more detail how deep learning impacts the primary strategies of computational photography: focal plane modulation, lens design, and robotic control. With focal plane modulation, we show that deep learning improves signal inference to enable faster hyperspectral, polarization, and video capture while reducing the power per pixel by 10−100×. With lens design, deep learning improves multiple aperture image fusion to enable task-specific array cameras. With control, deep learning enables dynamic scene-specific control that may ultimately enable cameras that capture the entire optical data cube (the “light field”), rather than just a focal slice. Finally, we discuss how these three strategies impact the physical camera design as we seek to balance physical compactness and simplicity, information capacity, computational complexity, and visual fidelity.

中文翻译：

---

智能相机

我们回顾了深度学习技术对相机架构的影响。相机的功能首先是捕捉视觉信息，其次是形成图像。传统上，这两种功能都在物理光学中实现。然而，在整个数字时代，物理采样和电子处理（例如计算成像）的联合设计已越来越多地应用于改进这些功能。在过去的五年中，深度学习从根本上提高了计算成像的能力。在这里，我们简要回顾了人工神经网络的发展及其最近与计算成像的交集。然后，我们更详细地考虑深度学习如何影响计算摄影的主要策略：焦平面调制、镜头设计和机器人控制。通过焦平面调制，我们展示了深度学习改进了信号推理，以实现更快的高光谱、偏振和视频捕获，同时将每个像素的功率降低 10-100 倍。通过镜头设计，深度学习改进了多光圈图像融合，以实现特定任务的阵列相机。通过控制，深度学习实现了动态场景特定控制，最终可能使相机能够捕获整个光学数据立方体（“光场”），而不仅仅是一个焦点切片。最后，我们讨论这三种策略如何影响物理相机设计，因为我们寻求平衡物理紧凑性和简单性、信息容量、计算复杂性和视觉保真度。深度学习改进了多孔径图像融合，以实现特定任务的阵列相机。通过控制，深度学习实现了动态场景特定控制，最终可能使相机能够捕获整个光学数据立方体（“光场”），而不仅仅是一个焦点切片。最后，我们讨论这三种策略如何影响物理相机设计，因为我们寻求平衡物理紧凑性和简单性、信息容量、计算复杂性和视觉保真度。深度学习改进了多孔径图像融合，以实现特定任务的阵列相机。通过控制，深度学习实现了动态场景特定控制，最终可能使相机能够捕获整个光学数据立方体（“光场”），而不仅仅是一个焦点切片。最后，我们讨论这三种策略如何影响物理相机设计，因为我们寻求平衡物理紧凑性和简单性、信息容量、计算复杂性和视觉保真度。