Most modern commodity imaging systems we use directly for photography—or indirectly rely on for downstream applications—employ optical systems of multiple lenses that must balance deviations from perfect optics, manufacturing constraints, tolerances, cost, and footprint. Although optical designs often have complex interactions with downstream image processing or analysis tasks, today’s compound optics are designed in isolation from these interactions. Existing optical design tools aim to minimize optical aberrations, such as deviations from Gauss’ linear model of optics, instead of application-specific losses, precluding joint optimization with hardware image signal processing (ISP) and highly parameterized neural network processing. In this article, we propose an optimization method for compound optics that lifts these limitations. We optimize entire lens systems jointly with hardware and software image processing pipelines, downstream neural network processing, and application-specific end-to-end losses. To this end, we propose a learned, differentiable forward model for compound optics and an alternating proximal optimization method that handles function compositions with highly varying parameter dimensions for optics, hardware ISP, and neural nets. Our method integrates seamlessly atop existing optical design tools, such as Zemax . We can thus assess our method across many camera system designs and end-to-end applications. We validate our approach in an automotive camera optics setting—together with hardware ISP post processing and detection—outperforming classical optics designs for automotive object detection and traffic light state detection. For human viewing tasks, we optimize optics and processing pipelines for dynamic outdoor scenarios and dynamic low-light imaging. We outperform existing compartmentalized design or fine-tuning methods qualitatively and quantitatively, across all domain-specific applications tested.

中文翻译：

---

用于端到端相机设计的微分复合光学和处理流程优化

我们直接用于摄影或间接依赖于下游应用的大多数现代商品成像系统都采用多个镜头的光学系统，必须平衡与完美光学、制造限制、公差、成本和占地面积的偏差。尽管光学设计通常与下游图像处理或分析任务有复杂的相互作用，但今天的复合光学设计是与这些相互作用隔离开来的。现有的光学设计工具旨在最大程度地减少光学像差，例如偏离高斯光学线性模型，而不是针对特定应用的损耗，从而排除与硬件图像信号处理 (ISP) 和高度参数化的神经网络处理的联合优化。在本文中，我们提出了一种复合光学优化方法，可以消除这些限制。我们优化全部的镜头系统共同具有硬件和软件图像处理管道、下游神经网络处理和特定于应用程序的端到端损失。为此，我们提出了一个博学的、复合光学的可微前向模型和交替近端优化处理光学、硬件 ISP 和神经网络参数维度变化很大的函数组合的方法。我们的方法与现有的光学设计工具无缝集成，例如泽马克斯. 因此，我们可以在许多相机系统设计和端到端应用程序中评估我们的方法。我们在汽车摄像头光学设置中验证了我们的方法 - 连同硬件 ISP 后处理和检测 - 在汽车物体检测和交通灯状态检测方面优于经典光学设计。对于人类观看任务，我们针对动态户外场景和动态低光成像优化光学和处理管道。我们在定性和定量上优于现有的分区设计或微调方法，跨越全部测试特定领域的应用程序。