In this article, we introduce a learning-based vision dynamics approach to nonlinear model predictive control (NMPC) for autonomous vehicles, coined learning-based vision dynamics (LVD) NMPC. LVD-NMPC uses an a-priori process model and a learned vision dynamics model used to calculate the dynamics of the driving scene, the controlled system’s desired state trajectory, and the weighting gains of the quadratic cost function optimized by a constrained predictive controller. The vision system is defined as a deep neural network designed to estimate the dynamics of the image scene. The input is based on historic sequences of sensory observations and vehicle states, integrated by an augmented memory component. Deep Q-learning is used to train the deep network, which once trained can also be used to calculate the desired trajectory of the vehicle. We evaluate LVD-NMPC against a baseline dynamic window approach (DWA) path planning executed using standard NMPC and against the PilotNet neural network. Performance is measured in our simulation environment GridSim, on a real-world 1:8 scaled model car as well as on a real size autonomous test vehicle and the nuScenes computer vision dataset.

在本文中，我们介绍了一种基于学习的视觉动力学方法，用于自动驾驶汽车的非线性模型预测控制 (NMPC)，创造了基于学习的视觉动力学 (LVD) NMPC。LVD-NMPC 使用先验过程模型和学习的视觉动力学模型来计算驾驶场景的动力学、受控系统的期望状态轨迹以及由约束预测控制器优化的二次成本函数的加权增益。视觉系统被定义为一个深度神经网络，旨在估计图像场景的动态。输入基于感官观察和车辆状态的历史序列，由增强记忆组件集成。深度 Q-learning 用于训练深度网络，经过训练后还可以用于计算车辆的期望轨迹。我们根据使用标准 NMPC 和 PilotNet 神经网络执行的基线动态窗口方法 (DWA) 路径规划来评估 LVD-NMPC。性能是在我们的仿真环境 GridSim、真实世界的 1:8 比例模型汽车以及真实尺寸的自主测试车辆和 nuScenes 计算机视觉数据集上进行测量的。