Fatal accidents are a major issue hindering the wide acceptance of safety-critical systems that employ machine learning and deep learning models, such as automated driving vehicles. In order to use machine learning in a safety-critical system, it is necessary to demonstrate the safety and security of the system through engineering processes. However, thus far, no such widely accepted engineering concepts or frameworks have been established for these systems. The key to using a machine learning model in a deductively engineered system is decomposing the data-driven training of machine learning models into requirement, design, and verification, particularly for machine learning models used in safety-critical systems. Simultaneously, open problems and relevant technical fields are not organized in a manner that enables researchers to select a theme and work on it. In this study, we identify, classify, and explore the open problems in engineering (safety-critical) machine learning systems—that is, in terms of requirement, design, and verification of machine learning models and systems—as well as discuss related works and research directions, using automated driving vehicles as an example. Our results show that machine learning models are characterized by a lack of requirements specification, lack of design specification, lack of interpretability, and lack of robustness. We also perform a gap analysis on a conventional system quality standard SQuaRE with the characteristics of machine learning models to study quality models for machine learning systems. We find that a lack of requirements specification and lack of robustness have the greatest impact on conventional quality models.

中文翻译：

---

机器学习系统中的工程问题

致命事故是阻碍采用机器学习和深度学习模型的安全关键系统（例如自动驾驶汽车）被广泛接受的主要问题。为了在安全关键系统中使用机器学习，有必要通过工程过程来证明系统的安全性和安全性。然而，到目前为止，还没有为这些系统建立这样被广泛接受的工程概念或框架。在演绎工程系统中使用机器学习模型的关键是将机器学习模型的数据驱动训练分解为需求、设计和验证，特别是对于安全关键系统中使用的机器学习模型。同时地，开放性问题和相关技术领域的组织方式并未使研究人员能够选择主题并进行研究。在这项研究中，我们识别、分类和探索工程（安全关键）机器学习系统中的开放问题——即机器学习模型和系统的需求、设计和验证——并讨论相关工作和研究方向，以自动驾驶汽车为例。我们的结果表明，机器学习模型的特点是缺乏需求规范、缺乏设计规范、缺乏可解释性和缺乏鲁棒性。我们还对具有机器学习模型特征的常规系统质量标准 SQuaRE 进行了差距分析，以研究机器学习系统的质量模型。