Nowadays, systems containing components based on machine learning (ML) methods are becoming more widespread. In order to ensure the intended behavior of a software system, there are standards that define necessary qualities of the system and its components (such as ISO/IEC 25010). Due to the different nature of ML, we have to re-interpret existing qualities for ML systems or add new ones (such as trustworthiness). We have to be very precise about which quality property is relevant for which entity of interest (such as completeness of training data or correctness of trained model), and how to objectively evaluate adherence to quality requirements. In this article, we present how to systematically construct quality models for ML systems based on an industrial use case. This quality model enables practitioners to specify and assess qualities for ML systems objectively. In addition to the overall construction process described, the main outcomes include a meta-model for specifying quality models for ML systems, reference elements regarding relevant views, entities, quality properties, and measures for ML systems based on existing research, an example instantiation of a quality model for a concrete industrial use case, and lessons learned from applying the construction process. We found that it is crucial to follow a systematic process in order to come up with measurable quality properties that can be evaluated in practice. In the future, we want to learn how the term quality differs between different types of ML systems and come up with reference quality models for evaluating qualities of ML systems.

如今，包含基于机器学习 (ML) 方法的组件的系统正变得越来越普遍。为了确保软件系统的预期行为，有一些标准定义了系统及其组件的必要质量（例如 ISO/IEC 25010）。由于 ML 的不同性质，我们必须重新解释 ML 系统的现有品质或添加新品质（例如可信度）。我们必须非常精确地确定哪个质量属性与哪个感兴趣的实体相关（例如训练数据的完整性或训练模型的正确性），以及如何客观地评估对质量要求的遵守情况。在本文中，我们将介绍如何基于工业用例系统地构建 ML 系统的质量模型。该质量模型使从业者能够客观地指定和评估 ML 系统的质量。除了描述的整体构建过程之外，主要结果还包括用于指定 ML 系统质量模型的元模型、有关相关视图、实体、质量属性的参考元素，以及基于现有研究的 ML 系统度量，一个示例实例化具体工业用例的质量模型，以及从应用构建过程中吸取的经验教训。我们发现遵循一个系统的过程对于提出可在实践中进行评估的可衡量的质量属性至关重要。将来，我们希望了解术语质量在不同类型的 ML 系统之间有何不同，并提出用于评估 ML 系统质量的参考质量模型。