Unsupervised machine learning is the training of an artificial intelligence system using information that is neither classified nor labeled, with a view to modeling the underlying structure or distribution in a dataset. Since unsupervised machine learning systems are widely used in many real-world applications, assessing the appropriateness of these systems and validating their implementations with respect to individual users’ requirements and specific application scenarios/contexts are indisputably two important tasks. Such assessments and validation tasks, however, are fairly challenging due to the absence of a priori knowledge of the data. In view of this challenge, in this article, we develop a METamorphic Testing approach to assessing and validating unsupervised machine LEarning systems, abbreviated as mettle. Our approach provides a new way to unveil the (possibly latent) characteristics of various machine learning systems, by explicitly considering the specific expectations and requirements of these systems from individual users’ perspectives. To support mettle, we have further formulated 11 generic metamorphic relations (MRs), covering users’ generally expected characteristics that should be possessed by machine learning systems. We have performed an experiment and a user evaluation study to evaluate the viability and effectiveness of mettle. Our experiment and user evaluation study have shown that, guided by user-defined MR-based adequacy criteria, end users are able to assess, validate, and select appropriate clustering systems in accordance with their own specific needs. Our investigation has also yielded insightful understanding and interpretation of the behavior of the machine learning systems from an end-user software engineering's perspective, rather than a designer's or implementor's perspective, who normally adopts a theoretical approach.

中文翻译：

---

METTLE：一种用于评估和验证无监督机器学习系统的变形测试方法

无监督机器学习是使用既未分类也未标记的信息训练人工智能系统，以对数据集中的底层结构或分布进行建模。由于无监督机器学习系统广泛用于许多现实世界的应用程序，因此评估这些系统的适用性并根据个人用户的要求和特定的应用场景/上下文验证它们的实现无疑是两项重要任务。然而，由于缺乏数据的先验知识，此类评估和验证任务相当具有挑战性。鉴于这一挑战，在本文中，我们开发了一种 Metamorphic 测试方法来评估和验证无监督机器学习系统，缩写为 mettle。我们的方法通过从个人用户的角度明确考虑这些系统的特定期望和要求，提供了一种揭示各种机器学习系统（可能潜在的）特征的新方法。为了支持勇气，我们进一步制定了 11 个通用变形关系 (MR)，涵盖了机器学习系统应具备的用户普遍预期的特征。我们进行了一项实验和一项用户评估研究，以评估 Mettle 的可行性和有效性。我们的实验和用户评估研究表明，在用户定义的基于 MR 的充分性标准的指导下，最终用户能够根据自己的特定需求评估、验证和选择合适的聚类系统。