In many science and engineering fields that require computational models to predict certain physical quantities, we are often faced with the selection of the best model under the constraint that only a small sample set can be physically measured. One such example is the prediction of human perception of visual quality, where sample images live in a high dimensional space with enormous content variations. We propose a new methodology for model comparison named group maximum differentiation (gMAD) competition. Given multiple computational models, gMAD maximizes the chances of falsifying a “defender” model using the rest models as “attackers”. It exploits the sample space to find sample pairs that maximally differentiate the attackers while holding the defender fixed. Based on the results of the attacking-defending game, we introduce two measures, aggressiveness and resistance , to summarize the performance of each model at attacking other models and defending attacks from other models, respectively. We demonstrate the gMAD competition using three examples—image quality, image aesthetics, and streaming video quality-of-experience. Although these examples focus on visually discriminable quantities, the gMAD methodology can be extended to many other fields, and is especially useful when the sample space is large, the physical measurement is expensive and the cost of computational prediction is low.

中文翻译：

---

小组最大差异化竞争：使用少量样本进行模型比较。

在许多需要计算模型来预测某些物理量的科学和工程领域中，我们常常面临只能以物理方式测量少量样本的最佳模型的选择。一个这样的例子是对人类视觉质量感知的预测，其中样本图像生活在具有巨大内容变化的高维空间中。我们提出了一种新的模型比较方法，称为群体最大分化（gMAD）竞争。给定多个计算模型，gMAD将其余模型用作“攻击者”，从而最大程度地篡改“防御者”模型。它利用样本空间来查找样本对，从而在使防御者固定的同时最大程度地区分攻击者。根据保卫游戏的结果，我们引入了两种措施：侵略性 和 反抗 ，以总结每种模型分别攻击其他模型和防御其他模型的性能。我们使用三个示例展示了gMAD竞赛-图像质量，图像美学和流媒体视频体验质量。尽管这些示例着眼于视觉上可辨别的数量，但gMAD方法可以扩展到许多其他领域，并且在样本空间很大，物理测量昂贵且计算预测的成本较低的情况下特别有用。