A key to collaborative decision making is to aggregate individual evaluations into a group decision. One of its fundamental challenges lies in the difficulty in identifying and dealing with irregular or unfair ratings and reducing their impact on group decisions. Little research has attempted to identify irregular ratings in a collaborative assessment task, let alone develop effective approaches to reduce their negative impact on the final group judgment. In this article, based on the synergy theory, we propose a novel consensus-based collaborative evaluation (CE) method called Collaborative Evaluation based on rating DIFFerence (CE-DIFF) for identifying irregular ratings and mitigating their impact on collaborative decisions. CE-DIFF determines and assigns different weights automatically to individual evaluators or ratings based on the level of consistency of one's ratings with the group assessment outcome through continuous iterations. We conducted two empirical experiments to evaluate the proposed method. The results show that CE-DIFF has higher accuracy in dealing with irregular ratings than existing CE methods, such as arithmetic mean and trimmed mean. In addition, the effectiveness of CE-DIFF is independent of group size. This study provides a new and more effective method for collaborative assessment, as well as novel theoretical insights and practical implications on how to improve collaborative assessment.

中文翻译：

---

CE-DIFF：一种在协作决策中识别和应对不规则评级的方法

协作决策的一个关键是将个人评估汇总到一个群体决策中。其基本挑战之一在于难以识别和处理不规则或不公平的评级并减少它们对集体决策的影响。很少有研究试图识别协作评估任务中的不规则评级，更不用说开发有效的方法来减少它们对最终群体判断的负面影响。在本文中，基于协同理论，我们提出了一种新的基于共识的协作评估 (CE) 方法，称为基于评级差异的协作评估 (CE-DIFF)，用于识别不规则评级并减轻其对协作决策的影响。CE-DIFF 通过连续迭代，根据个人评级与小组评估结果的一致性水平，自动确定并分配不同的权重给个人评估者或评级。我们进行了两个实证实验来评估所提出的方法。结果表明，CE-DIFF在处理不规则评分方面比现有的CE方法（例如算术平均和修剪平均）具有更高的准确性。此外，CE-DIFF 的有效性与组大小无关。本研究为协作评估提供了一种新的、更有效的方法，并为如何改进协作评估提供了新颖的理论见解和实践意义。我们进行了两个实证实验来评估所提出的方法。结果表明，CE-DIFF在处理不规则评分方面比现有的CE方法（例如算术平均和修剪平均）具有更高的准确性。此外，CE-DIFF 的有效性与组大小无关。本研究为协作评估提供了一种新的、更有效的方法，并为如何改进协作评估提供了新颖的理论见解和实践意义。我们进行了两个实证实验来评估所提出的方法。结果表明，CE-DIFF在处理不规则评分方面比现有的CE方法（例如算术平均和修剪平均）具有更高的准确性。此外，CE-DIFF 的有效性与组大小无关。本研究为协作评估提供了一种新的、更有效的方法，并为如何改进协作评估提供了新颖的理论见解和实践意义。