Ontology alignment is vital in Semantic Web technologies with numerous applications in diverse disciplines. Due to diversity and abundance of ontology alignment systems, a proper evaluation can portray the evolution of ontology alignment and depicts the efficiency of a system for a particular domain. Evaluation can help system designers recognize the strength and shortcomings of their systems, and aid application developers to select a proper alignment system. This article presents a new evaluation and comparison methodology based on multiple performance metrics that accommodates experts’ preferences via a multi-criteria decision-making (MCDM) method, i.e., Bayesian best–worst method (BWM). First, the importance of a performance metric for a specific task/application is determined according to experts’ preferences. The alignment systems are then evaluated based on proposed expert-based collective performance (ECP) that takes into account multiple metrics as well as their calibrated importance. For comparison, the alignment systems are ranked based on a probabilistic scheme, where it includes the extent to which one alignment system is preferred over another. The proposed methodology is applied to six tracks from ontology alignment evaluation initiative (OAEI), where the importance of performance metrics are calibrated by designing a survey and eliciting the preferences of ontology alignment experts. Accordingly, the participating alignment systems in the OAEI 2018 are evaluated and ranked. While the proposed methodology is applied to six OAEI tracks to demonstrate its applicability, it can also be applied to any benchmark or application of ontology alignment.

本体对齐在语义Web技术中至关重要，在各种学科中都有大量应用程序。由于本体对齐系统的多样性和丰富性，适当的评估可以描述本体对齐的演变并描述系统在特定领域的效率。评估可以帮助系统设计人员识别其系统的优点和缺点，并帮助应用程序开发人员选择合适的对齐系统。本文提出了一种基于多种性能指标的新评估和比较方法，该方法通过多准则决策（MCDM）方法（即贝叶斯最佳-最差方法（BWM））适应专家的偏爱。首先，根据专家的喜好确定性能指标对于特定任务/应用程序的重要性。然后，基于建议的基于专家的集体绩效（ECP）对对齐系统进行评估，其中考虑了多个指标及其校准的重要性。为了进行比较，基于概率方案对对齐系统进行排序，其中包括一个对齐系统相对于另一个对齐系统的优先程度。所提出的方法论被应用到来自本体一致性评估倡议（OAEI）的六个轨道上，其中性能指标的重要性通过设计调查和引起本体一致性专家的喜好来进行校准。因此，对OAEI 2018中参与的对齐系统进行了评估和排名。虽然将拟议的方法应用于OAEI的六个轨道以证明其适用性，