Product category matching is an important task in digital marketplaces and e-commerce, helping to power better search and recommendations in an online context. While variants of the problem have received some attention in academia, there is no documented guidance on how to efficiently acquire annotations for evaluating multiple (current and future) models, many of which rely on modern machine learning techniques such as neural representation learning. In this paper, we motivate and formalize the problem of product category matching in e-commerce, and present a rigorously designed set of guidelines and methodology for acquiring annotations in a cost-effective and reliable manner. We also present a methodology for using the annotations to compare solutions of two or more product category matching methods, including comparing models both before and after annotation. Three widely used e-commerce product category taxonomies, and multiple metrics, are used to demonstrate the utility of our proposals.

产品类别匹配是数字市场和电子商务中的一项重要任务，有助于在在线环境中提供更好的搜索和推荐。虽然该问题的变体在学术界受到了一些关注，但没有关于如何有效获取注释以评估多个（当前和未来）模型的文档指导，其中许多模型依赖于现代机器学习技术，例如神经表示学习。在本文中，我们对电子商务中的产品类别匹配问题进行了激励和形式化，并提出了一套严格设计的指导方针和方法，以经济高效且可靠的方式获取注释。我们还提出了一种使用注释来比较两种或多种产品类别匹配方法的解决方案的方法，包括比较注释前后的模型。三个广泛使用的电子商务产品类别分类法和多个指标用于证明我们建议的实用性。