API recommendation is a promising approach which is widely used during software development. However, the evaluation of API recommendation is not explored with sufficient rigor. The current evaluation of API recommendation mainly focuses on correctness, the measurement is conducted by matching recommended results with ground-truth results. In most cases, there is only one set of ground-truth APIs for each recommendation attempt, but the object code can be implemented in dozens of ways. The neglect of code diversity results in a possible defect in the evaluation. To address the problem, we invite 15 developers to analyze the unmatched results in a user study. The online evaluation confirms that some unmatched APIs can also benefit to programming due to the functional correlation with ground-truth APIs. Then we measure the API functional correlation based on the relationships extracted from API knowledge graph, API method name, and API documentation. Furthermore, we propose an approach to improve the measurement of correctness based on API functional correlation. Our measurement is evaluated on a dataset of 6141 requirements and historical code fragments from related commits. The results show that 28.2% of unmatched APIs can contribute to correctness in our experiments.

API推荐是一种有前途的方法，在软件开发过程中被广泛使用。但是，API推荐的评估没有足够严格的探讨。当前对API推荐的评估主要集中在正确性上，通过将推荐结果与真实结果相匹配来进行测量。在大多数情况下，每次推荐尝试只有一组基本的API，但是目标代码可以通过多种方式实现。忽视代码多样性会导致评估中的缺陷。为了解决这个问题，我们邀请15个开发人员在用户研究中分析不匹配的结果。在线评估证实，由于与地面真实API的功能相关性，某些不匹配的API也可以从编程中受益。然后，我们根据从API知识图，API方法名称和API文档中提取的关系来测量API功能相关性。此外，我们提出了一种基于API功能相关性来改进正确性度量的方法。我们对6141个需求和相关提交的历史代码片段的数据集进行了评估。结果表明，在我们的实验中，28.2％的不匹配API可以有助于正确性。