In this paper, we do automatic correctness assessment for patches generated by program repair systems. We consider the human-written patch as ground truth oracle and randomly generate tests based on it, a technique proposed by Shamshiri et al., called Random testing with Ground Truth (RGT) in this paper. We build a curated dataset of 638 patches for Defects4J generated by 14 state-of-the-art repair systems, we evaluate automated patch assessment on this dataset. The results of this study are novel and significant: First, we improve the state of the art performance of automatic patch assessment with RGT by 190% by improving the oracle; Second, we show that RGT is reliable enough to help scientists to do overfitting analysis when they evaluate program repair systems; Third, we improve the external validity of the program repair knowledge with the largest study ever.

在本文中，我们对程序修复系统生成的补丁进行了自动正确性评估。我们将人工编写的补丁程序视为地面事实预告片，并基于该事实随机生成测试，这是Shamshiri等人提出的一种技术，在本文中称为“具有地面真实性的随机测试（RGT）”。我们建立了由14个最新维修系统生成的Defects4J 638个补丁的精选数据集，我们对该数据集进行了自动补丁评估。这项研究的结果是新颖而有意义的：首先，我们通过改进Oracle，将RGT的自动补丁评估的先进性能提高了190％。其次，我们证明RGT足够可靠，可以帮助科学家在评估程序修复系统时进行过度拟合分析。第三，