Quality assessment is a critical component in crowdsourcing-based software engineering (CBSE) as software products are developed by the crowd with unknown or varied skills and motivations. In this paper, we propose a novel metric called the project score to measure the performance of projects and the quality of products for competition-based software crowdsourcing development (CBSCD) activities. To the best of our knowledge, this is the first work to deal with the quality issue of CBSE in the perspective of projects instead of contests. In particular, we develop a hierarchical quality evaluation framework for CBSCD projects and come up with two metric aggregation models for project scores. The first model is a modified squale model that can locate the software modules of poor quality, and the second one is a clustering-based aggregation model, which takes different impacts of phases into account. To test the effectiveness of the proposed metrics, we conduct an empirical study on TopCoder, which is a famous CBSCD platform. Results show that the proposed project score is a strong indicator of the performance and product quality of CBSCD projects. We also find that the clustering-based aggregation model outperforms the Squale one by increasing the percentage of the performance evaluation criterion of aggregation models by an additional 29%. Our approach to quality assessment for CBSCD projects could potentially facilitate software managers to assess the overall quality of a crowdsourced project consisting of programming contests.

中文翻译：

---

基于竞争的软件众包中的质量评估

质量评估是基于众包的软件工程（CBSE）的重要组成部分，因为软件产品是由具有未知或不同技能和动机的人群开发的。在本文中，我们提出了一种新的度量标准，称为项目得分，以衡量基于竞争的软件众包开发（CBSCD）活动的项目绩效和产品质量。据我们所知，这是从项目而非竞赛的角度处理CBSE质量问题的第一项工作。特别是，我们为CBSCD项目开发了一个分级质量评估框架，并针对项目得分提出了两种度量汇总模型。第一个模型是修改后的squale模型，可以定位质量较差的软件模块，第二个模型是基于集群的聚合模型，它考虑了阶段的不同影响。为了测试所提出指标的有效性，我们对TopCoder（这是著名的CBSCD平台）进行了实证研究。结果表明，拟议项目得分是CBSCD项目绩效和产品质量的有力指标。我们还发现，通过将聚合模型的性能评估标准的百分比再增加29％，基于聚类的聚合模型的性能优于Squale。我们对CBSCD项目进行质量评估的方法可能会潜在地促进软件经理评估由编程竞赛组成的众包项目的整体质量。这是著名的CBSCD平台。结果表明，拟议项目得分是CBSCD项目绩效和产品质量的有力指标。我们还发现，通过将聚合模型的性能评估标准的百分比再增加29％，基于聚类的聚合模型的性能优于Squale。我们对CBSCD项目进行质量评估的方法可能会潜在地促进软件经理评估由编程竞赛组成的众包项目的整体质量。这是著名的CBSCD平台。结果表明，拟议的项目得分是CBSCD项目绩效和产品质量的有力指标。我们还发现，通过将聚合模型的性能评估标准的百分比再增加29％，基于聚类的聚合模型的性能优于Squale。我们对CBSCD项目进行质量评估的方法可能会潜在地促进软件经理评估由编程竞赛组成的众包项目的整体质量。我们还发现，通过将聚合模型的性能评估标准的百分比再增加29％，基于聚类的聚合模型的性能优于Squale。我们对CBSCD项目进行质量评估的方法可能会潜在地促进软件经理评估由编程竞赛组成的众包项目的整体质量。我们还发现，通过将聚合模型的性能评估标准的百分比再增加29％，基于聚类的聚合模型的性能优于Squale。我们对CBSCD项目进行质量评估的方法可能会潜在地促进软件经理评估由编程竞赛组成的众包项目的整体质量。