Software engineers get questions of “how much testing is enough” on a regular basis. Existing approaches in software testing management employ experience-, risk-, or value-based analysis to prioritize and manage testing processes. However, very few is applicable to the emerging crowdtesting paradigm to cope with extremely limited information and control over unknown, online crowdworkers. In practice, deciding when to close a crowdtesting task is largely done by experience-based guesswork and frequently results in ineffective crowdtesting. More specifically, it is found that an average of 32% testing cost was wasteful spending in current crowdtesting practice. This article intends to address this challenge by introducing automated decision support for monitoring and determining appropriate time to close crowdtesting tasks. To that end, it first investigates the necessity and feasibility of close prediction of crowdtesting tasks based on an industrial dataset. Next, it proposes a close prediction approach named iSENSE2.0, which applies incremental sampling technique to process crowdtesting reports arriving in chronological order and organizes them into fixed-sized groups as dynamic inputs. Then, a duplicate tagger analyzes the duplicate status of received crowd reports, and a CRC-based (Capture-ReCapture) close estimator generates the close decision based on the dynamic bug arrival status. In addition, a coverage-based sanity checker is designed to reinforce the stability and performance of close prediction. Finally, the evaluation of iSENSE2.0 is conducted on 56,920 reports of 306 crowdtesting tasks from one of the largest crowdtesting platforms. The results show that a median of 100% bugs can be detected with 30% saved cost. The performance of iSENSE2.0 does not demonstrate significant difference with the state-of-the-art approach iSENSE , while the later one relies on the duplicate tag, which is generally considered as time-consuming and tedious to obtain.

中文翻译：

---

iSENSE2.0

软件工程师会定期收到“多少测试才足够”的问题。软件测试管理中的现有方法采用基于经验、风险或价值的分析来确定和管理测试过程的优先级。然而，很少有人适用于新兴的众测范式，以应对极其有限的信息和对未知的在线众筹工作者的控制。在实践中，决定何时结束众测任务主要是通过基于经验的猜测来完成的，并且经常导致无效的众测。更具体地说，发现在当前的众测实践中，平均 32% 的测试成本是浪费支出。本文旨在通过引入自动化决策支持来监控和确定关闭众测任务的适当时间来应对这一挑战。为此，它首先研究了基于工业数据集对众测任务进行密切预测的必要性和可行性。接下来，它提出了一种名为 iSENSE2.0 的紧密预测方法，该方法应用增量抽样技术来处理按时间顺序到达的众测报告，并将它们组织成固定大小的组作为动态输入。然后，重复标记器分析接收到的人群报告的重复状态，基于 CRC（Capture-ReCapture）的关闭估计器根据动态错误到达状态生成关闭决策。此外，基于覆盖率的健全性检查器旨在增强密切预测的稳定性和性能。最后，对来自最大众测平台之一的 306 个众测任务的 56,920 份报告进行了 iSENSE2.0 的评估。结果表明，可以检测到中位数为 100% 的错误，同时节省了 30% 的成本。iSENSE2.0 的性能与最先进的方法没有显着差异我感觉到，而后者依赖于重复标签，这通常被认为是耗时且获取繁琐的。