Crowdsourcing receives massive vote information from non-expert workers, for finishing tasks that can hardly be handled by current technology of machine intelligence. Massive vote information and non-expert workers bring serious issues of labor costs and the efficiency of crowdsourcing. This paper focuses on the tasks, classifying objects in images or videos into a set of given candidates by letting workers vote on a set of options that characterize these candidates. Designing a good asking strategy, i.e., setting up the order of presenting the options to a worker and asking the worker whether an option is true or false, is one starting point to save labor costs and enhance efficiency of deciding the correct answer from the candidates. We propose the problem of determining the time steps of vote collection before stopping to set up the asking strategy. In terms of this problem, we establish a single-step collection based partially observable Markov decision process (POMDP) to analyze how a vote influences the whole system, for instance, influences the belief over each option. Formally define the multistep collection problem as the timed decision (TD) problem. We propose the MC-EVA algorithm based on Monte Carlo sampling to solve the TD problem. Evaluate the MC-EVA algorithm over three simple but typical cases and a real-world Galaxy Zoo 2 project. Experiments show MC-EVA’s great superiority in runtime over the state-of-the-art single-step collection algorithm, and its superiority in effectiveness than other multistep collection algorithms; show its labor cost saving and enhanced efficiency with the use of calculated asking strategies.

众包从非专家工作者那里接收大量投票信息，以完成当前机器智能技术难以处理的任务。海量的投票信息和非专家工作者带来了劳动力成本和众包效率的严重问题。本文侧重于任务，通过让工作人员对表征这些候选对象的一组选项进行投票，将图像或视频中的对象分类为一组给定的候选对象。设计一个好的询问策略，即设置向工人展示选项的顺序并询问工人选项是真是假，是节省人力成本和提高候选人决定正确答案效率的一个起点. 我们提出了在停止设置询问策略之前确定投票收集的时间步长的问题。针对这个问题，我们建立了一个基于部分可观察马尔可夫决策过程（POMDP）的单步集合来分析投票如何影响整个系统，例如，影响对每个选项的信念。将多步收集问题正式定义为定时决策（TD）问题。我们提出了基于蒙特卡罗采样的 MC-EVA 算法来解决 TD 问题。通过三个简单但典型的案例和一个真实的 Galaxy Zoo 2 项目评估 MC-EVA 算法。实验表明，MC-EVA在运行时间上优于最先进的单步收集算法，在有效性上优于其他多步收集算法；