Machine learning models are often implemented in cohort with humans in the pipeline, with the model having an option to defer to a domain expert in cases where it has low confidence in its inference. Our goal is to design mechanisms for ensuring accuracy and fairness in such prediction systems that combine machine learning model inferences and domain expert predictions. Prior work on "deferral systems" in classification settings has focused on the setting of a pipeline with a single expert and aimed to accommodate the inaccuracies and biases of this expert to simultaneously learn an inference model and a deferral system. Our work extends this framework to settings where multiple experts are available, with each expert having their own domain of expertise and biases. We propose a framework that simultaneously learns a classifier and a deferral system, with the deferral system choosing to defer to one or more human experts in cases of input where the classifier has low confidence. We test our framework on a synthetic dataset and a content moderation dataset with biased synthetic experts, and show that it significantly improves the accuracy and fairness of the final predictions, compared to the baselines. We also collect crowdsourced labels for the content moderation task to construct a real-world dataset for the evaluation of hybrid machine-human frameworks and show that our proposed learning framework outperforms baselines on this real-world dataset as well.

中文翻译：

---

争取向多位专家公正无误地进行递延

机器学习模型通常是与管道中的人员一起实施的，如果模型对推理的置信度较低，则可以选择向领域专家求助。我们的目标是设计一种机制，以确保结合了机器学习模型推断和领域专家预测的此类预测系统的准确性和公平性。在分类设置中有关“递延系统”的先前工作主要集中于由一位专家进行的管道设置，目的是适应该专家在同时学习推理模型和递延系统时的不准确性和偏见。我们的工作将这个框架扩展到可以聘请多位专家的环境，每位专家都有各自的专业知识和偏见。我们提出了一个框架，该框架可同时学习分类器和递延系统，在分类器具有低置信度的输入情况下，递延系统选择延迟一位或多位人类专家的意见。我们在有偏见的综合专家的综合数据集和内容审核数据集上测试了我们的框架，并表明与基线相比，该框架显着提高了最终预测的准确性和公平性。我们还收集了用于内容审核任务的众包标签，以构建用于评估机器人机混合框架的真实世界数据集，并表明我们提出的学习框架也优于该真实世界数据集的基线。我们在有偏见的综合专家的综合数据集和内容审核数据集上测试了我们的框架，并表明与基线相比，该框架显着提高了最终预测的准确性和公平性。我们还收集了用于内容审核任务的众包标签，以构建用于评估机器人机混合框架的真实世界数据集，并表明我们提出的学习框架也优于该真实世界数据集的基线。我们在有偏见的综合专家的综合数据集和内容审核数据集上测试了我们的框架，并表明与基线相比，该框架显着提高了最终预测的准确性和公平性。我们还收集了用于内容审核任务的众包标签，以构建用于评估机器人机混合框架的真实世界数据集，并表明我们提出的学习框架也优于该真实世界数据集的基线。