Learning to reject is a special kind of self-awareness (the ability to know what you do not know), which is an essential factor for humans to become smarter. Although machine intelligence has become very accurate nowadays, it lacks such kind of self-awareness and usually acts as omniscient, resulting in overconfident errors. This article presents a comprehensive overview of this topic from three perspectives: confidence, calibration, and discrimination. Confidence is an important measurement for the reliability of model predictions. Rejection can be realized by setting thresholds on confidence. However, most models, especially modern deep neural networks, are usually overconfident. Therefore, calibration is a process to ensure confidence matching the actual likelihood of correctness, including two approaches: post-calibration and self-calibration. Calibration reflects the global characteristic of confidence, and the local distinguishing property of confidence is also important. In light of this, discrimination focuses on the performance of accepting positive samples while rejecting negative samples. As a binary classification problem, the challenge of discrimination comes from the missing and nonrepresentativeness of the negative data. Three discrimination tasks are comprehensively analyzed and discussed: failure rejection, unknown rejection, and fake rejection. By rejecting failures, the risk could be controlled especially for mission-critical applications. By rejecting unknowns, the awareness of the knowledge blind zone would be enhanced. By rejecting fakes, security and privacy could be protected. We provide a general taxonomy, organization, and discussion of the methods for solving these problems, which are studied separately in the literature. The connections between different approaches and future directions that are worth further investigation are also presented. With a discriminative and calibrated confidence, learning to reject will let the decision-making process be more practical, reliable, and secure.

中文翻译：

---

学会拒绝的调查

学会拒绝是一种特殊的自我意识（知道自己不知道的能力），是人类变得更聪明的必备因素。虽然现在机器智能已经非常准确了，但它缺乏这种自我意识，通常表现得无所不知，导致过度自信的错误。本文从三个角度全面概述了该主题：置信度、校准和辨别力。置信度是衡量模型预测可靠性的重要指标。可以通过设置置信度阈值来实现拒绝。然而，大多数模型，尤其是现代深度神经网络，通常都过于自信。因此，校准是一个确保置信度匹配实际正确可能性的过程，包括两种方法：后校准和自校准。标定体现了置信度的全局特征，置信度的局部区分性也很重要。鉴于此，判别重点在于接受正样本而拒绝负样本的表现。作为一个二元分类问题，判别的挑战来自于负数据的缺失和不具有代表性。综合分析讨论了三种判别任务：失败拒绝、未知拒绝和假拒绝。通过拒绝故障，可以控制风险，尤其是对于关键任务应用程序。通过拒绝未知，将增强知识盲区的意识。通过拒绝假货，可以保护安全和隐私。我们提供一般分类、组织、以及解决这些问题的方法的讨论，这在文献中是单独研究的。还介绍了值得进一步研究的不同方法和未来方向之间的联系。有了辨别和校准的信心，学会拒绝会让决策过程更加实际、可靠和安全。