In this article, we argue that the unsatisfactory out-of-distribution (OOD) detection performance of neural networks is mainly due to the SoftMax loss anisotropy and propensity to produce low entropy probability distributions in disagreement with the principle of maximum entropy. On the one hand, current OOD detection approaches usually do not directly fix the SoftMax loss drawbacks, but rather build techniques to circumvent it. Unfortunately, those methods usually produce undesired side effects (e.g., classification accuracy drop, additional hyperparameters, slower inferences, and collecting extra data). On the other hand, we propose replacing SoftMax loss with a novel loss function that does not suffer from the mentioned weaknesses. The proposed IsoMax loss is isotropic (exclusively distance-based) and provides high entropy posterior probability distributions. Replacing the SoftMax loss by IsoMax loss requires no model or training changes. Additionally, the models trained with IsoMax loss produce as fast and energy-efficient inferences as those trained using SoftMax loss. Moreover, no classification accuracy drop is observed. The proposed method does not rely on outlier/background data, hyperparameter tuning, temperature calibration, feature extraction, metric learning, adversarial training, ensemble procedures, or generative models. Our experiments showed that IsoMax loss works as a seamless SoftMax loss drop-in replacement that significantly improves neural networks’ OOD detection performance. Hence, it may be used as a baseline OOD detection approach to be combined with current or future OOD detection techniques to achieve even higher results.

中文翻译：

---

熵分布外检测：未知示例的无缝检测


在本文中，我们认为神经网络的分布外（OOD）检测性能不理想主要是由于 SoftMax 损失各向异性以及产生与最大熵原理不一致的低熵概率分布的倾向。一方面，当前的 OOD 检测方法通常不会直接修复 SoftMax 损失的缺点，而是构建技术来规避它。不幸的是，这些方法通常会产生不需要的副作用（例如，分类精度下降、额外的超参数、较慢的推理以及收集额外的数据）。另一方面，我们建议用一种新的损失函数替换 SoftMax 损失，该损失函数不会受到上述缺点的影响。所提出的 IsoMax 损失是各向同性的（完全基于距离），并提供高熵后验概率分布。用 IsoMax 损失替换 SoftMax 损失不需要更改模型或训练。此外，使用 IsoMax 损失训练的模型可以与使用 SoftMax 损失训练的模型一样快速且节能地进行推理。此外，没有观察到分类精度下降。所提出的方法不依赖于异常值/背景数据、超参数调整、温度校准、特征提取、度量学习、对抗性训练、集成过程或生成模型。我们的实验表明，IsoMax 损失可以作为 SoftMax 损失的无缝替代品，显着提高神经网络的 OOD 检测性能。因此，它可以用作基线 OOD 检测方法，与当前或未来的 OOD 检测技术相结合，以获得更高的结果。