Regression evaluation has been performed for decades. Some metrics have been identified to be robust against shifting and scaling of the data but considering the different distributions of data is much more difficult to address (imbalance problem) even though it largely impacts the comparability between evaluations on different datasets. In classification, it has been stated repeatedly that performance metrics like the F-Measure and Accuracy are highly dependent on the class distribution and that comparisons between different datasets with different distributions are impossible. We show that the same problem exists in regression. The distribution of odometry parameters in robotic applications can for example largely vary between different recording sessions. Here, we need regression algorithms that either perform equally well for all function values, or that focus on certain boundary regions like high speed. This has to be reflected in the evaluation metric. We propose the modification of established regression metrics by weighting with the inverse distribution of function values $Y$ or the samples $X$ using an automatically tuned Gaussian kernel density estimator. We show on synthetic and robotic data in reproducible experiments that classical metrics behave wrongly, whereas our new metrics are less sensitive to changing distributions, especially when correcting by the marginal distribution in $X$. Our new evaluation concept enables the comparison of results between different datasets with different distributions. Furthermore, it can reveal overfitting of a regression algorithm to overrepresented target values. As an outcome, non-overfitting regression algorithms will be more likely chosen due to our corrected metrics.

中文翻译：

---

迈向分布不变回归指标的第一步

回归评估已经进行了几十年。已确定一些指标对于数据的移动和缩放是稳健的，但考虑到数据的不同分布更难以解决（不平衡问题），尽管它在很大程度上影响了对不同数据集的评估之间的可比性。在分类中，已经反复说明了 F-Measure 和 Accuracy 等性能指标高度依赖于类分布，并且不可能在具有不同分布的不同数据集之间进行比较。我们表明回归中存在同样的问题。例如，机器人应用中里程计参数的分布在不同的记录会话之间可能会有很大差异。这里，我们需要回归算法，要么对所有函数值都表现得一样好，要么专注于某些边界区域，如高速。这必须反映在评估指标中。我们建议通过使用自动调整的高斯核密度估计器对函数值 $Y$ 或样本 $X$ 的逆分布进行加权来修改已建立的回归指标。我们在可重复实验中对合成和机器人数据表明，经典指标的行为是错误的，而我们的新指标对变化的分布不太敏感，尤其是在通过 $X$ 中的边际分布进行校正时。我们的新评估概念可以比较具有不同分布的不同数据集之间的结果。此外，它可以揭示回归算法对过度表示的目标值的过度拟合。因此，由于我们修正了指标，更有可能选择非过度拟合回归算法。