Background

The 2021 NIJ recidivism forecasting challenge asks participants to construct predictive models of recidivism while balancing false positive rates across groups of Black and white individuals through a multiplicative fairness score. We investigate the performance of several models for forecasting 1-year recidivism and optimizing the NIJ multiplicative fairness metric.

Methods

We consider standard linear and logistic regression, a penalized regression that optimizes a convex surrogate loss (that we show has an analytical solution), two post-processing techniques, linear regression with re-balanced data, a black-box general purpose optimizer applied directly to the NIJ metric and a gradient boosting machine learning approach.

Results

For the set of models investigated, we find that a simple heuristic of truncating scores at the decision threshold (thus predicting no recidivism across the data) yields as good or better NIJ fairness scores on held out data compared to other, more sophisticated approaches. We also find that when the cutoff is further away from the base rate of recidivism, as is the case in the competition where the base rate is 0.29 and the cutoff is 0.5, then simply optimizing the mean square error gives nearly optimal NIJ fairness metric solutions.

Conclusions

The multiplicative metric in the 2021 NIJ recidivism forecasting competition encourages solutions that simply optimize MSE and/or use truncation, therefore yielding trivial solutions that forecast no one will recidivate.

中文翻译：

---

关于 NIJ 累犯预测挑战中乘法公平分数的说明

背景

2021 年 NIJ 累犯预测挑战赛要求参​​与者构建累犯预测模型，同时通过乘法公平分数平衡黑人和白人群体的误报率。我们调查了几种模型的性能，用于预测 1 年累犯和优化 NIJ 乘法公平度量。

方法

我们考虑标准线性和逻辑回归、优化凸代理损失的惩罚回归（我们展示了一个解析解）、两种后处理技术、重新平衡数据的线性回归、直接应用的黑盒通用优化器NIJ 指标和梯度提升机器学习方法。

结果

对于所研究的模型集，我们发现，与其他更复杂的方法相比，在决策阈值处截断分数的简单启发式方法（从而预测数据中没有累犯）会在保留数据上产生同样或更好的 NIJ 公平性分数。我们还发现，当截止点离再犯的基本率更远时，就像在比赛中基本率是 0.29 而截止点是 0.5 的情况一样，那么简单地优化均方误差就可以得到近乎最优的 NIJ 公平度量解.

结论

2021 年 NIJ 累犯预测竞赛中的乘法度量鼓励仅优化 MSE 和/或使用截断的解决方案，从而产生预测没有人会再犯的微不足道的解决方案。