The pervasive application of algorithmic decision-making is raising concerns on the risk of unintended bias in AI systems deployed in critical settings such as healthcare. The detection and mitigation of model bias is a very delicate task that should be tackled with care and involving domain experts in the loop. In this paper we introduce FairLens, a methodology for discovering and explaining biases. We show how this tool can audit a fictional commercial black-box model acting as a clinical decision support system (DSS). In this scenario, the healthcare facility experts can use FairLens on their historical data to discover the biases of the model before incorporating it into the clinical decision flow. FairLens first stratifies the available patient data according to demographic attributes such as age, ethnicity, gender and healthcare insurance; it then assesses the model performance on such groups highlighting the most common misclassifications. Finally, FairLens allows the expert to examine one misclassification of interest by explaining which elements of the affected patients’ clinical history drive the model error in the problematic group. We validate FairLens’ ability to highlight bias in multilabel clinical DSSs introducing a multilabel-appropriate metric of disparity and proving its efficacy against other standard metrics.

算法决策的普遍应用引发了人们对部署在医疗保健等关键环境中的人工智能系统意外偏见风险的担忧。模型偏差的检测和缓解是一项非常微妙的任务，应该小心处理并让领域专家参与进来。在本文中，我们介绍了 FairLens，这是一种发现和解释偏差的方法。我们展示了该工具如何审核充当临床决策支持系统 (DSS) 的虚构商业黑盒模型。在这种情况下，医疗机构专家可以在他们的历史数据上使用 FairLens 来发现模型的偏差，然后再将其纳入临床决策流程。FairLens 首先根据年龄、种族、性别和医疗保险等人口统计属性对可用的患者数据进行分层；然后评估这些组的模型性能，突出显示最常见的错误分类。最后，FairLens 允许专家通过解释受影响患者临床病史的哪些元素导致问题组中的模型错误来检查感兴趣的错误分类。我们验证了 FairLens 在多标签临床 DSS 中突出偏差的能力，引入了适合多标签的差异度量，并证明其对其他标准度量的有效性。