Government documents must be manually reviewed to identify any sensitive information, e.g., confidential information, before being publicly archived. However, human-only sensitivity review is not practical for born-digital documents due to, for example, the volume of documents that are to be reviewed. In this work, we conduct a user study to evaluate the effectiveness of sensitivity classification for assisting human sensitivity reviewers. We evaluate how the accuracy and confidence levels of sensitivity classification affects the number of documents that are correctly judged as being sensitive (reviewer accuracy) and the time that it takes to sensitivity review a document (reviewing speed). In our within-subject study, the participants review government documents to identify real sensitivities while being assisted by three sensitivity classification treatments , namely None (no classification predictions), Medium (sensitivity predictions from a simulated classifier with a balanced accuracy (BAC) of 0.7), and Perfect (sensitivity predictions from a classifier with an accuracy of 1.0). Our results show that sensitivity classification leads to significant improvements (ANOVA, p < 0.05) in reviewer accuracy in terms of BAC (+37.9% Medium , +60.0% Perfect ) and also in terms of F 2 (+40.8% Medium , +44.9% Perfect ). Moreover, we show that assisting reviewers with sensitivity classification predictions leads to significantly increased (ANOVA, p < 0.05) mean reviewing speeds (+72.2% Medium , +61.6% Perfect ). We find that reviewers do not agree with the classifier significantly more as the classifier’s confidence increases. However, reviewing speed is significantly increased when the reviewers agree with the classifier (ANOVA, p < 0.05). Our in-depth analysis shows that when the reviewers are not assisted with sensitivity predictions, mean reviewing speeds are 40.5% slower for sensitive judgements compared to not-sensitive judgements. However, when the reviewers are assisted with sensitivity predictions, the difference in reviewing speeds between sensitive and not-sensitive judgements is reduced by ˜10%, from 40.5% to 30.8%. We also find that, for sensitive judgements, sensitivity classification predictions significantly increase mean reviewing speeds by 37.7% when the reviewers agree with the classifier’s predictions ( t -test, p < 0.05). Overall, our findings demonstrate that sensitivity classification is a viable technology for assisting human reviewers with the sensitivity review of digital documents.

中文翻译：

---

敏感度分类的准确性和置信度如何影响数字敏感度审查

必须人工审查政府文件以识别任何敏感的公开存档之前的信息，例如机密信息。然而，由于例如要审查的文件量大，人工敏感性审查对于原生数字文件是不切实际的。在这项工作中，我们进行了一项用户研究，以评估敏感性分类的有效性协助人类敏感性审查员。我们评估敏感性分类的准确性和置信度如何影响被正确判断为敏感的文档数量（审阅者准确性）以及敏感度审查文档所需的时间（审阅速度）。在我们的主题内研究中，参与者审查政府文件以识别真正的敏感性，同时得到三个敏感性分类的帮助治疗，即没有（没有分类预测），中等的（灵敏度预测来自模拟的平衡精度（BAC）为 0.7 的分类器），以及完美的（来自分类器的灵敏度预测，精度为 1.0）。我们的结果表明，敏感性分类导致显着改善（ANOVA，p< 0.05) 在 BAC 方面的审稿人准确度 (+37.9%中等的, +60.0%完美的) 以及在 F 方面2(+40.8%中等的, +44.9%完美的）。此外，我们表明，协助审稿人进行敏感性分类预测会导致显着增加（ANOVA，p< 0.05) 平均审查速度 (+72.2%中等的, +61.6%完美的）。我们发现，随着分类器置信度的增加，审阅者对分类器的同意程度明显不同。但是，当审稿人同意分类器（ANOVA，p< 0.05)。我们的深入分析表明，当审稿人不协助进行敏感性预测时，敏感判断的平均审阅速度比非敏感判断慢 40.5%。然而，当审稿人是在敏感性预测的辅助下，敏感判断和不敏感判断之间的审查速度差异减少了约 10%，从 40.5% 降低到 30.8%。我们还发现，对于敏感判断，当审稿人同意分类器的预测时，敏感度分类预测会显着提高平均审阅速度 37.7%（吨-测试，p< 0.05)。总体而言，我们的研究结果表明，敏感性分类是一种可行的技术，可帮助人工审阅者对数字文档进行敏感性审查。