Contextual representations learned by language models can often encode undesirable attributes, like demographic associations of the users, while being trained for an unrelated target task. We aim to scrub such undesirable attributes and learn fair representations while maintaining performance on the target task. In this paper, we present an adversarial learning framework "Adversarial Scrubber" (ADS), to debias contextual representations. We perform theoretical analysis to show that our framework converges without leaking demographic information under certain conditions. We extend previous evaluation techniques by evaluating debiasing performance using Minimum Description Length (MDL) probing. Experimental evaluations on 8 datasets show that ADS generates representations with minimal information about demographic attributes while being maximally informative about the target task.

中文翻译：

---

用于文本分类的人口统计信息的对抗性清理

通过语言模型学习的上下文表示通常可以编码不需要的属性，例如用户的人口统计关联，同时针对不相关的目标任务进行训练。我们的目标是去除这些不受欢迎的属性并学习公平的表示，同时保持目标任务的性能。在本文中，我们提出了一个对抗性学习框架“Adversarial Scrubber”（ADS），以消除上下文表示的偏差。我们进行理论分析以表明我们的框架在某些条件下收敛而不会泄漏人口统计信息。我们通过使用最小描述长度 (MDL) 探测评估去偏差性能来扩展先前的评估技术。