Leveraging text, such as social media posts, for causal inferences requires the use of NLP models to 'learn' and adjust for confounders, which could otherwise impart bias. However, evaluating such models is challenging, as ground truth is almost never available. We demonstrate the need for empirical evaluation frameworks for causal inference in natural language by showing that existing, commonly used models regularly disagree with one another on real world tasks. We contribute the first such framework, generalizing several challenges across these real world tasks. Using this framework, we evaluate a large set of commonly used causal inference models based on propensity scores and identify their strengths and weaknesses to inform future improvements. We make all tasks, data, and models public to inform applications and encourage additional research.

中文翻译：

---

用文本调整混杂因素：挑战和因果推理的实证评估框架

利用社交媒体帖子等文本进行因果推断需要使用 NLP 模型来“学习”并调整混杂因素，否则可能会产生偏见。然而，评估此类模型具有挑战性，因为基本事实几乎永远无法获得。我们通过展示现有的、常用的模型在现实世界的任务中经常彼此不一致，证明了对自然语言因果推理的经验评估框架的必要性。我们贡献了第一个这样的框架，概括了这些现实世界任务中的几个挑战。使用这个框架，我们根据倾向得分评估了大量常用的因果推理模型，并确定它们的优势和劣势，以告知未来的改进。我们将所有任务、数据、