We propose a general method for measuring complex variables on a continuous, interval spectrum by combining supervised deep learning with the Constructing Measures approach to faceted Rasch item response theory (IRT). We decompose the target construct, hate speech in our case, into multiple constituent components that are labeled as ordinal survey items. Those survey responses are transformed via IRT into a debiased, continuous outcome measure. Our method estimates the survey interpretation bias of the human labelers and eliminates that influence on the generated continuous measure. We further estimate the response quality of each labeler using faceted IRT, allowing responses from low-quality labelers to be removed. Our faceted Rasch scaling procedure integrates naturally with a multitask deep learning architecture for automated prediction on new data. The ratings on the theorized components of the target outcome are used as supervised, ordinal variables for the neural networks' internal concept learning. We test the use of an activation function (ordinal softmax) and loss function (ordinal cross-entropy) designed to exploit the structure of ordinal outcome variables. Our multitask architecture leads to a new form of model interpretation because each continuous prediction can be directly explained by the constituent components in the penultimate layer. We demonstrate this new method on a dataset of 50,000 social media comments sourced from YouTube, Twitter, and Reddit and labeled by 11,000 U.S.-based Amazon Mechanical Turk workers to measure a continuous spectrum from hate speech to counterspeech. We evaluate Universal Sentence Encoders, BERT, and RoBERTa as language representation models for the comment text, and compare our predictive accuracy to Google Jigsaw's Perspective API models, showing significant improvement over this standard benchmark.

中文翻译：

---

通过分面 Rasch 测量和多任务深度学习构建区间变量：仇恨言论应用

我们通过将监督深度学习与分面 Rasch 项目响应理论 (IRT) 的构造测量方法相结合，提出了一种测量连续区间谱上复杂变量的通用方法。我们将目标结构（在我们的案例中的仇恨言论）分解为多个标记为有序调查项目的组成部分。这些调查回复通过 IRT 转换为无偏见的、连续的结果衡量标准。我们的方法估计了人工标记者的调查解释偏差，并消除了对生成的连续测量的影响。我们使用分面 IRT 进一步估计每个贴标机的响应质量，允许删除来自低质量贴标机的响应。我们的分面 Rasch 缩放程序自然地与多任务深度学习架构集成，以自动预测新数据。对目标结果的理论成分的评级用作神经网络内部概念学习的监督有序变量。我们测试了激活函数（序数 softmax）和损失函数（序数交叉熵）的使用，旨在利用序数结果变量的结构。我们的多任务架构导致了一种新的模型解释形式，因为每个连续预测都可以由倒数第二层中的组成部分直接解释。我们在来自 YouTube、Twitter 和 Reddit 并由 11,000 名美国用户标记的 50,000 条社交媒体评论数据集上演示了这种新方法 基于 Amazon Mechanical Turk 的工作人员测量从仇恨言论到反言论的连续范围。我们评估 Universal Sentence Encoders、BERT 和 RoBERTa 作为评论文本的语言表示模型，并将我们的预测准确性与 Google Jigsaw 的 Perspective API 模型进行比较，显示出比此标准基准的显着改进。