Abstract. A significant problem that assessments of scientific reasoning face at the level of higher education is the question of domain generality, that is, whether a test will produce biased results for students from different domains. This study applied three recently developed methods of analyzing differential item functioning (DIF) to evaluate the domain generality assumption of a common scientific reasoning test. Additionally, we evaluated the usefulness of these new, tree- and lasso-based, methods to analyze DIF and compared them with methods based on classical test theory. We gave the scientific reasoning test to 507 university students majoring in physics, biology, or medicine. All three DIF analysis methods indicated a domain bias present in about one-third of the items, mostly benefiting biology students. We did not find this bias by using methods based on classical test theory. Those methods indicated instead that all items were easier for physics students compared to biology students. Thus, the tree- and lasso-based methods provide a clear added value to test evaluation. Taken together, our analyses indicate that the scientific reasoning test is neither entirely domain-general, nor entirely domain-specific. We advise against using it in high-stakes situations involving domain comparisons.

中文翻译：

---

使用微分项函数分析普通科学推理测试的领域普遍性

摘要。高等教育水平的科学推理评估面临的一个重要问题是领域普遍性问题，即测试是否会对来自不同领域的学生产生有偏见的结果。本研究应用了三种最近开发的分析差异项目功能 (DIF) 的方法来评估常见科学推理测试的领域普遍性假设。此外，我们评估了这些基于树和套索的新方法分析 DIF 的有用性，并将它们与基于经典测试理论的方法进行了比较。我们对507名物理学、生物学或医学专业的大学生进行了科学推理测试。所有三种 DIF 分析方法都表明大约三分之一的项目存在领域偏差，主要有利于生物学学生。通过使用基于经典测试理论的方法，我们没有发现这种偏差。相反，这些方法表明，与生物学生相比，物理学生的所有项目都更容易。因此，基于树和套索的方法为测试评估提供了明确的附加值。总之，我们的分析表明，科学推理测试既不完全是领域通用的，也不完全是领域特定的。我们建议不要在涉及域比较的高风险情况下使用它。也不完全是特定领域的。我们建议不要在涉及域比较的高风险情况下使用它。也不完全是特定领域的。我们建议不要在涉及域比较的高风险情况下使用它。