Background

Standard methods for analysing data from large-scale assessments (LSA) cannot merely be adopted if hierarchical (or multilevel) regression modelling should be applied. Currently various approaches exist; they all follow generally a design-based model of estimation using the pseudo maximum likelihood method and adjusted weights for the corresponding hierarchies. Specifically, several different approaches to using and scaling sampling weights in hierarchical models are promoted, yet no study has compared them to provide evidence of which method performs best and therefore should be preferred. Furthermore, different software programs implement different estimation algorithms, leading to different results.

Objective and method

In this study, we determine based on a simulation, the estimation procedure showing the smallest distortion to the actual population features. We consider different estimation, optimization and acceleration methods, and different approaches on using sampling weights. Three scenarios have been simulated using the statistical program R. The analyses have been performed with two software packages for hierarchical modelling of LSA data, namely Mplus and SAS.

Results and conclusions

The simulation results revealed three weighting approaches performing best in retrieving the true population parameters. One of them implies using only level two weights (here: final school weights) and is because of its simple implementation the most favourable one. This finding should provide a clear recommendation to researchers for using weights in multilevel modelling (MLM) when analysing LSA data, or data with a similar structure. Further, we found only little differences in the performance and default settings of the software programs used, with the software package Mplus providing slightly more precise estimates. Different algorithm starting settings or different accelerating methods for optimization could cause these distinctions. However, it should be emphasized that with the recommended weighting approach, both software packages perform equally well. Finally, two scaling techniques for student weights have been investigated. They provide both nearly identical results. We use data from the Programme for International Student Assessment (PISA) 2015 to illustrate the practical importance and relevance of weighting in analysing large-scale assessment data with hierarchical models.

中文翻译：

---

多层次建模中的抽样权重：使用PISA抽样结构的调查

背景

如果应应用分层（或多级）回归建模，则不能仅采用用于分析来自大规模评估（LSA）的数据的标准方法。当前存在各种方法。它们通常都遵循基于设计的估计模型，该模型使用伪最大似然法和针对相应层次结构的调整后的权重。具体来说，提倡使用几种不同的方法来使用和缩放层次模型中的采样权重，但尚无研究对其进行比较以提供哪种方法效果最好的证据，因此应首选。此外，不同的软件程序实现不同的估计算法，从而导致不同的结果。

目的与方法

在这项研究中，我们基于模拟确定了估计过程，该过程显示出对实际人口特征的最小失真。我们考虑使用不同的估计，优化和加速方法，以及使用采样权重的不同方法。使用统计程序R模拟了三种情况。已使用两个软件包对LSA数据进行分层建模，即Mplus和SAS，进行了分析。

结果与结论

仿真结果表明，三种加权方法在检索真实总体参数方面表现最佳。其中之一意味着仅使用二级权重（此处为最终学校的权重），并且由于其简单的实现，因此是最有利的一种。这一发现应该为研究人员在分析LSA数据或具有类似结构的数据时在多级建模（MLM）中使用权重提供明确的建议。此外，我们发现所用软件程序的性能和默认设置几乎没有差异，软件包Mplus提供的估算值稍为精确。不同的算法开始设置或不同的优化加速方法可能导致这些区别。但是，应该强调的是，采用建议的加权方法，两种软件包的性能均相同。最后，研究了两种用于学生权重的缩放技术。它们提供几乎相同的结果。我们使用来自2015年国际学生评估计划（PISA）的数据来说明权重在使用分层模型分析大规模评估数据时的实际重要性和相关性。