Using the Linear Mixed-Effect Model Framework to Estimate Generalizability Variance Components in R
A lme4 Package Application
Abstract
Abstract. Extending from classical test theory, G theory allows more sources of variations to be investigated and therefore provides the accuracy of generalizing observed scores to a broader universe. However, G theory has been used less due to the absence of analytic facilities for this purpose in popular statistical software packages. Besides, there is rarely a systematic G theory introduction in the linear mixed-effect model context, which is a widely taught technique in statistical analysis curricula. The present paper fits G theory into linear mixed-effect models and estimates the variance components via the well-known lme4 package in R. Concrete examples, modeling procedures, and R syntax are illustrated so that practitioners may use G theory for their studies. Realizing the G theory estimation in R provides more flexible features than other platforms, such that users need not rely on specialized software such as GENOVA and urGENOVA.
References
(2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59, 390–412. https://doi.org/10.1016/j.jml.2007.12.005
(2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01
(1994).
Variance components in generalizability theory . In C. R. ReynoldsEd., Cognitive Assessment: A Multidisciplinary Perspective (pp. 175–207). New York, NY: Plenum Press. https://doi.org/10.1007/978-1-4757-9730-5_9(2001a). Generalizability theory https://doi.org/10.1007/978-1-4757-3456-0. New York, NY: Springer
(2001). Manual for mGENOVA. Iowa City, IA: Iowa Testing Programs, University of Iowa.
(2010). Applying generalizability theory using EduG. New York, NY: Routledge.
(1956). Average values of mean squares in factorials. The Annals of Mathematical Statistics, 27, 907–949. https://doi.org/10.1214/aoms/1177728067
(1976). Restricted maximum likelihood (REML) estimation of variance components in the mixed model. Technometrics, 18, 31–38. https://doi.org/10.1080/00401706.1976.10489397
(1972). The dependability of behavioral measurements: Theory of generalizability scores and profiles. New York, NY: Wiley.
(1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 137–163. https://doi.org/10.1111/j.2044-8317.1963.tb00206.x
(1991). Quality control in the development and use of performance assessments. Applied Measurement in Education, 4, 289–303. https://doi.org/10.1207/s15324818ame0404_3
(2016). Extending the linear model with R: Generalized linear, mixed effects and nonparametric regression models (2nd ed.). Boca Raton, FL: CRC Press.
(1960). Complex analyses of variance: General problems. Psychometrika, 25, 127–152. https://doi.org/10.1007/BF02288577
(1953). Estimation of variance and covariance components. Biometrics, 9, 226–252. https://doi.org/10.2307/3001853
(2008). Categorical data analysis: Away from ANOVAs (transformation or not) and towards logit mixed models. Journal of memory and language, 59, 434–446. https://doi.org/10.1016/j.jml.2007.11.007
(2017). A Bayesian approach to estimating variance components within a multivariate generalizability theory framework. Behavior Research Methods. Advance online publication. https://doi.org/10.3758/s13428-017-0986-3
(1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
(1987). An alternative method for variance component estimation: Applications to generalizability theory (Unpublished doctoral dissertation). Los Angeles, CA: University of California.
(1990). An alternative method for estimating variance components in generalizability theory. Psychological Reports, 66, 102–109. https://doi.org/10.2466/pr0.1990.66.2.379
(1996). Estimating variance components in generalizability theory: The covariance structure analysis approach. Structural Equation Modelling, 3, 290–299. https://doi.org/10.1080/10705519609540045
(2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods, 7, 105–125. https://doi.org/0.1037/1082-989X.7.1.105
(2008). Ill-structured measurement designs in organizational research: Implications for estimating interrater reliability. Journal of Applied Psychology, 93, 959–981. https://doi.org/10.1037/0021-9010.93.5.959
. (2017). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.Rproject.org/
(1970). Estimation of heteroscedastic variances in linear models. Journal of the American Statistical Association, 65(329), 161–172. https://doi.org/10.1080/01621459.1970.10481070
(2009). Measurement precision of spoken English proficiency scores on the USMLE Step 2 Clinical Skills Examination. Academic Medicine, 84, S83–S85. https://doi.org/10.1097/ACM.0b013e3181b37d01
(2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147–177. https://doi.org/10.1037/1082-989X.7.2.147
(1971). Linear models. New York, NY: Wiley.
(2009). Variance components (Vol. 391) New York, NY: Wiley.
(1991). Generalizability theory. A primer. Newbury Park, CA: Sage.
(2012). Generalized linear mixed models: Modern concepts, methods and applications. Boca Raton, FL: CRC Press.
(2006). 4 Reliability Coefficients and Generalizability Theory. Handbook of Statistics, 26, 81–124. https://doi.org/10.1016/S0169-7161(06)26004-8
(2001). Comparison of GEE, MINQUE, ML, and REML estimating equations for normally distributed data. The American Statistician, 55, 125–130. https://doi.org/10.1198/000313001750358608
