Abstract
This study provides a systematic review of the methodological features of meta-analyses in second language learning. The synthesis aims to inform how meta-analyses in L2 learning have been conducted, evaluate whether methodological decisions are aligned with norms and standards, identify issues, and suggest solutions based on expert advice, statistical guides, and best practices. A total of 120 meta-analyses were retrieved and coded for key features related to bibliographic and demographic characteristics, search and selection, publication bias, quality control, data coding, data analysis, and effect size interpretation. The synthesis showed that 98 meta-analyses examined the effectiveness of instructional treatments, 21 investigated correlations, and one explored the occurrence of events. These meta-analyses included an average of 37 primary studies (range = 9–302). Common selection criteria the meta-analyses applied included publication type, availability of data for effect size calculation, learner traits, learners’ target languages, publication dates, publication language, independent variables, and dependent variables. Major strategies used to detect publication bias included creating a funnel plot, using trim-and-fill analysis, and calculating a fail-safe N. Typical moderators examined in the meta-analyses related to research context, treatment features, sample characteristics, and outcome measures. The synthesis also identified a number of issues, including failure to report key features such as model selection (fixed- vs. random-effects model), effect size weighting, and so on; conducting moderator analysis based on very small cell sizes (e.g., only one study in a subgroup); lack of justification for certain methodological decisions such as using d instead of g, using confidence intervals rather than Q-tests to identify significant moderators; lack of quality control; and confounding study-based and synthesis-based moderators. We offer advice on identified issues and call for more transparency of reporting.
References
Ahn, Soyeon & Betsy Jane Becker. 2011. Incorporating quality scores in meta-analysis. Journal of Educational and Behavioral Statistics 36(5). 555–585. https://doi.org/10.3102/1076998610393968.Search in Google Scholar
Al-Hoorie, Ali H. 2018. The L2 motivational self system: A meta-analysis. Studies in Second Language Learning and Teaching 8(4). 721–754. https://doi.org/10.14746/ssllt.2018.8.4.2.Search in Google Scholar
Ardasheva, Yuliya, Zhe Wang, Olusola O. Adescope & Jeffrey C. Valentine. 2017. Exploring effectiveness and moderators of language learning strategy instruction on second language and self-regulated learning outcomes. Review of Educational Research 87(3). 544–582. https://doi.org/10.3102/0034654316689135.Search in Google Scholar
Bibauw, Serge, Thomas François, Wim Van den Noortgate & Piet Desmet. 2022. Dialogue systems for language learning: A meta-analysis. Language, Learning and Technology 26(1). 1–25.Search in Google Scholar
Boers, Frank, Lara Bryfonski, Farahnaz Faez & Todd McKay. 2021. A call for cautious interpretation of meta-analytic reviews. Studies in Second Language Acquisition 43(1). 2–24. https://doi.org/10.1017/s0272263120000327.Search in Google Scholar
Borenstein, Michael, Larry Hedges, Julian Higgins & Hannah Rothstein. 2008. Comprehensive metaanalysis. Englewood, NJ: Biostat.Search in Google Scholar
Brown, Dan. 2016. The type and linguistic foci of oral corrective feedback in the L2 classroom: A meta-analysis. Language Teaching Research 20(4). 436–458. https://doi.org/10.1177/1362168814563200.Search in Google Scholar
Chen, Xinjie, Jinbo He, Elizabeth Swanson, Zhihui Cai & Xitao Fan. 2021. Big five personality traits and second language learning: A meta-analysis of 40 years’ research. Educational Psychology Review 34. 851–887. https://doi.org/10.1007/s10648-021-09641-6.Search in Google Scholar
Chiu, Yi-Hui. 2013. Computer-assisted second language vocabulary instruction: A meta-analysis. British Journal of Educational Technology 44(2). E52–E56. https://doi.org/10.1111/j.1467-8535.2012.01342.x.Search in Google Scholar
Chiu, Yi-Hui, Chian-Wen Kao & Barry Lee Reynolds. 2012. The relative effectiveness of digital game-based learning types in English as a foreign language setting: A meta-analysis. British Journal of Educational Technology 43(4). E104–E107. https://doi.org/10.1111/j.1467-8535.2012.01295.x.Search in Google Scholar
Cohen, Jacob. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1). 37–46. https://doi.org/10.1177/001316446002000104.Search in Google Scholar
Cohen, Jacob. 1988. Statistical power analysis for the behavioral sciences, 2nd edn. Mahwah, NJ: Lawrence Erlbaum Associates Inc.Search in Google Scholar
Cooper, Harris. 2010. Research synthesis and meta-analysis: A step-by-step approach. Los Angeles, USA: Sage.Search in Google Scholar
Cuijpers, Pim, Clara Miguel, Marketa Ciharova, Pauline Aalten, Neeltje Batelaan, Elske Salemink, Philip Spinhoven, Sascha Struijs, Leonore de Wit, Claudio Gentili, David Ebert, Mathias Harrer, Ronny Bruffaerts, Ronald C. Kessler & Eirini Karyotaki. 2021. Prevention and treatment of mental health and psychosocial problems in college students: An umbrella review of meta-analyses. Clinical Psychology: Science and Practice 28(3). 229–244. https://doi.org/10.1037/cps0000030.Search in Google Scholar
Dahabreh, Issa J., Mei Chung, Georgios D. Kitsios, Teruhiko Terasawa, Gowri Raman, Athina Tatsioni, Annette Tobar, Joseph Lau, Thomas A. Trikalinos & Christopher H. Schmid. 2013. Survey of the methods and reporting practices in published meta-analyses of test performance: 1987 to 2009. Research Synthesis Methods 4(3). 242–255. https://doi.org/10.1002/jrsm.1083.Search in Google Scholar
Dong, Yang & Bonnie Wing-Yin Chow. 2022. Home literacy environment and English as a second language acquisition: A meta-analysis. Language Learning and Development 18. 485–499. https://doi.org/10.1080/15475441.2021.2003197.Search in Google Scholar
Durant, Philip. 2014. Corpus frequency and second language learners’ knowledge of collocations: A meta-analysis. International Journal of Corpus Linguistics 19(4). 443–477. https://doi.org/10.1075/ijcl.19.4.01dur.Search in Google Scholar
Field, Andy P. & Raphael Gillett. 2010. How to do a meta-analysis. British Journal of Mathematical and Statistical Psychology 63(3). 665–694. https://doi.org/10.1348/000711010x502733.Search in Google Scholar
Hedges, Larry V. 1981. Distribution theory for Glass’s estimator of effect size and related estimators. Journal of Educational Statistics 6. 107–128. https://doi.org/10.2307/1164588.Search in Google Scholar
Huang, Shufen, Victor Wilson & Zohreh Eslami. 2012. The effects of task involvement load on L2 incidental vocabulary learning: A meta-analytic study. The Modern Language Journal 96(4). 544–557. https://doi.org/10.1111/j.1540-4781.2012.01394.x.Search in Google Scholar
Jia, Chengyuan & Khe Foon Hew. 2021. Toward a set of design principles for decoding training: A systematic review of studies of English as a foreign/second language listening education. Educational Research Review 33. 100392. https://doi.org/10.1016/j.edurev.2021.100392.Search in Google Scholar
Kim, Su Kyung & Stuart Webb. 2022. The effects of spaced practice on second language learning: A meta-analysis. Language Learning 72(1). 269–319. https://doi.org/10.1111/lang.12479.Search in Google Scholar
Li, Shaofeng. 2010. The effectiveness of corrective feedback in SLA: A meta-analysis. Language Learning 60(2). 309–365. https://doi.org/10.1111/j.1467-9922.2010.00561.x.Search in Google Scholar
Li, Shaofeng. 2016. The construct validity of language aptitude. Studies in Second Language Acquisition 38. 801–842. https://doi.org/10.1017/s027226311500042x.Search in Google Scholar
Li, Shaofeng. 2017. Teacher and learner beliefs about corrective feedback. In Hossein Nassaji & Eva Kartchava (eds.), Corrective feedback in second language teaching and learning, 143–157. New York: Routledge.10.4324/9781315621432-11Search in Google Scholar
Li, Shaofeng & Hong Wang. 2018. Traditional literature review and research synthesis. In Aek Phakiti, Peter De Costa, Luke Plonsky & Sue Starfield (eds.), The Palgrave handbook of applied linguistics research methodology, 123–144. London: Palgrave Macmillan.10.1057/978-1-137-59900-1_6Search in Google Scholar
Li, Shaofeng, Natsuko Shintani & Rod Ellis. 2012. Doing meta-analysis in SLA: Practices, choices, and standards. Contemporary Foreign Language Studies 384(12). 1–17.Search in Google Scholar
Lin, Huifen. 2015. A meta-synthesis of empirical research on the effectiveness of computer-mediated communication (CMC) in SLA. Language Learning & Technology 19(2). 85–117. http://llt.msu.edu/issues/june2015/lin.pdf.Search in Google Scholar
Lin, Huifen, Tsuiping Chen & Hsien-Chin Liou. 2018. Transparency of reporting in CALL meta-analyses between 2003 and 2015. ReCALL 30(3). 253–277. https://doi.org/10.1017/s0958344017000271.Search in Google Scholar
Lipsey, Mark W. & David B. Wilson. 2001. Practical meta-analysis. Thousand Oaks, CA: SAGE Publications.Search in Google Scholar
Masgoret, Anne-Marie & Robert C. Gardner. 2003. Attitudes, motivation, and second language learning: A meta-analysis of studies conducted by gardner and associates. Language Learning 53(1). 123–163. https://doi.org/10.1111/1467-9922.00227.Search in Google Scholar
Morris, Scott B. & Richard P. DeShon. 2002. Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods 7(1). 105–125. https://doi.org/10.1037/1082-989x.7.1.105.Search in Google Scholar
Norouzian, Reza. 2021. Interrater reliability in second language meta-analyses: The case of categorical moderators. Studies in Second Language Acquisition 43. 896–915. https://doi.org/10.1017/s0272263121000061.Search in Google Scholar
Oswald, Frederick L. & Luke Plonsky. 2010. Meta-analysis in second language research: Choices and challenges. Annual Review of Applied Linguistics 30. 85–110. https://doi.org/10.1017/s0267190510000115.Search in Google Scholar
Page, Matthew J., Jonathan A. C. Sterne, Julian P. T. Higgins & Matthias Egger. 2021. Investigating and dealing with publication bias and other reporting biases in meta-analyses of health research: A review. Research Synthesis Methods 12(2). 248–259. https://doi.org/10.1002/jrsm.1468.Search in Google Scholar
Plonsky, Luke. 2011. The effectiveness of second language strategy instruction: A meta-analysis. Language Learning 61(4). 993–1038. https://doi.org/10.1111/j.1467-9922.2011.00663.x.Search in Google Scholar
Plonsky, Luke. 2013. Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition 35(4). 655–687. https://doi.org/10.1017/s0272263113000399.Search in Google Scholar
Plonsky, Luke & Deirdre J. Derrick. 2016. A meta-analysis of reliability coefficients in second language research. The Modern Language Journal 100(2). 538–553. https://doi.org/10.1111/modl.12335.Search in Google Scholar
Plonsky, Luke & Frederick L. Oswald. 2012. The Routledge encyclopedia of second language acquisition. London: Routledge.Search in Google Scholar
Plonsky, Luke & Frederick L. Oswald. 2014. How big is big? Interpreting effect sizes in L2 research. Language Learning 64(4). 878–912. https://doi.org/10.1111/lang.12079.Search in Google Scholar
Plonsky, Luke & Frederick L. Oswald. 2015. Meta-analyzing second language research. In Luke Plonsky (ed.), Advancing quantitative methods in second language research, 106–128. London: Routledge.10.4324/9781315870908-6Search in Google Scholar
Plonsky, Luke, Jesse Egbert & Geoffrey T. Laflair. 2015. Bootstrapping in applied linguistics: Assessing its potential using shared data. Applied Linguistics 36(5). 591–610.10.1093/applin/amu001Search in Google Scholar
Plonsky, Luke & Nicole Ziegler. 2016. The CALL–SLA interface: Insights from a second-order synthesis. Language, Learning and Technology 20(2). 17–37.Search in Google Scholar
Russell Valezy, Jane & Nina Spada. 2006. The effectiveness of corrective feedback for the acquisition of L2 grammar: A meta-analysis of the research. In John M. Norris & Lourdes Ortega (eds.), Synthesizing research on language learning and teaching, 133–164. Amsterdam: John Benjamins.10.1075/lllt.13.09valSearch in Google Scholar
Schäfer, Thomas & Marcus A. Schwarz. 2019. The meaningfulness of effect sizes in psychological research: Differences between sub-disciplines and the impact of potential biases. Frontiers in Psychology 10. 813. https://doi.org/10.3389/fpsyg.2019.00813.Search in Google Scholar
Shintani, Natsuko, Shaofeng Li & Rod Ellis. 2013. Comprehension-based versus production-based instruction: A meta-analysis of comparative studies. Language Learning 63(2). 296–329. https://doi.org/10.1111/lang.12001.Search in Google Scholar
Uchihara, Takumi, Stuart Webb & Akifumi Yanagisawa. 2019. The effects of repetition on incidental vocabulary learning: A meta-analysis of correlational studies. Language Learning 69(3). 559–599. https://doi.org/10.1111/lang.12343.Search in Google Scholar
Xu, Zhihong, Manjari Banerjee, Gilbert Ramirez, Gang Zhu & Kausalai (Kay) Wijekumar. 2019. The effectiveness of educational technology applications on adult English language learners’ writing quality: A meta-analysis. Computer Assisted Language Learning 32(1-2). 132–162. https://doi.org/10.1080/09588221.2018.1501069.Search in Google Scholar
Supplementary Material
This article contains supplementary material (https://doi.org/10.1515/applirev-2022-0192).
© 2023 Walter de Gruyter GmbH, Berlin/Boston
