Skip to main content

Advertisement

SpringerLink
Log in

A critical deconstruction of computer-based test application in Turkish State University

  • Published:
Education and Information Technologies Aims and scope Submit manuscript

Abstract

Artificial Intelligence (AI) is growing – as can be clearly observed not only from the rising recognition of assistance tools such as Siri (Apple) but also from the newly introduced Google Voiced Translator. Yet, some crucial benchmarks still have to be supplied before it can act as a proxy for a real instructor: imagination, creativity, and spontaneity. Automated assessment containing the use of AI is one of the recent education practices. It accelerates the time for exam grading, eliminates human prejudice, and is as precise as human assessors. However, it has encountered many criticisms in education community, in our case, English as foreign language (EFL) learning community. Therefore, this phenomenological inquiry examined Turkish EFL students’ and instructors’ conceptions on the Versant English Test (VET), an automated test of spoken and written language functioning by means of an AI software. Using semi-structured interview questions and a focus-group discussion, the study adopted a qualitative research design in order to collect the required data. The findings show that EFL university students developed negative attitudes towards VET and that VET is not a reliable and valid test because same questions were observed to have appeared in the computer-based test. In addition, copying and pasting similar sentences produced better results, which decreased the validity and reliability of the test. Another important finding was that the test was reported to have measured only their memory skills but not their language skills. Besides, the curriculum was totally incongruent with the content of the test, which caused a severe washback in EFL learners.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Au, W. (2013). What’sa nice test like you doing in a place like this. Rethinking Schools, 27(4), 22–27.

    Google Scholar 

  • Bach, J. (2020). When artificial intelligence becomes general enough to understand itself. Commentary on Pei Wang’s paper “on defining artificial intelligence”. Journal of Artificial General Intelligence, 11(2), 15–18.

    Google Scholar 

  • Barr, A., & Feigenbaum, E. A. (Eds.). (2014). The handbook of artificial intelligence (Vol. 2). Burlington: Morgan Kaufmann Publications.

  • Bernstein, J., & Cheng, J. (2007). Logic and validation of fully automatic spoken English test. In V. M. Holland & F. P. Fisher (Eds.), The path of speech technologies in computer assisted language learning: From research toward practice (pp. 174–194). New York: Routledge.

    Google Scholar 

  • Bettis, R. A., & Hitt, M. A. (1995). The new competitive landscape. Strategic Management Journal, 16(S1), 7–19.

    Google Scholar 

  • Bin, Y., & Mandal, D. (2019). English teaching practice based on artificial intelligence technology. Journal of Intelligent & Fuzzy Systems, (preprint), 37(3), 1–11.

  • Booth, S. (1997). On phenomenography, learning and teaching. Higher Education Research & Development, 16(2), 135–158.

    Google Scholar 

  • Bowden, J. (2000). Phenomenography. Melbourne: RMIT University Press.

    Google Scholar 

  • Carmines, E. G., & Zeller, R. A. (1979). Reliability and validity assessment (Vol. 17). New York: Sage publications.

    Google Scholar 

  • Chapelle, C. A., & Douglas, D. (2006). Assessing language through computer technology. Stuttgart: Ernst Klett Sprachen.

    Google Scholar 

  • Chapelle, C. A. (2010). Research and practice: A look at issues in technology for second language learning. Language Learning & Technology, 14(3), 27–30.

    Google Scholar 

  • Cheng, L., Klinger, D., Fox, J., Doe, C., Jin, Y., & Wu, J. (2014). Motivation and test anxiety in test performance across three testing contexts: The CAEL, CET, and GEPT. Tesol Quarterly, 48(2), 300–330.

    Google Scholar 

  • Clark, P., & Etzioni, O. (2016). My computer is an honor student—But how intelligent is it? Standardized tests as a measure of AI. AI Magazine, 37(1), 5–12.

    Google Scholar 

  • Dascalu, M., Chioasca, E. V., & Trausan-Matu, S. (2008). ASAP-an advanced system for assessing chat participants. In International Conference on Artificial Intelligence: Methodology, Systems, and Applications (pp. 58–68). Berlin, Heidelberg: Springer.

    Google Scholar 

  • Devedžić, V. (2004). Web intelligence and artificial intelligence in education. Educational Technology & Society, 7(4), 29–39.

    Google Scholar 

  • Downey, R., Farhady, H., Present-Thomas, R., Suzuki, M., & Van Moere, A. (2008). Evaluation of the usefulness of the versant for English test: A response. Language Assessment Quarterly, 5(2), 160–167.

    Google Scholar 

  • Duarte, M., Shiplacoff, D., Simon, G., & Raju, R. (2012). U.S. Patent No. 8,291,348. Washington, DC: U.S. Patent and Trademark Office.

    Google Scholar 

  • Eagly, A. H., & Chaiken, S. (1993). The psychology of attitudes. San Diego: Harcourt Brace Jovanovich College Publishers.

    Google Scholar 

  • Elder, C., Iwashita, N., & McNamara, T. (2002). Estimating the difficulty of oral proficiency tasks: what does the test-taker have to offer? Language Testing, 19(4), 347–368.

    Google Scholar 

  • Fan, J., & JI, P. (2014). Test candidates' attitudes and their test performance: The case of the Fudan English Test. University of Sydney Papers in TESOL, 9(1–35).

  • Freedman, R. A., Klein, R. A., & Lederman, J. (1994). Artificial intelligence in the capital markets: State-of-the-art applications for institutional investors, bankers and traders. New York: McGraw-Hill Publishing.

  • Gartner Group (2016). Gartner’s top 10 strategic technology trends for 2017. Retreived on the 20th of 2019 from http://www.gartner.com/smarterwithgartner/gartners-top-10-technology-trends-2017/.

  • Ghahramani, Z. (2015). Probabilistic machine learning and artificial intelligence. Nature, 521(7553), 452–459.

    Google Scholar 

  • Ginther, A., & McIntosh, K. (2018). Language testing and assessment. In The Palgrave handbook of applied linguistics research methodology (pp. 845–867). London: Palgrave Macmillan.

    Google Scholar 

  • Giroux, H. (2002). Neoliberalism, corporate culture, and the promise of higher education: The university as a democratic public sphere. Harvard Educational Review, 72(4), 425–464.

    Google Scholar 

  • Giroux, H. A. (2020). On critical pedagogy. London: Bloomsbury Publishing.

    Google Scholar 

  • Goldberg, Y. (2017). Neural network methods for natural language processing. Synthesis Lectures on Human Language Technologies, 10(1), 1–309.

    Google Scholar 

  • Greene, D., Hoffmann, A. L., & Stark, L. (2019). Better, nicer, clearer, fairer: A critical assessment of the movement for ethical artificial intelligence and machine learning. In Proceedings of the 52nd Hawaii International Conference on System Sciences.

  • Hausser, R. (2012). Computation of language: An essay on syntax, semantics and pragmatics in natural man-machine communication. Berlin: Springer-Verlag Publications.

  • Heckman, J. J., & Rubinstein, Y. (2001). The importance of noncognitive skills: Lessons from the GED testing program. American Economic Review, 91(2), 145–149.

    Google Scholar 

  • Hirschberg, J., & Manning, C. D. (2015). Advances in natural language processing. Science, 349(6245), –261, 266.

  • Jones, M. (1985). Applications of artificial intelligence within education. Computers & Mathematics with Applications, 11(5), 517–526.

    Google Scholar 

  • Kang, O., Thomson, R., & Moran, M. (2019). The effects of international accents and shared first language on listening comprehension tests. TESOL Quarterly, 53(1), 56–81.

    Google Scholar 

  • Kasabov, N. (2015). Evolving connectionist systems: From Neuro-fuzzy-, to spiking-and Neuro-genetic. In Springer handbook of computational intelligence (pp. 771–782). Berlin, Heidelberg: Springer.

    Google Scholar 

  • Keerthiwansha, N. B. S. (2018). Artificial intelligence education (AIEd) in English as a second language (ESL) classroom in Sri Lanka. Artificial Intelligence, 6(1), 31–36.

    Google Scholar 

  • Kim, N. Y. (2019). A study on the use of artificial intelligence Chatbots for improving English grammar skills. Journal of Digital Convergence, 17(8), 37–46.

    Google Scholar 

  • Kovács, G., & Kot, S. (2016). New logistics and production trends as the effect of global economy changes. Polish Journal of Management Studies, 14, 115–126.

    Google Scholar 

  • Landauer, T. K. (2003). Automatic essay assessment. Assessment in education: Principles, policy & practice, 10(3), 295–308.

    Google Scholar 

  • Letier, E., & Van Lamsweerde, A. (2002). Deriving operational software specifications from system goals. ACM SIGSOFT Software Engineering Notes, 27(6), 119–128.

    Google Scholar 

  • Lu, X. (2019). An empirical study on the artificial intelligence writing evaluation system in China CET. Big Data, 7(2), 121–129.

    MathSciNet  Google Scholar 

  • Luckin, R. (2017). Towards artificial intelligence-based assessment systems. Nature Human Behaviour, 1(28), 1–3.

  • Ma, Z. (2019). Application and practice of artificial intelligence in maker education and teaching. Advances in Higher Education, 3(2), 105–108.

    Google Scholar 

  • McTear, M. F. (2016). The rise of the conversational interface: A new kid on the block? In International workshop on future and emerging trends in language technology (pp. 38–49). Cham: Springer.

    Google Scholar 

  • Nichols, J., Myers, B. A., & Litwack, K. (2004). Improving automatic interface generation with smart templates. In Proceedings of the 9th international conference on intelligent user interfaces (pp. 286-288). ACM.

  • Nilsson, N. J. (2014). Principles of artificial intelligence. Massachusetts: Morgan Kaufmann Publications.

  • Ockey, G. J. (2009). Developments and challenges in the use of computer-based testing for assessing second language ability. The Modern Language Journal, 93, 836–847.

    Google Scholar 

  • Ordem, E. (2017). Participatory approach in EFL classes. Electronic Turkish Studies, 12(3), 505–516.

    Google Scholar 

  • Ordem, E., & Yükselir, C. (2017). Views of Turkish EFL instructors on critical pedagogy. Electronic Turkish Studies, 12(14), 285–294.

    Google Scholar 

  • Patton, M. Q. (2002). Two decades of developments in qualitative inquiry: A personal, experiential perspective. Qualitative Social Work, 1(3), 261–283.

    Google Scholar 

  • Plonsky, L. (2015). Quantitative considerations for improving Replicability in CALL and applied linguistics. CALICO Journal, 32(2), 232–244.

    Google Scholar 

  • Pollach, I. (2012). Taming textual data: The contribution of corpus linguistics to computer-aided text analysis. Organizational Research Methods, 15(2), 263–287.

    Google Scholar 

  • Poorsoti, S., & Asl, H. D. (2015). Iranian Candidates' attitudes toward TOEFL iBT. Journal of Applied Linguistics and Language Research, 2(8), 263–272.

    Google Scholar 

  • Quesada, J. F., Mateos, F. J. M., & Soto, T. L. (Eds.). (2017). Future and emerging trends in language technology. Machine learning and big data: Second International Workshop, FETLT 2016, Seville, Spain, November 30–December 2, 2016, revised selected papers (Vol. 10341). Springer.

  • Reiter, E., & Dale, R. (1997). Building applied natural language generation systems. Natural Language Engineering, 3(1), 57–87.

    Google Scholar 

  • Reiter, E., & Dale, R. (2000). Building natural language generation systems. Cambridge: Cambridge University Press.

    Google Scholar 

  • Riggio, R. E. (1986). Assessment of basic social skills. Journal of Personality and Social Psychology, 51(3), 649.

    Google Scholar 

  • Russell, S. J., & Norvig, P. (2016). Artificial intelligence: a modern approach. New York: Pearson Education Limited.

  • Sang, E. F. T. K., & Nerbonne, J. (1999, June). Learning the logic of simple phonotactics. In International conference on learning language in logic (pp. 110–124). Springer, Berlin, Heidelberg.

  • Shin, M. H. (2018). How to use artificial intelligence in the English language learning classroom. Indian Journal of Public Health Research & Development, 9(9), 557–561.

    Google Scholar 

  • Sproat, R. W. (1997). Multilingual text-to-speech synthesis. New York: KLUWER academic publishers.

    Google Scholar 

  • Underwood, J. H. (1984). Linguistics, computers, and the language teacher. A communicative approach. New York: Newbury House Publishers.

  • Underwood, J. (2017). Exploring AI language assistants with primary EFL students. CALL in a climate of change: Adapting to turbulent global conditions–short papers from EUROCALL, 317-321.

  • Van der Spek, R., & Spijkervet, A. (1997). Knowledge management: dealing intelligently with Knowledge. In J. Liebowitz & L. C. Wilcox (Eds.), Knowledge management and its integrative elements (pp. 31-58). NewYork: CRC Press.

  • Vicente, M., & Lloret, E. (2016). Exploring flexibility in natural language generation through discursive analysis of new textual genres. In International workshop on future and emerging trends in language technology (pp. 98–109). Cham: Springer.

    Google Scholar 

  • Wacker, P., & Joscelyne, A. (2016). Improving collaboration between the European language technology industry and research: A new framework for supply and demand. In international workshop on future and emerging trends in language technology (pp. 61-69). Springer, Cham.

  • Wu, C. W. (2009). Decision-making in testing process performance with fuzzy data. European Journal of Operational Research, 193(2), 499–509.

    MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. English Language Teaching Department, Education Faculty, Mersin University, Mersin, Turkey

    Ömer Gökhan Ulum

Authors
  1. Ömer Gökhan Ulum
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ömer Gökhan Ulum.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ulum, Ö.G. A critical deconstruction of computer-based test application in Turkish State University. Educ Inf Technol 25, 4883–4896 (2020). https://doi.org/10.1007/s10639-020-10199-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10639-020-10199-z

Keywords

Search

Navigation