Skip to main content

Advertisement

SpringerLink
Log in

A systematized review of research with adaptive comparative judgment (ACJ) in higher education

  • Published:
International Journal of Technology and Design Education Aims and scope Submit manuscript

Abstract

Adaptive Comparative Judgment (ACJ), an approach to the assessment of open-ended problems which utilizes a series of comparisons to produce a standardized score, rank order, and a variety of other statistical measures, has demonstrated high levels of reliability and validity and the potential for application in a wide variety of areas. Further, research into using ACJ, both as a formative and summative assessment tool, has been conducted in multiple contexts across higher education. This systematized review of ACJ research outlines our approach to identifying, classifying, and organizing findings from research with ACJ in higher education settings as well as overarching themes and questions that remain. The intent of this work is to provide readers with an understanding of the current state of the field and several areas of potential further inquiry related to ACJ in higher education.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  • Akister, J., Bannon, A., & Mullender-Lock, H. (2000). Poster presentations in social work education assessment: A case study. Innovations in Education and Training International, 37(3), 229–233.

    Google Scholar 

  • Baartman, L. K., Bastiaens, T. J., Kirschner, P. A., & Van der Vleuten, C. P. (2007). Evaluating assessment quality in competence-based education: A qualitative comparison of two frameworks. Educational Research Review, 2(2), 114–129.

    Google Scholar 

  • Barber, J. (2018). Five go marking an exam question: The use of adaptive comparative judgement to manage subjective bias. Practitioner Research in Higher Education, 11(1), 94–100.

    Google Scholar 

  • Bartholomew, S. R. (2017). Assessing open-ended design problems. Technology and Engineering Education Teacher, 76(6), 13–17.

    Google Scholar 

  • Bartholomew, S. R., Mentzer, N., & Jones, M. (2019a). Learning by evaluating (LbE). In Mississippi Valley Technology Education Conference. Nashville, TN.

  • Bartholomew, S. R., Strimel, G. J., & Jackson, A. (2018a). A comparison of traditional and adaptive comparative judgment assessment techniques for freshman engineering design projects. International Journal of Engineering Education, 34(1), 20–33.

    Google Scholar 

  • Bartholomew, S. R., & Yoshikawa, E. (2018). A systematic review of research around adaptive comparative judgment (ACJ) in K-16 education. 2018 CTETE Monograph Series. Retrieved from https://doi.org/10.21061/ctete-rms.v1.c.1.

  • Bartholomew, S. R., Yoshikawa, E., & Connolly, P. E. (2018b). Exploring the potential for identifying student competencies in design education through adaptive comparative judgment. In PATT35 Athlone Institute of Technology, Athlone, Ireland 18–21 June, 2018 (pp. 187–194).

  • Bartholomew, S., Zhang, L., Bravo, E. G., & Strimel, G. J. (2019b). A tool for formative assessment and learning in a graphics design course: Adaptive comparative judgement. The Design Journal, 22(1), 73–95.

    Google Scholar 

  • Bartholomew, S., Zhang, L., Bravo, E. G., & Strimel, G. J. (2019c). First, last, elsewhere… positioning adaptive comparative judgment in the design learning experience. In PATT 37 University of Malta, Malta, June 2–6, 2019 (pp. 65–74).

  • Bell, S. (2010). Project-based learning for the 21st century: Skills for the future. The Clearing House, 83(2), 39–43.

    Google Scholar 

  • Benton, T., & Gallagher, T. (2018). Is comparative judgement just a quick form of multiple marking? Research Matters: A Cambridge Assessment Publication, 26, 22–28.

    Google Scholar 

  • Borrego, M., Foster, M. J., & Froyd, J. E. (2014). Systematic literature reviews in engineering education and other developing interdisciplinary fields. Journal of Engineering Education, 103(1), 45–76.

    Google Scholar 

  • Boud, D., & Feletti, G. (2013). The challenge of problem-based learning. London: Routledge.

    Google Scholar 

  • Bramley, T. (2015). Investigating the reliability of adaptive comparative judgment (p. 36). Cambridge: Cambridge Assessment.

    Google Scholar 

  • Bramley, T., & Vitello, S. (2018). The effect of adaptivity on the reliability coefficient in adaptive comparative judgement. Assessment in Education: Principles, Policy & Practice, 26(1), 43–58.

    Google Scholar 

  • Buckley, J., Canty, D., & Seery, N. (2020). An exploration into the criteria used in assessing design activities with adaptive comparative judgment in technology education. Irish Educational Studies. https://doi.org/10.1080/03323315.2020.1814838.

  • Canty, D. (2012). The impact of holistic assessment using adaptive comparative judgment of student learning. PhD Thesis, University of Limerick, Ireland.

  • Canty, D., Buckley, J., & Seery, N. (2019). Inducting ITE students in assessment practices through the use of comparative judgment. In PATT 37 University of Malta, Malta, June 2–6, 2019 (pp. 117–124).

  • Canty, D., Seery, N., Hartell, E., & Doyle, A. (2017). Integrating peer assessment in technology education through adaptive comparative judgment. In PATT34 Technology & Engineering Education–Fostering the Creativity of Youth Around the Globe, Millersville University, Pennsylvania, USA (pp. 10–14).

  • Demonacos, C., Ellis, S., & Barber, J. (2019). Student peer assessment using adaptive comparative judgment: Grading accuracy versus quality of feedback. Practitioner Research in Higher Education, 12(1), 50–59.

    Google Scholar 

  • Dobson, S. (2006). The assessment of student PowerPoint presentations—Attempting the impossible? Assessment & Evaluation in Higher Education, 31(1), 109–119.

    Google Scholar 

  • Duran, M., & Dökme, İ. (2016). The effect of the inquiry-based learning approach on student’s critical-thinking skills. Eurasia Journal of Mathematics, Science & Technology Education, 12(12), 2887–2908.

    Google Scholar 

  • Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26(2), 91–108.

    Google Scholar 

  • Hannafin, M. J., Hall, C., Land, S., & Hill, J. (1994). Learning in open-ended environments: Assumptions, methods, and implications. Educational Technology, 34(8), 48–55.

    Google Scholar 

  • Jones, I., & Alcock, L. (2012). Summative peer assessment of undergraduate calculus using adaptive comparative judgement. In P. Iannone & A. Simpson (Eds.), Mapping university mathematics assessment practices. Norwich: University of East Anglia.

    Google Scholar 

  • Kimbell, R. (2007). E-assessment in project e-scape. Design & Technology Education: An International Journal, 12(2), 66–76.

    Google Scholar 

  • Kimbell, R. (2012a). Evolving project e-scape for national assessment. International Journal of Technology & Design Education, 22, 135–155.

    Google Scholar 

  • Kimbell, R. (2012b). The origins and underpinning principles of e-scape. International Journal of Technology & Design Education, 22, 123–134.

    Google Scholar 

  • Kirschner, P., Sweller, J., & Clark, R. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75–86.

    Google Scholar 

  • Kumar, M., & Natarajan, U. (2007). Alternative assessment in problem-based learning: Strengths, shortcomings and sustainability. i-Manager’s Journal on Educational Psychology, 1(1), 27.

    Google Scholar 

  • Lazonder, A. W., & Harmsen, R. (2016). Meta-analysis of inquiry-based learning: Effects of guidance. Review of Educational Research, 86(3), 681–718.

    Google Scholar 

  • Martin-Martin, A., Orduña-Malea, E., Harzing, A. W., & López-Cózar, E. D. (2017). Can we use Google Scholar to identify highly-cited documents? Journal of Informetrics, 11(1), 152–163.

    Google Scholar 

  • Metzgar, M. (2016). Using adaptive comparative judgement to assess student work in an MBA course. International Journal for Infonomics, 9(3), 1217–1219.

    Google Scholar 

  • Mills, J. E., & Treagust, D. F. (2003). Engineering education—Is problem-based or project-based learning the answer. Australasian Journal of Engineering Education, 3(2), 2–16.

    Google Scholar 

  • Moskal, B. M., Leydens, J. A., & Pavelich, M. J. (2002). Validity, reliability and the assessment of engineering education. Journal of Engineering Education, 91(3), 351–354.

    Google Scholar 

  • Munroe, L. (2015). The open-ended approach framework. European Journal of Educational Research, 4(3), 97–104.

    Google Scholar 

  • Newhouse, P. (2011). Comparative pairs marking supports authentic assessment of practical performance within constructivist learning environments. In Applications of Rasch measurement in learning environments research (pp. 141–180). Rotterdam: Sense Publishers.

    Google Scholar 

  • Noruzi, A. (2005). Google Scholar: The new generation of citation indexes. Libri, 55(4), 170–180.

    Google Scholar 

  • Pollitt, A. (2004). Let’s stop marking exams. Retrieved July 23, 2018, from http://www.cambridgeassessment.org.uk/images/109719-let-s-stop-marking-exams.pdf.

  • Pollitt, A. (2012). The method of adaptive comparative judgment. Assessment in Education: Principles, Policy & Practice, 19(3), 281–300.

    Google Scholar 

  • Pollitt, A. (2015). On ‘reliability’ bias in ACJ. Cambridge Exam Research. Retrieved February 2, 2018, from https://www.researchgate.net/publication/283318012_On_‘Reliability’_bias_in_ACJ.

  • Pollitt, A., & Murray, N. L. (1996). What raters really pay attention to. Studies in Language Testing, 3, 74–91.

    Google Scholar 

  • Potter, T., Englund, L., Charbonneau, J., MacLean, M. T., Newell, J., & Roll, I. (2017). ComPAIR: A new online tool using adaptive comparative judgement to support learning with peer feedback. Teaching & Learning Inquiry, 5(2), 89–113.

    Google Scholar 

  • Purzer, S., Fila, N., & Nataraja, K. (2016). Evaluation of current assessment methods in engineering entrepreneurship education. Advances in Engineering Education, 5(1), n1.

    Google Scholar 

  • Rangel-Smith, C., & Lynch, D. (2018). Addressing the issue of bias in the measurement of reliability in the method of adaptive comparative judgment. In PATT35 Athlone Institute of Technology, Athlone, Ireland 18–21 June, 2018 (pp. 378–387).

  • Rhind, S. M., Hughes, K. J., Yool, D., Shaw, D., Kerr, W., & Reed, N. (2017). Adaptive comparative judgment: A tool to support students’ assessment literacy. Journal of Veterinary Medical Education, 44(4), 686–691.

    Google Scholar 

  • Rowsome, P., Seery, N., & Lane, D. (2013). The development of pre-service design educator’s capacity to make professional judgments on design capability using adaptive comparative judgment. American Society for Engineering Education.

  • Seery, N., Buckley, J., Delahunty, T., & Canty, D. (2018). Integrating learners into the assessment process using adaptive comparative judgement with an ipsative approach to identifying competence based gains relative to student ability levels. International Journal of Technology and Design Education, 29, 701–715. https://doi.org/10.1007/s10798-018-9468-x.

    Article  Google Scholar 

  • Seery, N., Buckley, J., Doyle, A., & Canty, D. (2016). The validity and reliability of adaptive comparative judgements in the assessment of graphical capability. In Proceedings of the 71st Mid-Year Conference of the Engineering Design Graphics Division (pp. 104–109).

  • Seery, N., Canty, D., & Phelan, P. (2012). The validity and value of peer assessment using adaptive comparative judgement in design driven practical education. International Journal of Technology and Design Education, 22(2), 205–226.

    Google Scholar 

  • Seery, N., Lane, D., & Canty, D. (2011). Exploring the value of democratic assessment in design based activities of graphical education. In 118th Annual American Society of Engineering Education Conference. Vancouver, BC: American Society for Engineering Education.

    Google Scholar 

  • Steedle, J. T., & Ferrara, S. (2016). Evaluating comparative judgment as an approach to essay scoring. Applied Measurement in Education, 29(3), 211–223.

    Google Scholar 

  • Stevens, D. D., & Levi, A. J. (2013). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning. Sterling, VA: Stylus Publishing, LLC.

    Google Scholar 

  • Strimel, G. J., Bartholomew, S. R., Purzer, S., Yoshikawa, E., & Zhang, L. (2020). Informing engineering design through adaptive comparative judgment. European Journal of Engineering Education. https://doi.org/10.1080/03043797.2020.1718614.

  • Thomas, J. W. (2000). A review of research on project-based learning. Autodesk Foundation. Retrieved from https://www.asec.purdue.edu/lct/HBCU/documents/AReviewofResearchofProject-BasedLearning.pdf.

  • Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34, 273–286.

    Google Scholar 

  • Verhavert, S., Bouwer, R., Donche, V., & De Maeyer, S. (2019). A meta-analysis on the reliability of comparative judgement. Assessment in Education: Principles, Policy & Practice, 26(5), 541–562.

    Google Scholar 

  • Viseu, F., & Oliveira, I. B. (2017). Open-ended tasks in the promotion of classroom communication in mathematics. International Electronic Journal of Elementary Education, 4(2), 287–300.

    Google Scholar 

  • Zhang, L. (2019). Investigating differences in formative critiquing between instructors and students in graphic design. Doctoral dissertation, Purdue University Libraries.

  • Zientek, L. R., Werner, J. M., Campuzano, M. V., & Nimon, K. (2018). The use of Google Scholar for research and research dissemination. New Horizons in Adult Education and Human Resource Development, 30(1), 39–46.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Brigham Young University, 230 Snell, BYU, Provo, UT, 84602, USA

    Scott R. Bartholomew

  2. Purdue University, Young Hall, West Lafayette, IN, 47907, USA

    Matthew D. Jones

Authors
  1. Scott R. Bartholomew
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Matthew D. Jones
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Scott R. Bartholomew.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bartholomew, S.R., Jones, M.D. A systematized review of research with adaptive comparative judgment (ACJ) in higher education. Int J Technol Des Educ 32, 1159–1190 (2022). https://doi.org/10.1007/s10798-020-09642-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10798-020-09642-6

Keywords

Search

Navigation