Skip to main content
Log in

Learning by evaluating (LbE) through adaptive comparative judgment

  • Published:
International Journal of Technology and Design Education Aims and scope Submit manuscript

Abstract

Traditional efforts around improving assessment often center on the teacher as the evaluator of work rather than the students. These assessment efforts typically focus on measuring learning rather than stimulating, promoting, or producing learning in students. This paper summarizes a study of a large sample of undergraduate students (n = 550) in an entry-level design-thinking course who engaged with Adaptive Comparative Judgment (ACJ), a form of assessment, as a learning mechanism. Following random assignment into control and treatment sections, students engaged in identical activities with the exception of a 20-minute intervention we call learning by evaluating (LbE). Prior to engaging in a Point Of View (POV) creation activity, treatment group students engaged in LbE by viewing pairs of previously-collected POV statements through ACJ; in each case they viewed two POV statements side-by-side and selected the POV statement they believed was better. Following this experience, students created their own POV statements and then the final POV statements, from both the control and treatment students, were collected and evaluated by instructors using ACJ. In addition, qualitative data consisting of student comments, collected during ACJ comparisons, were coded by the researchers to further explore the potential for the students to use class knowledge while engaging in the LbE review of peer work. Both the quantitative and qualitative data sets were analyzed to investigate the impact of the LbE activity. Consistent with other ACJ research findings, significant positive learning gains were found for students who engaged in the intervention. Researchers also noted that these findings did not indicate the actual quality of the assignments, meaning the while students who engaged in the LbE intervention were better than their peers, they were not necessarily “good” at the assignment themselves. Discussion of these findings and areas for further inquiry are presented.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  • Baniya, S., Chesley, A., Mentzer, N., Bartholomew, S., Moon, C., & Sherman, D. (2019). Using adaptive comparative judgment in writing assessment: An investigation of reliability among interdisciplinary evaluators. Journal of Technology Studies, 45(2), 24–45.

    Article  Google Scholar 

  • Bartholomew, S. R. (2017). Assessing open-ended design problems. Technology and Engineering Education Teacher, 76(6), 13–17.

    Google Scholar 

  • Bartholomew, S. R. (2019). Teacher led assessment of learning to student led assessment as learning. Beyond multiple choice conference, Washington DC.

  • Bartholomew, S. R., & Jones, M. D. (2020). A systematic review of research of adaptive comparative judgement (ACJ) in higher education. International Journal of Technology and Design Education. https://doi.org/10.21061/ctete-rms.v1.c.1

  • Bartholomew, S. R., Strimel, G. J., Garcia Bravo, E., Zhang, L., & Yoshikawa, E. (2018). Formative feedback for improved student performance through adaptive comparative judgment. Paper presented at the paper presented at the 125th ASEE conference, Salt Lake City, Utah.

  • Bartholomew, S. R., Strimel, G. S., & Yoshikawa, E. (2018b). Using adaptive comparative judgment for student formative feedback and learning during a middle school open- ended design challenge. International Journal of Technology and Design Education, 29(2), 363–385.

    Article  Google Scholar 

  • Bartholomew, S. R., & Yoshikawa, E. (2018). A systematic review of research around Adaptive Comparative Judgment (ACJ) in K-16 education. 2018 CTETE Monograph Series. https://doi.org/10.21061/ctete-rms.v1.c.1.

  • Bartholomew, S. R., Zhang, L., Garcia Bravo, E., & Strimel, G. J. (2019). A tool for formative assessment and learning in a graphics design course: adaptive comparative judgement. The Design Journal, 22(1), 73–95.

    Article  Google Scholar 

  • Black, P., & Wiliam, D. (2005). Inside the black box: Raising standards through classroom assessment. Granada Learning.

  • Black, P., & Wiliam, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability (formerly: Journal of Personnel Evaluation in Education), 21(1), 5–31.

  • Black, P., & Wiliam, D. (2018). Classroom assessment and pedagogy. Assessment in Education: Principles, Policy & Practice, 25(6), 551–575.

    Google Scholar 

  • Box, C. (2019). Formative assessment in United States classrooms. Springer Science + Business Media.

  • Bramley, T. (2015). Investigating the reliability of Adaptive Comparative Judgment. Cambridge Assessment, Cambridge, 36

  • Brown, S. (2014). Learning, teaching and assessment in higher education: Global perspectives. Macmillan International Higher Education.

  • Canty, D. (2012). The impact of holistic assessment using adaptive comparative judgment of student learning. Limerick: University of Limerick.

    Google Scholar 

  • Denson, C. D., Buelin, J. K., Lammi, M. D., & D’Amico, S. (2015). Developing instrumentation for assessing creativity in engineering design. Journal of Technology Education, 27(1), 23–40.

    Google Scholar 

  • Ellis, A. K. (2017). Teaching, learning, and assessment together: Reflective assessments for elementary classrooms. Routledge.

  • Gielen, M., & de Wever, B. (2015). Structuring peer assessment: Comparing the impact of the degree of structure on peer feedback content. Computers in Human Behavior, 52, 315–325.

    Article  Google Scholar 

  • Giloi, S., & du Toit, P. (2013). Current approaches to the assessment of graphic design in a higher education. International Journal of Art & Design Education, 32(2), 256–268.

    Article  Google Scholar 

  • Hamer, J., Purchase, H., Luxton-Reilly, A., & Denny, P. (2015). A comparison of peer and tutor feedback. Assessment & Evaluation in Higher Education, 40(1), 151–164.

    Article  Google Scholar 

  • Jones, I., & Alcock, L. (2014). Peer assessment without assessment criteria. Studies in Higher Education, 39(10), 1774–1787.

    Article  Google Scholar 

  • Jurado, J. A. (2011). Group projects in interior design studio classes: Peer feedback benefits. Journal of Family and Consumer Sciences, 103(1), 34.

    Google Scholar 

  • Kimbell, R. (2007). E-assessment in project e-scape. Design & Technology Education: An International Journal, 12(2), 66–76.

    Google Scholar 

  • Kimbell, R. (2012a). Evolving project e-scape for national assessment. International Journal of Technology and Design Education, 22(2), 135–155.

    Article  Google Scholar 

  • Kimbell, R. (2012b). The origins and underpinning principles of e-scape. International Journal of Technology and Design Education, 22, 123–124.

    Article  Google Scholar 

  • Kimbell, R. (2018). Constructs of quality and the power of holism. Paper presented at the PATT35, Athlone Institute of Technology, Athlone, Ireland.

  • Lau, A. M. S. (2016). ‘Formative good, summative bad?’—A review of the dichotomy in assessment literature. Journal of Further and Higher Education, 40(4), 509–525.

    Article  Google Scholar 

  • Li, L., & Gao, F. (2016). The effect of peer assessment on project performance of students at different learning levels. Assessment & Evaluation in Higher Education, 41(6), 885–900.

    Article  Google Scholar 

  • Li, L., Liu, X., & Steckelberg, A. L. (2010). Assessor or assessee: How student learning improves by giving and receiving peer feedback. British Journal of Educational Technology, 41(3), 525–536.

    Article  Google Scholar 

  • López-Pastor, V., & Sicilia-Camacho, A. (2017). Formative and shared assessment in higher education. Lessons learned and challenges for the future. Assessment & Evaluation in Higher Education, 42(1), 77–97.

    Article  Google Scholar 

  • McCarthy, J. (2017). Enhancing feedback in higher education: Students’ attitudes towards online and in-class formative assessment feedback models. Active Learning in Higher Education, 18(2), 127–141.

    Article  Google Scholar 

  • McGarr, O., & Clifford, A. M. (2013). ‘Just enough to make you take it seriously’: Exploring students’ attitudes towards peer assessment. Higher Education, 65(6), 677–693.

    Article  Google Scholar 

  • Mehalik, M. M., Doppelt, Y., & Schuun, C. D. (2008). Middle-school science through design-based learning versus scripted inquiry: Better overall science concept learning and equity gap reduction. Journal of Engineering Education, 97(1), 71–85.

    Article  Google Scholar 

  • Nicol, D., Thomson, A., & Breslin, C. (2014). Rethinking feedback practices in higher education: A peer review perspective. Assessment & Evaluation in Higher Education, 39(1), 102–122.

    Article  Google Scholar 

  • Penuel, W. R., Harris, C. J., & DeBarger, A. H. (2015). Implementing the next generation science standards. Phi Delta Kappan, 96(6), 45–49.

    Article  Google Scholar 

  • Pollitt, A. (2004). Let’s stop marking exams. Retrieved from http://www.cambridgeassessment.org.uk/images/109719-let-s-stop-marking-exams.pdf.

  • Pollitt, A. (2012). The method of adaptive comparative judgement. Assessment in Education: Principles, Policy & Practice, 19(3), 281–300.

    Google Scholar 

  • Pollitt, A. (2015). On ‘Reliability’ bias in ACJ. Cambridge Exam Research. Retrieved 23, April 2020 from https://www.researchgate.net/publication/283318012_On_’Reliability’_bias_in_ACJ.

  • Potter, T., Englund, L., Charbonneau, J., MacLean, M. T., Newell, J., & Roll, I. (2017). ComPAIR: A new online tool using adaptive comparative judgement to support learning with peer feedback. Teaching & Learning Inquiry, 5(2), 89–113.

    Article  Google Scholar 

  • Rangel-Smith, C., & Lynch, D. (2018). Addressing the issue of bias in the measurement of reliability in the method of Adaptive Comparative Judgment. In 36th pupils’ attitudes towards technology conference, Athlone, Ireland, pp. 378–387.

  • Saldaña, J. (2015). The coding manual for qualitative Researchers. Thousand Oaks, CA: SAGE.

  • Seery, N., & Canty, D (2017). Assessment and learning: The proximal and distal effects of comparative judgment. Handbook of Technology Education, Springer. DOI: 10.1007/978-3-319-38889-2_54-1.

  • Sluijsmans, D., Dochy, F., & Moerkerke, G. (1998). Creating a learning environment by using self-, peer- and co-assessment. Learning Environments Research, 1(3), 293–319.

    Article  Google Scholar 

  • Stefani, L. A. J. (1994). Peer, self and tutor assessment: Relative reliabilities. Studies in Higher Education, 19, 69–75.

    Article  Google Scholar 

  • Strimel, G. J., Bartholomew, S. R., Purzer, S., Zhang, L., & Ruesch, E. Y. (2020). Informing engineering design through adaptive comparative judgment. European Journal of Engineering Education. https://doi.org/10.1080/03043797.2020.1718614.

    Article  Google Scholar 

  • Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34, 273–286.

    Article  Google Scholar 

  • Verhavert, S., Bouwer, R., Donche, V., & De Maeyer, S. (2019). A meta-analysis on the reliability of comparative judgement. Assessment in Education: Principles, Policy & Practice, 26(5), 541–562.

    Google Scholar 

  • Walker, M. (2015). The quality of written peer feedback on undergraduates’ draft answers to an assignment, and the use made of the feedback. Assessment & Evaluation in Higher Education, 40(2), 232–247.

    Article  Google Scholar 

  • Wells, J. G. (2016). PIRPOSAL Model of Integrative STEM Education: Conceptual and Pedagogical Framework for Classroom Implementation. Technology and Engineering Teacher, 75(6), 12–19.

    Google Scholar 

  • Zhang, L. (2019). Investigating differences in formative critiquing between instructors and students in graphic design. Doctoral dissertation, Purdue University Libraries.

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Scott R. Bartholomew.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bartholomew, S.R., Mentzer, N., Jones, M. et al. Learning by evaluating (LbE) through adaptive comparative judgment. Int J Technol Des Educ 32, 1191–1205 (2022). https://doi.org/10.1007/s10798-020-09639-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10798-020-09639-1

Keywords

Navigation