Skip to main content
SpringerLink
Log in

Development and Validation of a Cognitive Diagnostic Assessment with Ordered Multiple-Choice Items for Addition of Time

  • Published:
International Journal of Science and Mathematics Education Aims and scope Submit manuscript

Abstract

Cognitive Diagnostic Assessment (CDA) is an alternative assessment which can give a clear picture of pupils’ learning process and cognitive structures to education stakeholders so that appropriate instructional strategies can be designed to tailored pupils’ needs. Coincide with this function, the Ordered Multiple-Choice (OMC) items were incorporated into the CDA developed in this study. This paper describes the process of developing and validating a Cognitive Diagnostic Assessment with OMC items for assessing pupils’ attribute mastery on ‘Addition of Time’ and reports the psychometric properties of the items, the model-data fit, the attribute reliability and the overall reliability of the assessment. The sample of the study consisted of 30 Year Four National School (SK) pupils, 48 Year Four National-Type Chinese School (SJKC) pupils and 12 Year Four National-Type Tamil School (SJKT) pupils in Penang state, Malaysia. The data was analysed using two measurement models, namely Classical Test Theory and Attribute Hierarchy Method (AHM). The findings of the study indicated that the instrument developed consisted of good-quality OMC items with appropriate difficulty and high discrimination power. With the satisfactory model-data fit in AHM, the inference made about pupils’ attributes mastery based on their performance in CDA with OMC items was valid. The CDA with OMC items developed in this study was found to be reliable at both attribute level and assessment level. Perhaps, this instrument could be used in the mathematics classrooms for supporting teachers in diagnosing pupils’ cognitive strengths and weaknesses for ‘Addition of Time’.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Alonzo, A. C., & Steedle, J. T. (2009). Developing and assessing a force and motion learning progression. Science Education, 93(3), 389–421.

    Article  Google Scholar 

  • Alves, C. B. (2012). Making diagnostic inferences about student performance on the Alberta education diagnostic mathematics project: An application of the Attribute Hierarchy Method. (Doctoral Thesis), University of Alberta, Ann Arbor, Canada.

  • Bardhoshi, G., & Erford, B. T. (2017). Processes and procedures for estimating score reliability and precision. Measurement and Evaluation in Counseling and Development, 50(4), 256–263.

    Article  Google Scholar 

  • Boora, S., Pasiphol, S., & Tangdhanakanond, K. (2015). Development of cognitive diagnostic testing on basic arithmetic operation. Procedia-Social and Behavioral Sciences, 191, 769–772.

    Article  Google Scholar 

  • Bradshaw, L. (2017). Diagnostic classification models. In A. A. Rupp & J. P. Leighton (Eds.), The handbook of cognition and assessment: Frameworks, methodologies, and applications (1st ed., pp. 297–327). Wiley Blackwell.

  • Briggs, D. C., & Alonzo, A. (2012). The psychometric modeling of ordered multiple-choice item responses for diagnostic assessment with a learning progression. In A. C. Alonzo & A. W. Gotwals (Eds.), Learning progressions in science: Current challenges and future directions (pp. 293–316). Sense Publishers.

  • Briggs, D. C., Alonzo, A., Schwab, C., & Wilson, M. (2006). Diagnostic assessment with ordered multiple-choice items. Educational Assessment, 11(1), 33–63.

    Article  Google Scholar 

  • Broaddus, A. E. (2011). An investigation into foundational concepts related to slope: An application of the Attribute Hierarchy Method. (Doctoral Thesis), University of Kansas, Kansas, United State. ProQuest Dissertations and Theses database. (UMI No: 3487353).

  • Burton, R. F. (2001). Quantifying the effects of chance in multiple choice and true/false tests: Question selection and guessing of answers. Assessment & Evaluation in Higher Education, 26(1), 41–50.

    Article  Google Scholar 

  • Clements, D. H., & Sarama, J. (2004). Learning trajectories in mathematics education. Mathematical Thinking and Learning, 6(2), 81–89.

    Article  Google Scholar 

  • Cui, Y., Gierl, M. J., & Guo, Q. (2017). The rule space and attribute hierarchy methods. In A. A. Rupp & J. P. Leighton (Eds.), The handbook of cognition and assessment: Frameworks, methodologies, and applications (1st ed., pp. 354–378). Wiley Blackwell.

  • Cui, Y., & Leighton, J. P. (2009). The hierarchy consistency index: Evaluating person fit for cognitive diagnostic assessment. Journal of Educational Measurement, 46(4), 429–449.

    Article  Google Scholar 

  • De Champlain, A. F. (2010). A primer on classical test theory and item response theory for assessments in medical education. Medical Education, 44(1), 109–117.

    Article  Google Scholar 

  • Downing, S. M. (2004). Reliability: On the reproducibility of assessment data. Medical Education, 38(9), 1006–1012.

    Article  Google Scholar 

  • Ebel, R. L., & Frisbie, D. A. (1991). Essentials of educational measurement (5th ed.). Englewood Cliffs: Prentice-Hall.

    Google Scholar 

  • Friedman, W. J. (1978). Development of time concepts in children. Advances in Child Development and Behavior, 12, 267–298. https://doi.org/10.1016/S0065-2407(08)60040-3.

    Article  Google Scholar 

  • Fulmer, G. W. (2015). Validating proposed learning progressions on force and motion using the force concept inventory: Findings from Singapore secondary schools. International Journal of Science and Mathematics Education, 13(6), 1235–1254.

    Article  Google Scholar 

  • Gay, L. R., Mills, G. E., & Airasian, P. W. (2012). Educational research: Competencies for analysis and applications (10th ed.). Merrill.

  • Ghazali, M., & Sinnakaudan, S. (2014). Reasearch on teachers’ beliefs about mathematics teaching and learning between Sekolah Kebangsaan (SK), Sekolah Jenis Kebangsaan Cina (SJKC) ans Sekolah Jenis Kebangsaan Tamil (SJKT). Journal of Education and Practice, 5(31), 10–19.

    Google Scholar 

  • Gierl, M. J., Alves, C., & Taylor-Majeau, R. (2010). Using the Attribute Hierarchy Method to make diagnostic inferences about examinees’ knowledge and skills in mathematics: An operational implementation of cognitive diagnostic assessment. International Journal of Testing, 10(4), 318–341. https://doi.org/10.1080/15305058.2010.509554.

    Article  Google Scholar 

  • Gierl, M. J., Cui, Y., & Zhou, J. (2009). Reliability and attribute-based scoring in cognitive diagnostic assessment. Journal of Educational Measurement, 46(3), 293–313.

    Article  Google Scholar 

  • Gierl, M. J., & Zhou, J. (2008). Computer adaptive-attribute testing: A new approach to cognitive diagnostic assessment. Zeitschrift für Psychologie/Journal of Psychology, 216(1), 29–39. https://doi.org/10.1027/0044-3409.216.1.29.

    Article  Google Scholar 

  • Hadenfeldt, J. C., Bernholt, S., Liu, X., Neumann, K., & Parchmann, I. (2013). Using ordered multiple-choice items to assess students’ understanding of the structure and composition of matter. Journal of Chemical Education, 90(12), 1602–1608.

    Article  Google Scholar 

  • Hadenfeldt, J. C., Neumann, K., Bernholt, S., Liu, X., & Parchmann, I. (2016). Students’ progression in understanding the matter concept. Journal of Research in Science Teaching, 53(5), 683–708.

    Article  Google Scholar 

  • Haladyna, T. M. (2004). Developing and validating multiple-choice test items (3rd ed.). Routledge.

  • Harris, S. (2008). It’s about time: Difficulties in developing time concepts. Australian Primary Mathematics Classroom, 13(1), 28–31.

    Google Scholar 

  • Harrison, M. L. (1934). The nature and development of concepts of time among young children. The Elementary School Journal, 34(7), 507–514.

    Article  Google Scholar 

  • Herrera, S. G., Murry, K. G., & Cabral, R. M. (2012). Assessment accommodations for classroom teachers of culturally and linguistically diverse students. Pearson Higher Ed.

  • Hinton, P. R., McMurray, I., & Brownlow, C. (2014). SPSS explained. Routledge.

  • Huff, K., Warner, Z., & Schweid, J. (2017). Large-scale standards-based assessments of educational achievement. In A. A. Rupp & J. P. Leighton (Eds.), The handbook of cognition and assessment: Frameworks, methodologies, and applications (1st ed., pp. 399–476). New York: Wiley Blackwell.

    Google Scholar 

  • Huntley, R., Welch, C. J. (1993, April). Numerical answer options: Logical or random order? Paper presented at the annual meeting of the American Educational Research Association, Atlanta, GA.

  • Kamii, C., & Russell, K. A. (2012). Elapsed time: Why is it so difficult to teach? Journal for Research in Mathematics Education, 43(3), 296–345.

    Article  Google Scholar 

  • Lai, H., Gierl, M. J., & Babenko, O. (2015). Application of conditional means for diagnostic scoring. International Journal of Learning, Teaching and Educational Research, 12(3), 61–79.

    Google Scholar 

  • Leighton, J. P., & Gierl, M. J. (2007). Why cognitive diagnostic assessment? In J. P. Leighton & M. J. Gierl (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 3–18). New York: Cambridge University Press.

    Chapter  Google Scholar 

  • Leighton, J. P., Gierl, M. J., & Hunka, S. M. (2004). The Attribute Hierarchy Method for cognitive assessment: A variation on Tatsuoka’s Rule-Space Approach. Journal of Educational Measurement, 41(3), 205–237.

    Article  Google Scholar 

  • Levin, I. (1989). Principles underlying time measurement: The development of children’s constraints on counting time. In I. Levin & D. Zakay (Eds.), Advances in psychology (Vol. 59, pp. 145–183). Elsevier.

  • Liljedahl,  P. (2019) The International Journal of Science and Mathematics Education: A Beginner’s Guide to Writing for Publication

  • Multon, K. D., & Coleman, J. S. M. (2010). Coefficient alpha. In N. Salkind (Ed.), Encyclopedia of research design (pp. 159–162). Sage.

  • National Council of Teachers of Mathematics. (2000). Principles and standards for school mathematics. NCTM.

  • Nichols, P. D., Kobrin, J. L., Lai, E., & Koepfler, J. D. (2017). The role of theories of learning and cognition in assessment design and development. In A. A. Rupp & J. P. Leighton (Eds.), The handbook of cognition and assessment: Frameworks, methodologies, and applications (1st ed., pp. 41–74). Wiley Blackwell.

  • Nylund, K. L., Asparouhov, T., & Muthén, B. O. (2007). Deciding on the number of classes in latent class analysis and growth mixture modeling: A Monte Carlo simulation study. Structural Equation Modeling: A Multidisciplinary Journal, 14(4), 535–569.

    Article  Google Scholar 

  • Oakden, E. C., & Sturt, M. (1922). The development of the knowledge of time in children. British Journal of Psychology, 12(4), 309–336.

    Google Scholar 

  • Ojose, B. (2015). Common misconceptions in mathematics: Strategies to correct them. University Press of America.

  • Pistor, F. (1939). Measuring the time concepts of children. The Journal of Educational Research, 33(4), 293–300.

    Article  Google Scholar 

  • Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what’s being reported? Critique and recommendations. Research in Nursing & Health, 29(5), 489–497.

    Article  Google Scholar 

  • Roberts, M. R., Alves, C. B., Chu, M. W., Thompson, M., Bahry, L. M., & Gotzmann, A. (2014). Testing expert based versus student based cognitive models for a Grade 3 diagnostic mathematics assessment. Applied Measurement in Education, 27(3), 173–195.

    Article  Google Scholar 

  • Salkind, N. (2010). Convenience sampling. In N. Salkind (Ed.), Encyclopedia of research design (p. 254). Sage.

  • Schecter, D. E., Symonds, M., & Bernstein, I. (1955). Development of the concept of time in children. Journal of Nervous and Mental Disease, 121, 301–310.

    Article  Google Scholar 

  • Schultz, M., Lawrie, G. A., Bailey, C. H., Bedford, S. B., Dargaville, T. R., O'Brien, G., Wright, A. H. (2017). Evaluation of diagnostic tools that tertiary teachers can apply to profile their students’ conceptions. International Journal of Science Education, 39(5), 565–586. https://doi.org/10.1080/09500693.2017.1296980.

  • Shute, V. J., Leighton, J. P., Jang, E. E., & Chu, M. W. (2016). Advances in the science of assessment. Educational Assessment, 21(1), 34–59.

    Article  Google Scholar 

  • Sia, C. J. L., & Lim, C. S. (2018). Cognitive diagnostic assessment: An alternative mode of assessment for learning. In D. R. Thompson, M. Burton, A. Cusi, & D. Wright (Eds.), Classroom assessment in mathematics (pp. 123–137). Springer.

  • Siegler, R. S., & McGilly, K. (1989). Strategy choices in children’s time-telling. In I. Levin & D. Zakay (Eds.), Time and human cognition: A life span perspective (pp. 185–218). Elsevier Science Publishers.

  • Szilágyi, J., Clements, D. H., & Sarama, J. (2013). Young children’s understandings of length measurement: Evaluating a learning trajectory. Journal for Research in Mathematics Education, 44(3), 581–620.

    Article  Google Scholar 

  • Tan, P. L., Lim, C. S., & Kor, L. K. (2017). Diagnosing primary pupils’ learning of the concept of “after” in the topic “time” through knowledge states by using cognitive diagnostic assessment. Malaysian Journal of Learning and Instruction, 14(2), 145–175.

    Article  Google Scholar 

  • Tavakol, M., & Dennick, R. (2011). Post-examination analysis of objective tests. Medical Teacher, 33(6), 447–458.

    Article  Google Scholar 

  • Verschaffel, L., Van Dooren, W., & De Smedt, B. (2012). Mathematical learning. In N. M. Seel (Ed.), Encyclopedia of the sciences of learning (pp. 2107–2110). MA: Springer.

  • Wilson, M. (2005). Construct measures: An item response modelling approach. Lawrence Erlbaum Associates.

  • Wilson, M. (2009). Measuring progressions: Assessment structures underlying a learning progression. Journal of Research in Science Teaching, 46(6), 716–730.

  • Wilson, M. (2018). Making measurement important for education: The crucial role of classroom assessment. Educational Measurement: Issues and Practice, 37(1), 5–20.

  • Wilson, M., & Sloane, K. (2000). From principles to practice: An embedded assessment system. Applied Measurement in Education, 13(2), 181–208.

Download references

Acknowledgements

The authors would like to thank all the teachers and Year Four pupils who voluntarily participated in this study.

Funding

This study is made possible with funding from the Research University Grant (RUI) Scheme 1001/PGURU/8011027 of Universiti Sains Malaysia.

Author information

Authors and Affiliations

  1. School of Educational Studies, Universiti Sains Malaysia, 11800 USM, Penang, Malaysia

    Chin Huan, Chew Cheng Meng, Lim Hooi Lian & Thien Lei Mee

Authors
  1. Chin Huan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chew Cheng Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lim Hooi Lian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Thien Lei Mee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chew Cheng Meng.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chin, H., Chew, C.M., Lim, H.L. et al. Development and Validation of a Cognitive Diagnostic Assessment with Ordered Multiple-Choice Items for Addition of Time. Int J of Sci and Math Educ 20, 817–837 (2022). https://doi.org/10.1007/s10763-021-10170-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10763-021-10170-5

Keywords

Search

Navigation