Abstract With increasingly more educational tests being administered on computers and other digital devices, the comprehensive validity of tests has more potential to be improved. In the present investigation, 75 items were developed to measure educational technology competency (ETC) for pre-service teachers according to a framework constructed using the Delphi technique. In particular, the appropriateness of items was tested using the multitrait-multimethod (MTMM), higher-order factor model, and multidimensional item response theory (MIRT) to shape a program for developing and validating different modality items. The results were as follows: (1) There was robust convergent validity, weak discriminant validity, and an evident method effect of the items. (2) The graphic modality might explain more variance than the text and video modalities at discriminating examinees' ETC proficiencies. (3) The graphic modality was the most reliable modality among the three different modalities when assessing technology-supported teaching. Thus, this study makes a significant contribution to the design of assessments by developing and validating different modality items. The results could be generalized to some extent when items are developed based on other ETC standards.

中文翻译：

---

不同模式的验证项目以评估职前教师的教育技术能力

摘要 随着越来越多的教育考试在计算机和其他数字设备上进行，考试的综合效度有更大的提升潜力。在本调查中，根据使用德尔菲技术构建的框架，开发了 75 个项目来衡量职前教师的教育技术能力 (ETC)。特别是，使用多特征多方法 (MTMM)、高阶因素模型和多维项目反应理论 (MIRT) 测试项目的适当性，以制定开发和验证不同模式项目的程序。结果如下：（1）各题项收敛效度强，区分效度弱，方法效应明显。(2) 在区分考生的 ETC 熟练程度方面，图形模态可能比文本和视频模态解释更多的差异。(3) 在评估技术支持教学时，图形模态是三种不同模态中最可靠的模态。因此，本研究通过开发和验证不同的模态项目对评估设计做出了重大贡献。当项目基于其他 ETC 标准开发时，结果可以在一定程度上推广。