PURPOSE Consistent evaluation procedures based on objective and rational standards are essential for the sustainability of portfolio-based education, which has been widely introduced in medical education. We aimed to develop and implement a portfolio assessment system, and to assess its validity and reliability. METHODS We developed a portfolio assessment system from March 2019 to August 2019 and confirmed its content validity through expert assessment by an expert group comprising 2 medical education specialists, 2 professors involved in education at medical school, and a professor of basic medical science. Six trained assessors conducted 2 rounds of evaluation of 7 randomly selected portfolios for the “Self-Development and Portfolio II” course from January 2020 to July 2020. These data are used inter-rater reliability was evaluated using intra-class correlation coefficients (ICCs) in September 2020. RESULTS The portfolio assessment system is based on the following process; assessor selection, training, analytical/comprehensive evaluation, and consensus. Appropriately trained assessors evaluated portfolios based on specific assessment criteria and a rubric for assigning points. In the analysis of inter-rater reliability, the first round of evaluation grades was submitted, and all assessment areas except “goal-setting” showed a high ICC of 0.81 or higher. After the first round of assessment, we attempted to standardize objective assessment procedures. As a result, all components of the assessments showed close correlations, with ICCs of 0.81 or higher. CONCLUSION We confirmed that when assessors with an appropriate training conduct portfolio assessment based on specified standards through a systematic procedure, the results are reliable.

中文翻译：

---

开发并验证了韩国医学院校的档案袋评估系统。

目的基于客观和合理标准的一致评估程序对于基于组合的教育的可持续性至关重要，而基于组合的教育已在医学教育中广泛引入。我们旨在开发和实施投资组合评估系统，并评估其有效性和可靠性。方法我们于2019年3月至2019年8月开发了档案袋评估系统，并通过由2位医学教育专家，2位医学院教育教授和1位基础医学教授组成的专家小组的专家评估确认了其内容的有效性。从2020年1月至2020年7月，六名训练有素的评估人员对“自发展和投资组合II”课程中随机抽取的7个投资组合进行了2轮评估。这些数据用于评估者之间的可靠性，并于2020年9月使用类内相关系数（ICC）进行了评估。评估者的选择，培训，分析/综合评估和共识。经过适当培训的评估师根据特定的评估标准和分配分数的原则评估投资组合。在评估者之间的可靠性分析中，提交了第一轮评估等级，除“目标设定”之外的所有评估领域均显示出0.81或更高的ICC。在第一轮评估之后，我们尝试标准化客观评估程序。结果，评估的所有组成部分均显示出密切的相关性，ICC为0.81或更高。