PURPOSE Consistent evaluation procedures based on objective and rational standards are essential for the sustainability of portfolio-based education, which has been widely introduced in medical education. We aimed to develop and implement a portfolio assessment system, and to assess its validity and reliability. METHODS We developed a portfolio assessment system from March 2019 to August 2019 and confirmed its content validity through expert assessment by an expert group comprising 2 medical education specialists, 2 professors involved in education at medical school, and a professor of basic medical science. Six trained assessors conducted 2 rounds of evaluation of 7 randomly selected portfolios for the “Self-Development and Portfolio II” course from January 2020 to July 2020. These data are used inter-rater reliability was evaluated using intra-class correlation coefficients (ICCs) in September 2020. RESULTS The portfolio assessment system is based on the following process; assessor selection, training, analytical/comprehensive evaluation, and consensus. Appropriately trained assessors evaluated portfolios based on specific assessment criteria and a rubric for assigning points. In the analysis of inter-rater reliability, the first round of evaluation grades was submitted, and all assessment areas except “goal-setting” showed a high ICC of 0.81 or higher. After the first round of assessment, we attempted to standardize objective assessment procedures. As a result, all components of the assessments showed close correlations, with ICCs of 0.81 or higher. CONCLUSION We confirmed that when assessors with an appropriate training conduct portfolio assessment based on specified standards through a systematic procedure, the results are reliable.

中文翻译：

---

为韩国医学院开发和验证组合评估系统。

目的基于客观和合理标准的一致评估程序对于基于组合的教育的可持续性至关重要，这已被广泛引入医学教育。我们旨在开发和实施投资组合评估系统，并评估其有效性和可靠性。方法我们于2019年3月至2019年8月开发了作品集评估系统，并通过由2名医学教育专家、2名从事教育的医学院教授和1名基础医学教授组成的专家组的专家评估确认其内容有效性。从 2020 年 1 月至 2020 年 7 月，六名训练有素的评估员对“自我发展和投资组合 II”课程的 7 个随机选择的投资组合进行了 2 轮评估。这些数据用于在 2020 年 9 月使用类内相关系数 (ICC) 评估评估者间的可靠性。 结果 投资组合评估系统基于以下过程；评估员选择、培训、分析/综合评估和共识。受过适当培训的评估员根据特定的评估标准和评分标准评估作品集。在评估者间信度分析中，提交了第一轮评估等级，除“目标设定”外的所有评估领域均显示出 0.81 或更高的高 ICC。在第一轮评估之后，我们尝试规范客观评估程序。因此，评估的所有组成部分都显示出密切的相关性，ICC 为 0.81 或更高。