Reliability Analysis of Scoring - Rescoring Depending on the Consistency Type of Korean Language Teachers’ the Writing Assessment
Yoon Keumjoon Park Youngmin
Cheongmyeong High School, Teacher Korea National University of Education, Department of Korean Language Education, Professor
Korean Language Education Research Vol. 56 No. 1 pp.151-183 (2021)
Abstract
Yoon Keumjoon · Park Youngmin The purpose of this study is to analyze the reliability of raters based on the results of the writing assessment scoring-rescoring of Korean language teachers. For this purpose, 36 Korean language teachers scored and re-scored 20 students’ writings, and they were categorized into groups based on the type of consistency according to the results of the multi-faceted Rasch analysis. As a result, 25.00% of raters showed consistency in both scoring and rescoring and 52.78% of raters showed consistency only once. In addition, 22.22% of the raters were found to have failed to maintain consistency. Looking at the reliability of scoring and rescoring, the scoring-rescoring reliability tends to be high when consistency is suitable. In case of over-fitting, the reliability of scoring and rescoring was high, but the mean difference was significant because of the difference of the severity in scoring and rescoring. In case of mis-fitting, the reliability of scoring-rescoring was the lowest. This suggests that various measures should be sought to ensure the reliability of the writing assessment.
Keywords
Korean language teacherswriting assessmentrater consistencyscoring-rescoring reliabilitymany-facet Rasch measurement
