Korean Language Education Research

Sign In
한 EN
ISSN (P) 1225-8571 ISSN (E) 2713-699X

Article Info

Reliability Analysis of Scoring - Rescoring Depending on the Consistency Type of Korean Language Teachers’ the Writing Assessment

Yoon Keumjoon Park Youngmin

청명 High School, Teacher Korea National University of Education, Department of Korean Language Education, Professor

Korean Language Education Research Vol. 56 No. 1 pp.151-183 (2021)

DOI: 10.20880/kler.2021.56.151

Abstract

The purpose of this study is to analyze the reliability of raters based on the results of the writing assessment scoring-rescoring of Korean lan- guage teachers. For this purpose, 36 Korean language teachers scored and re-scored 20 students’ writings, and they were categorized into groups based on the type of consistency according to the results of the multi-faceted Rasch analysis. As a result, 25.00% of raters showed con- sistency in both scoring and rescoring and 52.78% of raters showed con- sistency only once. In addition, 22.22% of the raters were found to have failed to maintain consistency. Looking at the reliability of scoring and rescoring, the scoring-rescoring reliability tends to be high when consis- tency is suitable. In case of over-fitting, the reliability of scoring and re- scoring was high, but the mean difference was significant because of the difference of the severity in scoring and rescoring. In case of mis-fitting, the reliability of scoring-rescoring was the lowest. This suggests that vari- ous measures should be sought to ensure the reliability of the writing as- sessment.

Keywords

국어교사쓰기평가평가자 일관성채점-재채점 신뢰도다국면 Rasch 모형

References

  1. [1] [학위논문] 가은아/쓰기 발달의 양상과 특성 연구/박사/한국교원대학교 대학원/국어교육/2011
  2. [2] [단행본] 신뢰도/강승호/교육과학사/2004/~///
  3. [3] [단행본] 교육평가/권대훈/학지사/2008/~///
  4. [4] [학위논문] 권태현/쓰기 성취기준에 따른 학생 예시문 선정에 관한 연구/박사/한국교원대학교 대학원/국어교육전공/2014
  5. [5] [단행본] 교육평 : 이해와 적용/박도순/교육과학사/2012/~///
  6. [6] [단행본] 작문교육론/박영민/역락/2016/~///
  7. [7] [학술지(정기간행물)] 서수현/쓰기 평가 협의 과정에 나타난 쓰기 평가자의 인식 연구/국어교육학연구/2012/(44)/335~367/국어교육학회/10.20880/kler.2012..44.335
  8. [8] [단행본] 국어학과 국어교육학/성낙수/채륜/2015/~///
  9. [9] [단행본] 타당도와 신뢰도/성태제/학지사/2002/~///
  10. [10] [학술지(정기간행물)] 이재창/조현병 환자가 시행한 주의력 네트워크 검사 점수의 검사-재검사 신뢰도/정신신체의학/2017/25(2)/210~217/한국정신신체의학회/
  11. [11] [학술지(정기간행물)] 이혜원/학령전기용 어음청각검사 어표의 검사-재검사 신뢰도/Audiology and Speech Research/2014/10(1)/25~34/한국청각언어재활학회/
  12. [12] [단행본] 언어교육평가 연구를 위한 FACETS 프로그램/장소영/글로벌콘텐츠/2009/~///
  13. [13] [학위논문] 장은주/채점의 일관성 유형에 따른 국어교사의 쓰기 평가 특성 분석/박사/한국교원대학교//2015
  14. [14] [학위논문] 허소희/한국어 이명주요기능설문지의 검사-재검사 신뢰도 검증 및 간소화 버전 제안/석사/한림대학교//2019
  15. [15] [학술지(정기간행물)] Barrett, S./The impact of training on rater variability/International Education Journal/2001/2/49~58//
  16. [16] [학술지(정기간행물)] Eckes, T./Rater types in writing performance assessments : A classification approach to rater variability/Language Testing/2008/25(2)/155~185//
  17. [17] [단행본] Handbook of Inter-Rater Reliability/Gwet, K. L./Advanced Analytics, LLC/2012/~///
  18. [18] [단행본] Many-facet Rasch Measurement/Linacre, J. M./MESA Press/1989/~///
  19. [19] [학술지(정기간행물)] Linacre, J. M./Rating, judges, and fairness/Rasch Measurement Transactions/1998/12(2)/630~631//
  20. [20] [단행본] FACETS/Linacre, J. M./MESA Press/1990/~///
  21. [21] [학술지(정기간행물)] Lumley, T./Rater characteristics and rater bias : Implications for training/Language Testing/1995/12(1)/54~71//
  22. [22] [단행본] Measuring Second Language Performance/McNamara, T. F./Longman/1996/~///
  23. [23] [학술지(정기간행물)] Mukaka, M. M./A guide to appropriate use of correlation coefficient in medical research/Malawi Medical Journal/2012/24(3)/69~71//
  24. [24] [학술지(정기간행물)] Myford, C. M./Detecting and Measuring Rater Effects Using Many-Facet Rasch Measurement : Part Ⅰ/Journal of Applied Measurement/2003/4(4)/386~422//
  25. [25] [단행본] Designing writing tasks for the assessment of writing/Ruth, L./Ablex Publishing Corporation/1988/~///
  26. [26] [학술지(정기간행물)] White, E. M./Holisticism/College Composition and Communication/1984/35(4)/400~409//

Busan National University, 2nd College of Education Building, Room 315 (Prof. Eun Jung Jeon’s Office), Jangjeon-dong, Geumjeong-gu, Busan, Republic of Korea Tel: +82-51-510-2602 Society E-mail: koredu1991@daum.net Editorial Board E-mail: koredu1991_edit@daum.net Copyright © 2026 Association of Korean Language Education Research. All Rights Reserved.