In this study, we investigated how rater effects manifest in writing assessment by examining rater severity and criterion interpretation, with particular attention paid to differences associated with the gender of the raters and the school level they teach. To this end, the measurement characteristics of writing samples, scoring criteria, and raters were examined, and the rating results were compared across the raters’ genders and school levels. Follow-up interviews were conducted to determine the factors underlying the observed differences. The results revealed no significant rater effects associated with gender, whereas significant differences across school levels were observed for several scoring criteria. Follow-up interviews further revealed that raters at different school levels differed in how they interpreted and applied the scoring criteria. These findings suggest that rater effects in writing assessment may arise from differences in not only the severity of the raters but also their interpretation of scoring criteria. This study highlights the need for rater training to promote the consistent interpretation and application of scoring criteria.