1. Application of AI in General Education and Writing Education

In recent years, the emergence of generative artificial AI including ChatGPT has been transforming the overall modes of teaching, learning, and evaluation in education. Several studies have examined the functions, strengths, and limitations of ChatGPT and have argued that such technologies are expanding discussions about the future of learning, instruction, and evaluation in higher education(Allam et al., 2023; Rudolph et al., 2023). In particular, ChatGPT has been empirically tested within both teacher and learner contexts, and its educational applicability has been increasingly explored(Allam et al., 2023).

Although GPT-based tools have been utilized in various forms within higher education, few studies have systematically examined their effects at both micro and macro levels. To address this gap, Tajik(2024) analyzed the potential capabilities of GPT in three domains—student-facing, teacher-facing, and system-facing contexts. They have demonstrated that AI can support a wide range of educational activities in higher education. Meanwhile Mao, Chen, & Liu(2023) has noted that while generative AI holds potential for enhancing educational practices, it also leads to critical challenges and dilemmas when applied to teaching, learning, and evaluation.

On the other hand, Smolansky, Cram, Raduescu, Zeivots, Huber, & Kizilcec(2023) questioned whether traditional evaluation practices remain sustainable amid the rapid proliferation of generative AI. A university-based survey revealed that both instructors and students were already using AI to a certain extent, though their perceptions of its impact varied depending on the type of evaluation. Instructors tended to favor evaluation methods that assume AI use, whereas students expressed concerns about the possible decline in creativity. Such findings suggest that AI should not merely be treated as a supplementary tool, but that evaluation design and implementation must be restructured in alignment with AI-integrated learning environments (Smolansky et al., 2023). This line of inquiry has direct implications for writing education, where evaluation plays a central role in shaping both learning processes and instructional practices.

Building on this discussion, generative AI has also driven significant changes beyond general education into the domain of writing instruction. Large language model (LLM)-based AI systems such as ChatGPT have gained attention as tools that support students’ writing processes and supplement teachers’ feedback and evaluation. Previous studies have shown that ChatGPT goes beyond simple information transfer and plays a role in providing multifaceted assistance throughout the entire writing process(Al Afnan et al., 2023; Imran & Almusharraf, 2023).

At the stages of exploring information and generating ideas, ChatGPT has been found to offer students accurate and reliable information. It implies that ChatGPT has potential as an alternative to conventional search engines(Al Afnan et al., 2023). Learners can rapidly access background knowledge related to a topic, outline of writing, and develop logical structure of text through ChatGPT, which serve as cognitive scaffolds during the early stages of writing. ChatGPT’s responses go beyond factual explanations; they support learners’ organization of ideas and recognition of issues, thereby increasing learners’ engagement and efficiency in writing tasks(Imran & Almusharraf, 2023).

From the perspective of formative feedback, ChatGPT can serve as an alternative or supplementary tool in situations where immediate teacher intervention is difficult. Steiss et al.(2024) compared feedback provided by humans and ChatGPT and found that ChatGPT can deliver a certain level of criteria-based feedback that is useful for students in checking and revising their drafts. Although human feedback was found to be superior in qualitative depth, ChatGPT can still function as a practical support tool in large-scale writing classes or in contexts where timely teacher feedback is limited.

The potential of generative AI has also been highlighted in terms of linguistic quality and logical coherence. Herbold, Hautli-Janisz, Heuer, Kikteva, & Trautsch (2023) reported that argumentative texts produced by ChatGPT were rated higher than those written by humans in terms of logical structure and linguistic complexity. This finding suggests that generative AI can maintain a consistency in linguistic quality and produce texts with logical flow and refined expression.

ChatGPT has been applied to creative writing and genre-specific writing. Shidiq (2023) noted that while ChatGPT can support students’ thinking and expressive abilities in creative writing, it also poses a potential risk of diminishing learners’ creative capacities. Megawati (2023) examined the applicability of ChatGPT in the context of scientific writing, suggesting that AI can function as a writing assistant in articulating complex scientific concepts with greater clarity.

Comparative study between AI-generated and human-written texts have provided new directions for writing evaluation. Yan, Fauss, Hao, & Cui (2023) identified structural and linguistic differences between AI’s writing and human’s writing through large-scale textual analysis, and emphasized the need for methodological improvements to ensure the quality of writing evaluation. These findings indicate that AI has the potential to reshape the overall methodology of writing education and writing evaluation.

2. Applications of AI in Writing Evaluation

While ChatGPT has demonstrated potential across teaching & learning domains, recent research has increasingly explored its role in the field of writing evaluation. Writing evaluation is one of the most direct means of evaluating learners’ thinking and expressive abilities, yet it has also been regarded as one of the most challenging areas due to the inherent subjectivity of raters and the heavy time demand. Human raters are likely to make divergent judgments on the same task, which can reduce the reliability and validity of the evaluation (Awidi,2024; Wang, Engelhard Jr, Raczynski, Song, & Wolfe, 2017). Wang et al.(2017) reported that inter-rater consistency in writing evaluation was relatively low and that raters and experts differed in their perceptions about the degree of text borrowing, the development of ideas, and the maintenance of focus. These findings suggest that subjectivity is structurally embedded in human evaluation for written texts.

Awidi(2024) similarly pointed out that traditional human grading processes are inherently subjective, particularly when multiple raters are involved in large classes, often resulting in inconsistent scores. He noted that “writting assessment normally has subjective and inconsistent effects in grading,” emphasizing that inconsistent criteria and excessive workload among raters undermine both the consistency and efficiency of assessment. In educational contexts where one teacher is responsible for many classes and students, it is difficult to secure sufficient time for thorough evaluation and feedback, which may in turn lead to fluctuations in grading quality.

Given these practical constraints, the introduction of AI-based evaluation systems has been proposed as an alternative approach to enhance the objectivity and consistency of writing evaluation. Awidi(2024) raised the critical question, “Can AI tools like ChatGPT mark, assign a grade/score, and provide personalized feedback on reflective essays effectively?” He explored the potential of AI to evaluate open-ended writing tasks such as reflective essays in a rapid and consistent manner, arguing that generative AI tools like ChatGPT has the potential to complement human raters’ subjective judgments.

Jauhiainen and Guerra(2025) applied ChatGPT-4 to the evaluation of open-ended written examinations among university students to examine the extent to which AI scoring aligned with human grading. ChatGPT-4 evaluated 54 responses(ranging from 24 to 256 words) in a total of 3,240 times based on five criteria, showing a statistically significant correlation with human ratings. The researchers concluded that ChatGPT demonstrated promising potential in providing consistent and reproducible assessment of open-ended responses. This finding suggests that ChatGPT can serve not merely as a text generator but as an assistant rater, enhancing the efficiency and reliability of the evaluation process.

In South Korea, similar efforts have emerged to incorporate ChatGPT into writing assessment. Park and Lee(2024) developed an AI-based writing-communication-competency-assessment-tool using ChatGPT, integrating three major categories—content, organization, and language use—into an automated scoring algorithm that provides both scores and domain-specific feedback. This study was significant in that it presented a concrete procedure for integrating generative AI into assessment frameworks. Similarly, Park and Lee(2024) compared rubrics for writing evaluation developed by human and AI, exploring the characteristics and educational implications of AI-generated rubrics. Their results indicated that while both human and AI raters prioritized content-related elements, AI introduced additional categories for criteria such as format, citation, critical thinking and overall impression. These differences imply that AI can serve to supplement or expand the human’s evaluation framework.

Further, Korean studies have empirically examined the feasibility of AI-based automated scoring and feedback systems from multiple perspectives. Son(2023), Cha et al.(2024), and Choi et al.(2025) analyzed the potential of AI-assisted writing evaluation(AWE) systems and teacher–AI collaborative models, while Son(2025), Kim(2024), Lee(2025), and Choi(2025) explored the effectiveness and limitations of AI feedback tools in university writing instruction. These studies have shown that AI can improve the efficiency and consistency of assessment. However, most of them have focused primarily on automated scoring & feedback mechanism rather than on comparison between ChatGPT and human raters.

If we synthesize the results of previous studies, ChatGPT might have the potential to (1)mitigate raters’ subjectivity, (2)promote the standardization of evaluation criteria, and (3)provide rapid feedback on many pieces of writing. Despite these possibilities, existing research has not yet empirically verified whether ChatGPT can apply the same evaluation criteria as human raters or whether its scoring results are consistent and valid. This study compares the evaluation criteria developed by TEs and ChatGPT and analyzes what are the perspectives and frameworks for “good writing” by TEs and ChatGPT. Furthermore, this study examines how ChatGPT rates texts consistently based on the TEs’ criteria. Through this analysis, this study explores the level of validity and consistency exhibited by generative AI in both the process and outcomes of writing evaluation.

1. Procedure for Research Question 1

To address Research Question 1 three TEs collaboratively developed evaluation criteria and a rubric to rate 19 editorials written by PTs, and subsequently rated 19 texts. The TEs followed the steps summarized in <Table 2>

Before developing the rubric, the TEs carefully reviewed all 19 editorials. Then, due to all of us living in different cities, we had an online meeting. During the meeting we developed the rubric through discussion, taking into account both the characteristics of the university newspaper editorials and the quality levels of the PTs’ writing. The online meeting was recorded for research purposes. After developing the rubric, the TEs first independently rated four texts. We then had a second online meeting to discuss again the criteria and rubric as well as discrepancies among our scores. Through discussion, we adjusted the criteria and rubric. Following this calibration session, each TE rated the remaining 15 texts independently.

Inter-rater reliability among the three raters was calculated using the intraclass correlation coefficient (ICC), which yielded $$r = .969$$(95% CI = 0.934–0.987, p < .001), indicating a very high level of reliability. The final score for each text was determined by averaging the three raters’ scores.

Next, we instructed ChatGPT to generate a set of writing evaluation criteria for a newspaper editorial and a corresponding rubric. This study employed OpenAI’s ChatGPT Plus version(GPT-4 based), which provides enhanced data analysis and text-processing capabilities compared to the standard model(free subscription version). The Plus version was selected because it offered more stable performance for research purposes while remaining practically accessible for educational use.

ChatGPT was asked to develop its rubric under conditions equivalent to those of the TEs. To this end, the researchers allowed ChatGPT to review all 19 texts, and provided the same writing task(Table 1). No restrictions were imposed on the number of criteria, scoring range, or weighting of items, allowing ChatGPT to design its framework in its own way.

To ensure that ChatGPT’s rubric achieved a level of specificity and detail comparable to that of TEs, the researchers continued to refine its output through iterative prompting, issuing additional instructions after the initial dialogue(Table 3

2. Procedure for Research Question 2

To address Research Question 2 both the TEs and ChatGPT rated the same 19 texts based on the rubric created by the TEs. The rating process the TEs followed were the same as explained above(Table 2). The TEs’ rubric and scoring guide was provided to ChatGPT. To ensure that ChatGPT’s evaluation process resembled that of the TE’s, it was first instructed to rate a pilot set of four texts . Analysis of the initial results showed that ChatGPT tended to assign higher scores than the human raters. Therefore, to ensure consistency in scoring rigor, we provided additional prompts to adjust the strictness within the same rubric framework—corresponding to the rater calibration phase in human evaluation. After re-evaluation, ChatGPT’s score distribution aligned more closely with that of the TEs.

ChatGPT then evaluated all 19 texts, including the remaining 15. The range of scores given by ChatGPT was narrow. Accordingly, we additionally requested to apply the relative evaluation method to secure discrimination, and ChatGPT performed a re-evaluation accordingly. This adjustment was implemented to address the limited score range and to facilitate comparative analysis across texts. We instructed ChatGPT to specifically describe the grounds and reasons for scoring as well as the scores for each article, and the final results by ChatGPT were used as analysis data for the study. These explanations were used as key data for qualitative analysis¹.

All PTs’ texts used in AI evaluation were fully anonymized, and no personally identifiable information was retained. During the use of ChatGPT, the data storage and history functions were disabled to prevent any exposure of personal information. All procedures complied with institutional research ethics standards.

¹ For Research Question 2, a consistent evaluation prompt was applied to all texts, assigning ChatGPT the role of an evaluator, specifying the evaluation context, providing the rubric, and requesting both scores and rationales. The full prompt log was not included in the main text due to space limitations.

3. Data Analysis

This study employed both content analysis and comparative analysis to examine the evaluation criteria and results by the TEs and ChatGPT. For Research Question 1, the two rubrics were subjected to content analysis. Each criterion in a rubric was examined in terms of its conceptual scope, evaluative focus (e.g., logic, grounds, expression), and linguistic specificity or inclusiveness. We independently reviewed and discussed our interpretations through iterative meetings until full consensus was reached.

For Research Question 2, this study conducted a descriptive comparison of the scores assigned to each category in the TEs’ rubric. In addition, the qualitative content of the evaluative comments was compared to identify differences in focus(content-oriented vs. form-oriented), justification strategies, and feedback tone. These analyses were also reviewed independently by the three researchers and refined through repeated discussions to ensure consistency in interpretation.

1. Writing Evaluation Criteria & Rubric Developed by TEs and ChatGPT for a newspaper Editorial

First, the writing evaluation criteria and corresponding rubric developed by the TEs are presented below(Table 4

The overall aim of the rubric was designed to reflect the characteristics of university newspaper editorials in detail, enabling a step-by-step evaluation from topical relevance to linguistic accuracy. The criteria & rubric generated by ChatGPT is presented in Table 5

A major similarity between the TEs and ChatGPT is that both sought to faithfully reflect the characteristics of a genre, newspaper editorial. Both recognized logical reasoning & persuasiveness as the essential foundation of an editorial, and incorporated this principle into their criteria.

The TEs emphasized content-oriented criteria such as “Topic Relevance”(timeliness and consideration of university members), “Clarity of Claim,” “Appropriateness of Argument,” and “Validity of Grounds,” alongside structure/meaning/language-related criteria including “Appropriateness of Organization,” “Coherence,” “Connectivity of Sentences and Paragraphs,” “Appropriateness of Vocabulary and Sentences,” and “Grammatical Accuracy.”

Similarly, ChatGPT identified “Consistency of Logical Development,” “Objectivity and Validity of Grounds,” and “Logical Persuasiveness” as core elements of evaluation. These parallels indicate that both evaluator groups(TEs and ChatGPT) understood the fundamental principles of persuasive writing, centering their judgments on logic and persuasiveness in text construction.

Another commonality lay in the establishment of a explicit and concrete grading criteria. The TEs adopted a three-point scale(high–medium–low), while ChatGPT used a 10 or 15-point scale with corresponding rubric bands. These detailed descriptions can allow reducing subjective variation and enhancing consistency and reliability among evaluators. In particular, the fact that ChatGPT, despite being an automated evaluation model, has constructed detailed evaluation criteria similar to those used by human evaluators suggests that AI may reflect certain characteristics of human assessment beyond the level of a simple automated scoring algorithm. However, such similarity does not imply that ChatGPT replicates human evaluators’ value systems or cognitive processes. Rather, ChatGPT appears to approximate human reasoning by inferring evaluative judgments from statistical patterns of language use.

On the other hand, there were clear differences in the composition and focus of the evaluation categories. First, the TEs’ rubric contained 11 categories, while ChatGPT’s rubric consisted of 8 more integrated categories. This difference reflects distinct perspectives on evaluation: the TEs designed a fine-grained rubric to examine student writing in detail and to provide targeted feedback, whereas ChatGPT produced key components into broader categories.

Second, in topic selection, the TEs emphasized “timeliness” and “consideration of university members.” ChatGPT’s criteria also incorporated these elements but further added “contribution to public discourse.” In other words, ChatGPT conceptualized a university newspaper editorial not merely as an internal institutional text but as part of a larger social dialogue, expanding the scope of evaluation to include public relevance and potential for discourse formation. This aspect was rather overlooked by the TEs, which implies ChatGPT’s criteria and rubric can be supplementary tools or references for cross-check.

Third, regarding logical development, the TEs treated “clarity of claims,” “appropriateness of argument,” and “appropriateness of organization” as separate categories, assessing how clearly claims were presented, how claims were connected to grounds and how the text’s structure was logically formed. ChatGPT, by contrast, consolidated these aspects under the single category of “consistency of logical development,” focusing on the overall coherence among argument, claims, and grounds as well as paragraphrasing. Thus, while the TEs approached logic analytically through detailed components, ChatGPT used a criterion for overall logic coherence. ChatGPT evaluates ‘counterarguments’ as a part of “Logical persuasiveness that resonates with the reader ” but we are doubtful that considering counterarguments is seperated from judging ‘logical development.’ TEs think that when human evaluator rate a piece of persuasive writing, s/he can discern ①author’s claims, ②grounds(reasons and evidences) for the claims, and ③logical building up of arguments. We could discern these 3 features in 19 texts so rate them differently according to quality of each writing. The ChatGPT in this study didn’t or couldn’t perform this like human rators.

Fourth, the two groups diverged in their treatment of ‘persuasiveness’ and ‘considering readers’. The TEs did not assign persuasiveness as an independent criterion, viewing it instead as a quality embedded within other categories such as appropriateness of argument, clarity of claims, and validity of grounds. ChatGPT, on the other hand, established “Logical persuasiveness that resonates with the reader” as a distinct category, including the ability to anticipate and respond to counterarguments. ChatGPT dealt with persuasiveness as part of a logical process as well as an aspect related to readers. This may be caused by the ChatGPT’s info-processing of writing tasks “Logical organization that appeals to readers’ reasoning and empathy.” In contrast, the TEs viewed persuasiveness as an integrated effect—emerging from logical content-organization and coherence.

Fifth, in the domain of expression, the TEs separated “appropriateness of vocabulary and sentence use” from “grammatical accuracy,” focusing on suitable use of words & sentences, and grammatical correctness. ChatGPT combined these dimensions into two categories: “accuracy, originality, and persuasiveness of expression,” and “adherence to length and editorial completeness.” In the former, it deals with clarity of sentences & persuasive expression, and in the latter, it checks grammatical correctness. ChatGPT didn’t mention about the use of vocabulary and dealt with paragraphing as a part of completeness of text in “adherence to length and editorial completeness.” TEs are well aware that students often make mistakes when writing Korean sentences, such as using ungrammatical sentences and inappropriate vocabulary. Therefore, they have incorporated sentences and vocabulary into separate assessment criteria. Since ChatGPT is more adept at processing information in English than in Korean, this aspect is believed to have been overlooked.

Sixth, the two groups differed in “originality.” The TEs emphasized originality in thoughts or claims—the novelty of ideas and perspectives—whereas ChatGPT valued creativity in language use, such as expression.

Seventh, clear differences were also observed in terms of media appropriateness. While the TEs’ rubric did not include a specific category for media format, ChatGPT introduced “media appropriateness” as an independent item, assessing both adherence to the conventions of newspaper editorials and readability in online environments. This demonstrates that ChatGPT viewed the “university newspaper editorial” not solely as a printed text but as one that circulates within digital media contexts. It seems that ChatGPT reflects the requirement of writing tasks into the criteria and rubric thoroughly. This aspect was disregarded in TEs’ criteria.

Furthermore, in the “appropriateness of title” category, ChatGPT evaluated not only whether the title reflected the text’s content but also whether it could attract readers’ attention and encourage clicks. This approach shows that ChatGPT expanded the notion of writing evaluation beyond textual construction to include reader response and the digital environment. How such evaluation criteria should be interpreted in educational contexts remains open to further discussion. The TEs, by contrast, designed their rubric primarily for print-based editorials, focusing on representative feature of title rather than attracting readers’ clicks.

The TEs’ criteria are more detailed than ChatGPT’s but the latter presents points in its criteria & rubric that the TEs had overlooked. In this regard, GPT could serve as a meta-cognitive tool for assessment that educators examine and recalibrate their own practice in setting criteria and developing rubrics. For example, educators may refer to the evaluation elements suggested by ChatGPT to examine whether certain criteria have been omitted or overly condensed in their existing rubrics, and to revise or adjust them accordingly.

2. Comparison Between TEs’ and ChatGPT’s Evaluation Results

Based on the rubric developed by the TEs, both ChatGPT and the TEs evaluated the same set of editorials written for the university newspaper. Each evaluation was conducted on a 33-point scale, and the comparison of scores and rankings is summarized as follows.

Although the two groups shared some overlap in overall score range, they showed marked differences in score distribution and evaluative focus. To verify the correlation between the evaluation scores of TE and ChatGPT, Spearman’s rank correlation analysis was performed, and no significant relationship was found between the two variables (

First, ChatGPT exhibited a convergent rating tendency. The score distribution shows that ChatGPT clearly exhibited a central tendency bias. The TEs’ scores ranged from a high of 32(13 and 18) to a low of 18.6 (12), a spread of about 13 points, indicating strong discrimination among texts. In contrast, ChatGPT’s scores ranged only from 29(9 and 4) to 23(19), a spread of merely 6 points. This suggests that ChatGPT tended to avoid extreme scores and gravitated toward mid-range values.

This pattern indicates that ChatGPT tends to rely on mid-range scores when distinctions between levels are unclear. As a result, while its scoring was internally consistent, its capacity to discriminate subtle qualitative differences among texts was limited. In addition to this mid-range tendency, it is noteworthy that ChatGPT did not assign any text to the “Poor” band of its rubric. Because none of the student editorials were evaluated at the lowest performance level, the lower end of the scale remained unused, further narrowing the overall spread of ChatGPT’s scores.

Second, clear differences in evaluation focus emerged in the assessment of high-ranking and low-ranking writings. The TEs rated 13 and 18 the highest, while ChatGPT gave top scores to 9 and 4. The TEs prioritized the public relevance of the topic, logical coherence, and depth of argumentation, whereas ChatGPT favored clarity of expression, structural completeness, and surface-level consistency. A similar pattern appeared in the lower score range. The TEs penalized deficiencies in reasoning and insufficient grounds more heavily, while ChatGPT assigned moderate scores to texts that maintained clear structure and sentence-level accuracy despite weak argument. For example, 12, which received the lowest score from the TEs, was rated at a mid-level(25 points) by ChatGPT.

We interpret that these two tendencies are closely related. ChatGPT’s narrow score range reflects its tendency to rely on clearly identifiable linguistic and structural features—such as grammar, sentence organization, and paragraph structure—that can be applied consistently across different texts. In contrast, the TEs’ wider distribution stems from their interpretive engagement with each text’s ideas and argumentative depth, which allowed for finer distinctions in quality of the texts. This suggests that differences in scoring patterns are not merely quantitative but also reveal that human evaluators demonstrate the ability to analyze and judge a text with a sense of wholeness in detail.

These findings highlight a clear tendency in the two evaluation approaches(or processing). It seems that ChatGPT focused primarily on surface-level form and structural coherence, identifying grammatical or structural issues. Conversely, TEs emphasized the sophistication of argumentaion, the social implications of the topic/theme and awareness of readers, which are the dimensions involving interpretive understanding, along with accuracy in expression and well-structured form.