1. Generative AI and the Evolution of Writing Education

Writing instruction has consistently adapted to major technological transformations throughout history. The introduction of typewriters fundamentally altered the writing process, word processors redistributed cognitive demands across planning and revision phases, and the Internet expanded both the scope and complexity of audiences and knowledge sources available to writers (Graham, Mc Keown, Kiuhara, & Harris, 2012). Each technological shift prompted educational institutions to reconsider how writing instruction should evolve to support learners in leveraging new tools effectively (Karchmer, 2007; Rodriguez & Rodriguez, 1986). Large language models including ChatGPT represent perhaps the most significant pedagogical challenge to date, as these generative AI systems can produce written text autonomously, raising fundamental questions about the nature of writing as a learning activity (Godwin-Jones, 2021; Ranalli, 2018). The primary challenge for educators is to explore whether and in what ways AI integration can enhance writing skills, such as argumentative reasoning, rhetorical awareness, and critical thinking, rather than merely facilitating basic text creation or replacing crucial cognitive processes that contribute to meaningful writing development (Rudolph, Tan, & Tan, 2023).

Argumentation is widely used as a major evaluative practice in education. It is also taught intensively in secondary schools and higher education. This prominence underscores the essential function of argumentation in academic discourse (Mc Neil & Krajcik, 2009). Argumentative writing involves a complex reasoning process that necessitates the simultaneous integration of multiple cognitive and communicative skills (Duschl & Osborne, 2002). Effective argumentative writing requires structural fluency—the capacity to organize claims, evidence, and reasoning into coherent and logically sound arguments while acknowledging and addressing counterarguments (Van Eemeren, Henkemans, & Grootendorst, 2002). Argumentative writing also requires high levels of linguistic control. This is reflected in accurate word choice, appropriate management of tone, and stylistic decisions that strengthen persuasion without obscuring meaning. Argumentation requires dialogic fluency, which involves the ability to recognize social contexts, appreciate diverse perspectives, and craft appeals that resonate with specific audiences while addressing their concerns (Aull & Ross, 2020). These components operate in constant interaction throughout the writing process. Consequently, learners building argumentative skills need to manage structure, linguistic choices, and rhetorical moves simultaneously.

The prospect of integrating ChatGPT into argumentative writing instruction has generated substantial disagreement among researchers and teachers. Some scholars and educators advocate for thoughtful adoption, highlighting ChatGPT’s potential as an auxiliary tool that could provide feedback on both surface-level language issues and deeper argumentative structures, or function as a virtual learning partner through iterative dialogue (Fryer et al., 2019; Fuchs, 2023; Kasneci et al., 2023). Others, however, urge caution, arguing that integration may be premature without deeper understanding of the system’s significant limitations (Godwin-Jones, 2021). Chief among these limitations is ChatGPT’s insufficient consideration of social context—the variable cultural norms, audience expectations, historical circumstances, and power dynamics that fundamentally shape how arguments are constructed and received. Owing to its architecture based on pattern recognition within training data, generative AI lacks a genuine understanding of how social context varies across people, regions, cultures, and time periods, yet such context-sensitivity is essential to argumentation (Godwin-Jones, 2021). Moreover, ongoing concerns about the quality of logical reasoning, hallucinations, factual inaccuracies, and the system’s propensity to produce content that sounds plausible yet is misleading, raise questions about its reliability as a writing support tool (Su, Lin, & Lai, 2023).

2. Conceptualizing Argumentative Writing

The argumentative process is intrinsically dialogic, requiring authors to rebuild knowledge while actively considering social settings and opposing ideas (Duschl & Osborne, 2002; Van Eemeren et al., 2002). This emphasizes the importance of argumentative writing, as it requires the integration of social intelligence, communicative accuracy, and cognitive thinking (Aull & Ross, 2020). Any auxiliary tool that significantly improves argumentative competence must function on several levels at once: provide insightful feedback on the logical coherence and organization of arguments, help writers improve their language for accuracy and persuasive impact, and critically raise awareness of rhetorical contexts, audience considerations, and the viewpoints of those who might disagree with or challenge the writer’s claims.

Generative AI systems lack a genuine understanding of how cultural norms, audience expectations, historical particulars, and power dynamics vary across communities and contexts, and cannot meaningfully guide writers in developing the contextual sensitivity that effective persuasion demands (Godwin-Jones, 2021). This limitation suggests that while ChatGPT might serve a supplementary role in developing structural and linguistic competencies, educators seeking to develop students’ full argumentative sophistication—particularly the dialogic fluency essential to authentic persuasion—must carefully design instruction to compensate for the system’s fundamental blindness to social and rhetorical context (Zhang & Li, 2021; Thorp, 2023).

3. ChatGPT as an Auxiliary Tool

Recent studies have shown that students’ responses to ChatGPT are not binary. They neither accept nor reject it. Instead, they employ it in various ways that defy simple categorization (Kwak, 2024). Rather than wholesale adoption or avoidance, research on university students shows that they selectively absorb ChatGPT comments during revision processes, exercising critical judgment regarding which suggestions to implement and which to ignore (Kwak, 2024). This evidence shows conditional utility rather than transformative impact, implying that ChatGPT works best when integrated into intentional instructional contexts rather than as a stand-alone writing resource (Fryer, Nakao, & Thompson, 2019; Kasneci et al., 2023). Process-oriented writing instruction frameworks hold particular potential for establishing ChatGPT as a mediating tool during the drafting stages of composition, where exploratory discussion and idea production are key cognitive demands (Kwak, 2024).

4. Research gaps and future directions in AI-supported argumentative writing

A considerable research gap still exists between general awareness of ChatGPT’s technological capabilities and the specific uses of generative AI in writing processes (Kwak, 2024). While previous research has examined the quality of AI feedback and documented broad usage patterns, few studies have delved deeply into the decision-making processes that students use during composition and the actual shifts in argumentation quality that result from AI engagement. This discrepancy is especially significant because generic use patterns conceal crucial compositional behaviors, such as whether the work becomes superficially polished while losing argumentative depth. To understand this, we must extensively examine the decisions students make during the writing process rather than simply judging the finished work after it is completed.

1. Participants

Participants were 124 first-year undergraduate students enrolled in a required academic writing course at K University, a private university located in Seoul, South Korea. At K University, the first-year academic writing course is a mandatory component of the general education curriculum for all freshmen and is offered over a 15-week semester. All students in this study were taking a 15-week course at the time of data collection.

Only students who provided informed consent to participate in the study were included in the dataset. Thus, the final sample for analysis consisted of 124 consented students enrolled in the freshman writing course.

The research was conducted under the ethical regulations approved by the institution. The principal investigator completed the university’s Institutional Review Board (IRB) research ethics training and obtained IRB approval prior to data collection. At the start of the course, students received an explanation of the study’s purpose and procedures and were told that taking part was entirely optional and that they were free to discontinue their involvement at any point without affecting their academic progress. Students who chose to take part provided their written consent, and only data from these consenting students were used for research purposes. All identifying information was removed or anonymized prior to the analysis.

2. Course context and instructional procedure

The research was embedded within the existing first-year composition curriculum at K University. This course runs for 15 weeks and is designed to introduce first-year students to the core concepts and practices of academic writing, with particular emphasis on argumentative writing.

As part of the regular instruction, all participating students were taught the basic principles of argumentation based on Toulmin’s model of argument (Toulmin, 1969/2003). The instructions covered key elements of an argument (e.g., claim, evidence, warrant, counterargument, rebuttal) and how these elements function together in written argumentative texts.

During this course, the students completed a staged argumentative writing task. All research participants were required to complete the following:

These tasks were part of the normal course requirements for all the students in the class.

Writing Task

Can the expansion of online learning replace traditional classroom education?

With the recent rapid spread of online learning, the debate over the necessity of traditional classroom-based education has intensified. Online learning offers significant advantages in terms of accessibility and convenience, and some have argued that it can completely replace traditional educational methods. On the other hand, others argue that online learning, lacking face-to-face interaction and emotional exchange, cannot match the quality of traditional education, emphasizing the importance of traditional schooling and its benefits.

Write an essay logically presenting your opinion on whether online learning can replace traditional classroom education.

* The task was originally written in Korean and translated by the author.

This study gathered three categories of data to analyze ChatGPT use and the development of argumentative writing. One category comprised writing produced in class, including outlines in early and revised forms and the first and final versions of argumentative essays. The second category consisted of screenshots or URLs that recorded students’ interactions with ChatGPT.

Qualitative coding was performed on the collected ChatGPT conversation logs, dividing them into six categories: Surface Revision (SR), Content Elaboration (CE), Organization/Coherence (OC), Metacognitive Use (ME), Dependence (DE), and Collaborative Refinement (CR).

ChatGPT Log Codebook

1. Surface Revision (SR) Definition: Requests focused on improving the linguistic surface of the text, such as grammar, vocabulary choice, sentence structure, and fluency issues. Focus: Fixing errors and polishing the style. Exclusion: Does not include requests to change the logical structure or add new arguments.

2. Content Elaboration (CE) Definition: Requests to add to or strengthen the substance of the argument. This includes asking for evidence, examples, data, or counterarguments to support the claims. Focus: Adding key details and evidence to support the claims. Exclusion: Distinct from simply polishing the tone (SR) or reordering the paragraphs (OC).

3. Organization / Coherence (OC) Definition: Requests to improve the logical flow, structure, and unity of the content at the paragraph or essay level. Focus: Rearranging ideas, refining transitions, and double-checking logical connections. Exclusion: Does not cover the correction of single sentences (SR) or the development of new ideas(CE).

4. Metacognitive Use (ME) Definition: High-level requests in which students ask ChatGPT to examine, diagnose, or critique their writing. Focus: Self-regulation and identifying weaknesses rather than direct editing. Exclusion: Asking ChatGPT to "fix it" without explanation falls under other categories.

5. Dependence (DE) Definition: Instances in which the student relies on ChatGPT to write or completely rewrite the text, replacing their own cognitive effort. Focus: Passive acceptance of AI generation. Exclusion: If the student selectively uses parts of the AI's output while maintaining control, it is not considered dependence.

6. Collaborative Refinement (CR) Definition: A sophisticated, student-led partnership in which students edit their own work, ask for feedback, and critically select specific AI suggestions. Focus: A back-and-forth process of writing, obtaining AI advice, and selecting what to use. Exclusion: Mere copy-pasting or asking for simple grammar fixes does not constitute high-level collaboration.

3. Statistical analyses

All statistical analyses were conducted to examine how different patterns of ChatGPT use related to students’ learning gains in argumentative writing. Analyses were performed using Python 3.12.7, with a significance level set at $$\alpha = .05$$.

First, descriptive statistics were computed for all focal variables, including pre- and post-test scores for overall argumentative writing performance and for key components (evidence and counterargument), as well as the six ChatGPT-use subscales: Surface Revision (SR), Content Elaboration (CE), Organization/Coherence (OC), Metacognitive Use (ME), Dependence (DE), and Collaborative Refinement (CR). The four subscales associated with higher-order and constructive uses of ChatGPT (CE, OC, ME, CR) were combined to form a Positive_Use_Index, and the two subscales reflecting more surface-level and potentially problematic uses (SR, DE) were combined to form a Risk_Use_Index. A Balance_Index was then computed by subtracting the Risk_Use_Index from Positive_Use_Index, such that higher scores indicated that constructive uses of ChatGPT outweighed surface-level and dependency-oriented uses.

To explore the broad patterns of ChatGPT use, a cluster analysis was conducted on the six ChatGPT-use subscales (SR, CE, OC, ME, DE, and CR). Standardized scores for these subscales were used as input variables. The resulting clusters were interpreted as qualitatively distinct profiles of ChatGPT use (e.g., relatively positive, mixed, low use). These clusters were then compared on learning gains in argumentative writing, operationalized as gain scores (post–pre) for overall writing quality (Gain_Total) and for specific components such as Evidence (Gain_Evidence) and Counterargument (Gain_Counterargument).

Next, to focus more specifically on the role of constructive ChatGPT use, the Positive_Use_Index was used to form three groups based on tertile splits (low, mid, and high). A one-way ANOVA was conducted with the Positive_Use_Index group (low/mid/high) as the between-subjects factor and Gain_Evidence (post–pre Evidence score) as the dependent variable. This analysis tested whether students who made more frequent and constructive use of ChatGPT for content elaboration, organization, metacognitive engagement, and collaborative refinement showed greater gains in their use of evidence. The omnibus F-test and corresponding effect size were used to evaluate group differences, and post hoc comparisons were used to interpret the significant effects.

Finally, to examine the unique contribution of positive ChatGPT use to gains in addressing counterarguments while controlling for initial ability, multiple regression analysis was conducted. Gain_Counterargument (post–pre counterargument scores) served as the dependent variable. The predictors were Pre_Counterargument, Positive_Use_Index, and their interaction terms (Pre_Counterargument $$\times$$ Positive_Use_Index). The model fit (R²), significance and direction of each regression coefficient, and interaction term were examined.

The analyses resulted in a unified group of assessment tools. These tools linked broad patterns of ChatGPT use to both helpful and potentially risky practices. They also included detailed indicators of effective use, such as how students worked with the evidence and counterarguments.

1. Descriptive statistics and preliminary analyses

¹ The grading criteria and survey for argumentative writing were based on the Argumentative Writing Education Project (Newell, Bloome, & Hirvela, 2015). Considering that they emphasize Toulmin's argumentation model, which is also heavily relied upon in Korean argumentative writing education; that they are designed to measure only ‘argumentation’ elements, resulting in a relatively simpler form compared to comprehensive writing scoring rubrics; and that I personally participated as a co-researcher in the argumentative writing project in the United States, directly experiencing the rater training process and actual scoring, these criteria were applied to the present study.

Due to missing pre-test scores for the total scale, analyses involving pre–post changes in total scores were based on 89 students with complete data for both time points. In contrast, pre- and post-test scores for the evidence and counterargument components were available for 120 participants.

Figure 1. Pre-Post test results

The students showed a noticeable but limited improvement in their argumentative writing skills. Most of their pre-test scores fell within the lower to mid-range, suggesting considerable potential for further development. Overall, post-test scores were higher. The score differences between the post- and pre-tests were mostly positive. The same trend was observed for the major components of argumentative writing. Students showed notable mean increases in their evidence and counterargument skills. By the end of the semester, they were able to provide stronger evidence and address opposing views more effectively.

Students’ use of ChatGPT varied in terms of frequency and quality. Scores on the positive-use subscales—Content Elaboration (CE), Organization and Coherence (OC), Meta-cognitive Use (ME), and Collaborative Refinement (CR)—ranged from very low to relatively high, indicating that some students used ChatGPT only sparingly for higher-order writing support, while others used it extensively to generate ideas, structure arguments, and revise their reasoning. The average scores on this index were in the low-to-moderate range. However, the scores varied widely, indicating substantial individual differences in constructive AI-supported writing practices.

In contrast, the risk-oriented subscales, Surface Revision (SR) and Dependence (DE), tended to be somewhat lower on average, although again with notable variability. Some students appeared to use ChatGPT primarily for quick surface-level fixes or direct copying, whereas others reported a more strategic use of ChatGPT. Descriptively, most students had slightly higher positive than risky use scores.

As a whole, the students improved their argumentative writing outcomes. Improvements were also found in sub factors of argumentation such as evidence and counterarguments. Yet, the pre- and post-test data demonstrated marked differences across individual learners. This variation provides a solid platform for further research in this field. This enables academics to investigate how different patterns of ChatGPT use are related to growth in critical parts of argumentative writing.

2. Cluster-based comparisons of learning gains

To explore naturally the occurring patterns of ChatGPT use, a k-means clustering analysis was conducted using the six subscale scores (‘SR’, ‘CE’, ‘OC’, ‘ME’, ‘DE’, and ‘CR’). A three-cluster solution was chosen to represent roughly “low”, “moderate”, and “high” levels of ChatGPT use and sophistication, based on the relative profiles of the clusters.

I then compared clusters on pre-test total scores (‘Pre_Total’) and total gain scores (‘Gain_Total’) using one-way ANOVAs. Although there were small numerical differences in means across clusters, neither the difference in ‘Pre_Total’ nor in ‘Gain_Total’ reached statistical significance (both $$p > .05$$). In other words, students’ overall writing gains did not differ reliably as a function of these broad usage patterns.

This suggests that simply belonging to a certain cluster of ChatGPT usage (e.g., more vs. less use overall) does not, by itself, strongly differentiate students’ overall writing improvement. These findings motivated a shift from unsupervised clustering toward more theory-driven groupings based on the quality and balance of ChatGPT use.

3. Balance of positive vs. risky ChatGPT use: extreme-group comparison

I focused on the Balance_Index—Positive_Use_Index-Risk_Use_Index—which captures the extent to which students’ use of ChatGPT is dominated by constructive, higher-order strategies versus more superficial or potentially problematic ones.

To amplify contrasts, I selected the bottom 25% and top 25% of students on the Balance_Index as “Low_Balance” and “High_Balance” groups, respectively. These two extreme groups were compared on: total gain score (‘Gain_Total’) and evidence gain score (‘Gain_Evidence’).

Welch’s t-tests showed that the High_Balance group had higher mean gains in both total score and evidence than the Low_Balance group. However, these differences were not statistically significant ($$p > .05$$). In other words, there was a consistent trend in the expected direction—students whose positive use of ChatGPT outweighed risky use tended to improve more—but the magnitude of the difference, given the sample sizes ($$n \approx 22–29$$ per group) and variability, was insufficient to reach conventional significance.

Figure 2. Mean gain scores (post–pre) in Total and Evidence by Balance group

When considered alongside the cluster results, the findings imply that rough classifications do not produce substantial differences in the overall progress. This prompted further analyses that focused on specific argumentative writing components and more precise indicators of constructive use.

4. Positive ChatGPT use and gains in evidence

Constructive uses of ChatGPT, such as expanding reasons, identifying counterpoints, and refining claims, are closely related to the development of argumentative evidence. Based on this, I evaluated whether the Positive_Use_Index predicted improvements in the evidence components.

Students were divided into three groups based on tertiles of Positive_Use_Index: low, mid, and high positive use. A one-way ANOVA was conducted with ‘Gain_Evidence’ as the dependent variable and Positive_Use tertile as the independent variable. The analysis revealed a significant effect of Positive_Use_Index on Evidence gains the omnibus F-test was significant ($$F = 3.27, p < .05$$). Visually, the distribution of ‘Gain_Evidence’ across the three groups is illustrated in this boxplot.

Figure 3. Distribution of gain_evidence

As shown, students in the high Positive_Use group achieved the largest average gains in evidence, followed by the low and mid groups. While the exact pattern across the low and mid groups is modest, the overall ANOVA result indicates that students who engaged in more extensive and constructive use of ChatGPT exhibited significantly greater improvement in providing evidence in their argumentative writing.

These results suggest that students’ patterns of ChatGPT use shape the development of particular argumentative writing. This implies that higher-level generative or reflective practices are more strongly associated with these improvements than with overall writing performances. A one-way ANOVA indicated a significant effect of Positive_Use_Index tertiles on evidence, with students in the high tertile demonstrating the largest improvement in evidence. The descriptive statistics and ANOVA results are shown in Table 3

Although the overall pattern suggested that a higher Positive_Use_Index was associated with greater gains, the mid group did not outperform the low group in evidence. This non-linear pattern indicates that more positive use is not automatically better and that only consistently high levels of elaborative and reflective engagement may translate into robust improvements. One plausible explanation is that mid group students sometimes adopted ChatGPT’s suggestions without sufficient critical evaluation, leading to revisions that were more fragmented or even confusing at the level of evidence construction. By contrast, some low group students may have relied more on their own reasoning or course materials, achieving comparable gains despite lower reported AI use. These interpretations remain speculative, but they highlight the need for future research that triangulates survey-based indices with log data and qualitative analyses of drafts to understand how different patterns of AI engagement actually play out in students’ argumentative writing processes.

A one-way ANOVA revealed a significant effect of Positive_Use_Index tertiles on evidence gain, $$F(2, 109) = 3.27, p = .042$$. Students in the High Positive_Use_Index group showed the largest improvement in evidence (see Table 3).

5. Regression analysis: Positive use and gains in Counterargument

To further investigate the component-level effects, I examined the gains in counterargument (Gain_Counterargument) using a multiple regression model that included both initial performance and positive ChatGPT use, as well as their interaction. The model was:

The key findings from this model are Pre_Counterargument had a significant negative coefficient ($$\beta_1 < 0, t \approx -3.07$$). This indicates a clear ceiling effect: students who started with higher counterargument scores tended to show smaller subsequent gains, presumably because there was less room for improvement in their scores.

The multiple regression model predicting Gain_Counterargument from Pre_Counterargument, Positive_Use_Index, and their interaction was significant, $$F(3, 108) = 7.10, p < .001$$, explaining approximately 23% of the variance ($$R^2 = .23$$). Pre_Counterargument negatively predicted gain ($$B = -0.63, p = .003$$), whereas Positive_Use_Index positively predicted it ($$B = 0.05, p = .013$$). The interaction between Pre_Counterargument and Positive_Use_Index was not significant ($$B = -0.03, p = .444$$; see Table 5

The Positive_Use_Index had a significant positive coefficient ($$\beta_2 > 0, t \approx 2.53$$). Even after controlling for initial counterargument ability, students who made more extensive and constructive use of ChatGPT experienced greater gains in their counterarguments. This suggests that positive engagement with ChatGPT uniquely contributed to improvements in students’ ability to recognize, generate, or respond to counterarguments in their writing.

Third, the interaction term between Pre_Counterargument and Positive_Use_Index ($$\beta_3$$) was insignificant ($$t \approx -0.77$$). This implies that the beneficial effect of positive ChatGPT use on counterargument gains did not differ systematically across levels of initial ability. In other words, both lower- and higher-performing students appeared to benefit from positive ChatGPT use, although high initial performers still showed smaller gains overall due to ceiling effects.

From a pedagogical perspective, this pattern suggests that constructive uses of ChatGPT can support the development of complex argumentative moves such as addressing counterarguments, and that this support is not limited to only the weakest or strongest writers.