Enhancing University-Level English Proficiency with Generative AI: Empirical Insights into Automated Feedback and Learning Outcomes

Purpose – This study is meticulously designed to ascertain the effects of LLM-generated feedback on university students' writing proficiency. We focus in particular on examining the improvements in essay revisions, the degree of student engagement with writing tasks, and the emotional journey students undergo during the writing process. Utilizing a randomized controlled trial, we aim to draw a clear comparison between the experiences and performance of students who benefit from AI-generated feedback and those who do not, providing a comprehensive understanding of the potential of LLMs in fostering writing skills within higher education.

Design/methodology/approach – Our methodology incorporates a significant sample size, with 918 university students enrolled in English writing courses, offering a robust data set for analysis. The students were randomly assigned to either an experimental group, which interacted with feedback produced by the GPT-3.5-turbo LLM, or a control group that proceeded without such technological assistance. We assessed the impact of AI-generated feedback not only through objective metrics, such as automated essay scoring systems, but also through subjective indices, including detailed student surveys that captured motivational levels and emotional states. This holistic approach to data collection enabled us to gain a comprehensive understanding of the influence of AI-generated feedback on student writing development.

Findings – The incorporation of AI-generated feedback into the revision process displayed a significant improvement in the caliber of students' essays. The quantitative data indicated notable effect sizes, while qualitative feedback from students highlighted a surge in motivation and a more positive emotional experience during revisions among those who received AI feedback. These findings underscore the transformative potential of generative AI in enriching the language learning experience and in supporting the academic advancement of students at the tertiary level.

Originality/value/implications – The outcomes of this research carry far-reaching implications for the field of language education. The integration of generative AI tools promises to reshape pedagogical and assessment practices, leading to a more individualized, rapid, and accessible learning process. With these technological advancements comes the responsibility to ensure ethical implementation, which involves maintaining fairness, transparency, and adherence to educational equity. It is vital to navigate these ethical dimensions thoughtfully as AI becomes more embedded in educational practices.