Generative AI in Academic Writing Assessment: Insights from a Case Study in Higher Education

While the integration of AI technologies in education is growing, a gap remains in the empirical literature regarding the effectiveness of AI-assisted grading and feedback. At a university in Hong Kong, an initiative was launched to employ generative AI to provide formative feedback on students’ written work prior to final submission. Generative AI was tasked with assessing draft essays from 150 students. Each essay was also independently marked by three lecturers at the institution. Inter-rater reliability analysis revealed a high correlation between the AI-generated scores and those assigned by human markers, both in terms of individual scores and overall ranking. However, qualitative data from student interviews and teacher reflections on the AI feedback revealed significant concerns. Key stakeholders expressed skepticism about the AI’s ability to deliver clear, contextually relevant, and pedagogically meaningful guidance that improves student outcomes. It is clear from both the quantitative and qualitative findings that while AI can achieve a high degree of accuracy in grading, the human element remains essential for delivering feedback that is both meaningful and contextually relevant to students’ individual learning needs. The success of AI-assisted assessment initiatives depends not only on technological reliability, but also on the engagement and perspectives of students and teachers as key stakeholders. Ultimately, sustainable implementation of such systems requires ongoing collaboration with those directly impacted, ensuring that AI serves to complement, rather than replace, the invaluable expertise of educators.