Are AI Essay Graders the Future of Education? Exploring ChatGPT and Llama’s Potential

In an era of technological upheaval within education, where virtual classrooms and digital learning tools are rapidly transforming the academic landscape, the age-old challenge of essay grading meets a new contender: Artificial Intelligence. The quest to determine whether AI could grade essays as effectively as humans—and perhaps lend educators a helping hand—has become increasingly pertinent. In the study conducted by Anindita Kundu and Denilson Barbosa, the capabilities of large language models (LLMs) such as OpenAI’s ChatGPT and Meta’s Llama were scrutinized to explore this potential.

The Challenge: Traditional Essay Grading

Essay writing remains a core component of student assessment, essential to evaluating comprehension, critical thinking, and communication skills. Yet, grading these essays manually presents significant obstacles. Teachers, often overwhelmed by high student-teacher ratios, face daunting workload pressures, which can compromise feedback quality. Moreover, grading is not only labor-intensive but also time-consuming, leading to delayed responses and, consequently, hindered student development.

Besides logistical challenges, human grading can be inconsistent, potentially skewed by cognitive biases and subjective interpretations. Factors ranging from a student’s handwriting to an evaluator’s personal preferences can lead to variability, making the grading process less objective.

Enter Automated Essay Scoring (AES)—an appealing solution aimed at automating this process by leveraging cutting-edge natural language processing (NLP) technologies.

AI to the Rescue: The Rise of LLMs in Education

Over the past decades, AES systems have evolved from basic rule-based models to sophisticated machine learning technologies. Today, advancements in language models such as ChatGPT and Llama are revolutionizing these systems by pushing the boundaries of what AI can achieve in understanding and evaluating human language.

What are LLMs? Large Language Models are AI systems trained on vast datasets and capable of comprehending and generating human-like text. This training allows them to tackle tasks they weren’t explicitly trained for. In this context, they are used to assess essays by providing numeric scores and even offering explanations for those scores.

Putting LLMs to the Test: ChatGPT vs. Llama

Kundu and Barbosa’s research employed datasets like the Automated Student Assessment Prize (ASAP) to measure how these AI tools compared to human graders. Both models—ChatGPT and Llama—were tasked with scoring essays using the grading rubric applied by humans.

The Results

Interestingly, both LLMs exhibited stricter grading tendencies compared to human raters, often assigning lower scores. ChatGPT was found to be the harsher of the two, deviating more significantly from human scores. On the flip side, Llama demonstrated a somewhat closer alignment, though discrepancies remained prevalent.

Exploring Essay Traits

The researchers took a detailed look at specific essay features, such as the length of essays, use of transition words, readability, and the presence of spelling and grammar errors. Surprisingly, human raters often neglected these technical aspects, favoring longer essays despite errors. LLMs, by contrast, gave more weight to spelling and grammar precision.

Interpretations and Implications

The insights suggest that while LLMs don’t match human graders in overall alignment, they could act as valuable supplementary tools. They can accurately pinpoint mechanical errors—a task humans often overlook—which could enhance the grading process.

Toward a Synergistic Future in Education

The researchers envision a future where AI complements human effort, streamlining the grading process while human educators focus on the nuances of evaluating ideas and arguments. But reliance on AI also necessitates thorough considerations—such as the model’s training context, prompt specifics, and scoring method design—to ensure AI outputs are maximally beneficial and fair.

Key Takeaways

• AI’s Role in Education: There is immense potential for AI to alleviate the burdens of manual essay grading by providing consistent, unbiased evaluations, especially in handling mechanical and structural aspects.

• Current Gaps: While promising, LLMs still fall short of fully replacing human graders due to differences in grading strictness and alignment with human judgments.

• AI vs. Human Grading: The study found that humans often reward essay length and idea flow over mechanical precision, whereas AI leans towards accuracy in technical correctness.

• The Future: Transitioning to more effective AI essay grading will require enhancing models’ alignment with human evaluative tendencies and exploring how AI-generated feedback can be used to improve student outcomes.

Though the path ahead is rife with challenges, the integration of AI and human intelligence in education—not as a replacement but in partnership—holds captivating potential to redefine the learning landscape. As Kundu and Barbosa’s work establishes, with evolving technologies like Llama-3 continuing to outperform earlier versions, the journey toward more harmonious AI-assisted education is just beginning.

If you are looking to improve your prompting skills and haven’t already, check out our free Advanced Prompt Engineering course.

This blog post is based on the research article “Are Large Language Models Good Essay Graders?” by Authors: Anindita Kundu, Denilson Barbosa. You can find the original article here.