ChatGPT Takes a Stats Exam: Does AI Make the Grade?

Artificial Intelligence (AI) is no longer just a futuristic concept; it’s here and shaking up the world of education like never before. Imagine having a tutor that’s always available and can answer questions in mere seconds. Sounds promising, right? But what if the free version of this AI tutor isn’t as smart as the paid one? That’s exactly what researchers Monnie McGee and Bivin Sadler aimed to find out in their study. They pitted different versions of ChatGPT against each other in a battle of stats trivia to understand how free and paid AI options stack up. This experiment isn’t about AI acing exams—it’s about seeing if this technology can truly be a game-changer in education, especially for students who might not have access to the fancy stuff.

AI in Education: The Great Debate

Since its launch, ChatGPT has been the talk of the educational town. Should we ban it, embrace it, or use it with care? Schools are wondering if AI could be the great equalizer, helping students who need extra help. With ChatGPT offering everything from free to $20/month versions, the question arises: Do these options provide equally effective tutoring? If not, could the digital divide get even wider?

Meet the Contenders: ChatGPT3.5, ChatGPT4, and GPT4o-mini

To see how these AI versions perform, researchers put them to the test against a 16-question statistics exam designed for first-year graduate students in stats. Picture a classroom filled with eager faces, each staring down at a nerdy math challenge. Now, imagine replacing those faces with three versions of AI—one is the retired and supposedly less impressive GPT3.5, followed by its more advanced siblings, GPT4 and the newbie, GPT4o-mini. The goal? To see how each would do and how their answers stack up against human grad students.

Exam Time: AI Under the Spotlight

The results? Well, if AI were a student, GPT3.5 would be the one sneaking emojis into essays. It scored a measly 41 out of 100. GPT4, at the opposite end, scored a respectable 82, while GPT4o-mini held its ground with 72. These numbers suggest that the free versions didn’t quite cut it, especially when the questions involved anything visual—like readings from a chart. GPT3.5 visibly struggled with visuals, much like trying to explain abstract art without knowing it’s upside down.

The Art of Chatting: More Than Just Scores

Numbers tell one part of the story, but the real plot twist comes with analyzing the AI’s “chat.” Researchers used tools to analyze the text, looking at word frequency, reading level, and even the topics covered. It turns out, GPT4 isn’t just smarter in math but also speaks in more understandable and cohesive sentences. Remember those times when a chatbot seemed to drift away in nonsense town? Yeah, GPT3.5 was in that zone a tad too often.

Reading Level and Legibility: A Balancing Act

Legibility matters, especially for an AI tutor. Can students understand what’s being said without feeling like they’re reading Shakespeare in a dim-lit room? The study measured reading levels of responses and found most AI outputs required at least a high school diploma to understand, some even a college education. It indicates a need for AIs to simplify the language when requested.

An insightful detail? The complexity of AI responses often matched the complexity of the prompt it was given. As a result, a prompt set at a college level tended to elicit a response of similar difficulty, which could be both a blessing and a curse depending on who’s doing the asking.

More Than Math: Topic Modeling and Relevance

The AI’s answers were dissected to see what topics they revolved around. With fancy methods like “topic modeling” (imagine using a magnifying glass to look for hidden themes), researchers found GPT4 and GPT4o-mini had a knack for sticking to relevant and coherent statistical topics, unlike GPT3.5 which veered off-track now and then.

Real-World Implications: The AI Tutor of Tomorrow

This is not just an academic curiosity—it has real-world implications. If educational institutions want to leverage AI as a personal tutor, they need to ensure equitable access to the more capable (often paid) versions. It begs the question: How can schools bridge this gap without breaking the bank? Could AI one day become as common as textbooks, used in every classroom?

Practical Tips for Your AI Experience

So, if you’re considering using an AI to supplement your learning, here are some tips:

• Precise Prompting: The clarity and context in your questions can directly affect the clarity of the answers you get.
• Exploration of Paid Options: While the free version might be tempting, consider what’s worthwhile for your educational needs.
• Expect Some Fluctuations: AI isn’t perfect and can vary based on prompt ambiguities and context.

Key Takeaways

• Performance Gaps: There’s a clear performance difference between free and paid AI platforms with GPT4 leading the pack.
• Reading Levels Matter: AI responses often match the complexity of prompts they receive, sometimes requiring higher education to comprehend.
• Future of Education: AI as a full-fledged educational tool remains promising but requires resolving issues of equitable access and accurate responses.
• Prompt Strategy: For better responses, include context and clear wording in your prompts.

ChatGPT and other generative AIs hold the promise of democratizing education, offering personalized learning at a global scale. Yet, for this vision to become a reality, performances across free and paid versions must align closer, ensuring every student gets the chance to thrive, no matter what their economic situation is. In the journey from classroom chatbot to indispensable assistant, the road is promising, yet filled with learning curves.

If you are looking to improve your prompting skills and haven’t already, check out our free Advanced Prompt Engineering course.

This blog post is based on the research article “Generative AI Takes a Statistics Exam: A Comparison of Performance between ChatGPT3.5, ChatGPT4, and ChatGPT4o-mini” by Authors: Monnie McGee, Bivin Sadler. You can find the original article here.