How to Outwit AI: Stopping Cheating with Codewrecking Tricks
How to Outwit AI: Stopping Cheating with Codewrecking Tricks
Artificial Intelligence is reshaping our world in so many ways—sometimes for better, sometimes not so much. One of the latest battles brewing in the academic world revolves around Large Language Models (LLMs) like OpenAI’s ChatGPT and GitHub’s Copilot. These AI tools can code up a storm and seemingly provide developers with superhuman powers. But when newbies in computer science classes use them to ace their programming assignments, are we witnessing tech magic or tech misuse? A group of researchers sought to find out how to curb potential cheating without making life more complicated for genuine learners.
Welcome to the fascinating domain of “adversarial perturbations.” Try saying that three times fast! Simply put, this clever concept involves tweaking assignment descriptions to trip up AI tools while still being legible to students. Intrigued? Let’s dive in to see how this works and what it might mean for the future.
The AI Cheat Code: Problem or Possibility?
Large Language Models (LLMs) like ChatGPT have clocked countless hours helping budding coders. Why? Because they’re astonishingly good at turning brief prompts into clean, functional code. While these tools are invaluable for professional developers, in the classroom, they might as well be holding a cheat sheet right under the table.
Given that educators don’t have the power to directly control or restrict these AI behemoths, some inventive choices had to be made. This research investigation zeroed in on a thorny question: Can we tweak assignment prompts to make cheating noon-fundae?
Let’s break down how the researchers approached this clever battle of wits.
How AI Fumbles in School
To address this challenge, Saiful Islam Salim and the research team took advantage of AI’s weaknesses rather than directly combating its immense power.
Measuring AI: How Well Does It Really Perform?
First off, the researchers sized up AI’s ability on basic programming tasks, selecting assignments from CS1 and CS2—the building blocks for any computer science student. Think of CS1 and CS2 as the Hogwarts of Computer Programming. From a total of 58 assignments and various tasks within them, the idea was to see how well bot brains solve problems. The preliminary results? AI was all thumbs in many cases. It turns out, LLMs often flounder with assignments that require interactions across multiple functions and classes.
Tripping Up AI with Tricksy Prompts
Next, the researchers set their trap. They engineered perturbations—changes to the prompt designed to confuse and mislead the models. Picture Tiny Aliar twisting and turning Rubik’s Cubes of scrambled code prompts, hoping to stump the learning assistants. It included everything from swapping out words and muddling characters with Unicode lookalikes to entire sentences going missing.
Could Students Catch On?
Finally, an engaging experiment came to light. Would students sniff out the gaffes slipped into their changes? Could they reverse them? Working with undergrad students who’d already taken these courses ensured the study was on the level of experienced players, making real-world findings more credible. Surprisingly, even when students did detect oddities, the AI assistants weren’t so quick to recover, leading to flawed or incomplete code.
The Results: The Good, The Bad, and The Not-So-Smart
Ultimately, the blending of all these perturbations decreased AI’s “correctness score” by a gigantic 77%. This impressive downturn showcases that deliberate prompt tweaks can poke significant, uh, “holes” in the AI solution parade.
Here are some specifics:
- GitHub Copilot performed best but still fell flat on face-sprawling issues in CS1, achieving an average correctness of only 51.5% on simpler queries and about 27% on more complex ones.
- The Project’s Ten Tricks: Simple tweaks like replacing letters with similar-looking Unicode characters can bamboozle AI marvels ominously. Although such changes risk being spotted, they still confused the machine more often than you might hope.
So What?
Why is all of this worth caring about? Basically, it could redefine how we teach programming in AI’s new age, setting guidelines for designing coursework that maintains academic integrity.
- For Educators & Students: Perturbations like these can help educators design more challenging real-world problems while ensuring students can learn effectively—without simply copying and pasting from bots.
- For AI Innovators: As AI grows, wizards designing these tools must consider safety nets against misuse, much akin to anti-malware features in modern software.
Key Takeaways
-
AI Can Fumble: Designed well, perturbations can greatly diminish AI correctness, indicating multiple functions can be tough terrain for models.
-
Subtle Changes, Big Difference: Simple swaps and removals are particularly deadly for LLMs, even if perceptible to the hawk-eyed students.
-
The Future is Bright: This kind of research provides instruktors with cool methods to thwart AI-assisted cheating. It also nudges AI engineers to rev up the learning capabilities of their models.
In a world where technology grows unstoppable, sometimes it’s the human touch—the small changes, the joyous tweaks—that keeps the world interesting. Aptly using these perturbations, educators can ensure that AI becomes a meaningful aide in learning rather than a shortcut to the finish line.
If you are looking to improve your prompting skills and haven’t already, check out our free Advanced Prompt Engineering course.
This blog post is based on the research article “Impeding LLM-assisted Cheating in Introductory Programming Assignments via Adversarial Perturbations” by Authors: Saiful Islam Salim, Rubin Yuchan Yang, Alexander Cooper, Suryashree Ray, Saumya Debray, Sazzadur Rahaman. You can find the original article here.