How AI is Changing Software Testing: Can LLMs Really Help Find More Bugs?

Software testing can feel like trying to find a needle in a haystack—tedious, time-consuming, but absolutely necessary to make sure the software we rely on is free of critical defects. Over the years, developers have experimented with various ways to improve this process, and now, with the rise of Large Language Models (LLMs) like ChatGPT and GitHub Copilot, there’s a new contender promising to revolutionize defect detection.

A recent study by Rudolf Ramler, Philipp Straubinger, Reinhold Plösch, and Dietmar Winkler explored whether LLMs can actually help testers find more bugs and improve software testing efficiency. The findings were striking: LLMs significantly boosted productivity in unit testing, allowing testers to detect more defects—though with some trade-offs. Let’s break down what the study found and what it means for developers today.

The Evolution of Unit Testing: Then vs. Now

Unit testing—the process of testing small pieces of code to ensure they work correctly—has traditionally been a manual effort. Back in the day, developers would write test cases one by one, trying to cover all possible errors. While automated testing tools helped improve efficiency, the process still required significant effort from human testers.

A previous study, conducted over a decade ago, examined how well testers could find seeded defects (intentionally planted bugs) in a Java-based system within a 60-minute timeframe. The results back then showed that testers, on average, detected 3.71 bugs in an hour. This study has now been replicated, but with a key difference—testers had the option to use LLMs to assist with test creation.

And the results? Testers using LLMs wrote more tests, achieved higher code coverage, and found significantly more bugs—but they also generated more false positives (incorrect bug detections that weren’t actual defects).

Putting AI to the Test: The Experiment Setup

To keep things fair and comparable to the original study, the researchers followed the same methodology but with the added ability to use LLM support. Here’s how the experiment was structured:

1. Participants were given a Java-based system with 37 seeded defects (and one new, unintended defect that was also counted after being discovered).
2. They had 60 minutes to write as many JUnit tests as possible to detect these defects.
3. LLMs like ChatGPT, GitHub Copilot, Codium AI, and others were allowed (though their use was optional).
4. Test cases were then analyzed to measure:
5. How many defects were found.
6. How many unit tests were created.
7. How much code coverage was achieved.
8. How many false positives were generated.

The participants mainly consisted of master’s-level students with experience in software testing and development. While they weren’t industry professionals (yet), earlier studies found that student results in similar experiments closely mirrored those of experienced developers.

The Results: AI Makes a Huge Difference (With Some Caveats)

So, does AI give testers an advantage? Yes—but it comes with a trade-off.

• LLM users created way more tests: Participants using LLMs wrote an average of 59.3 tests in 60 minutes, compared to 27.1 tests in the manual testing group—a 119% increase in productivity!
• More tests meant higher coverage: The AI-assisted group achieved 26% average branch coverage (with the best individual coverage hitting 65%), compared to only 16% in the manual testing group.
• They found more bugs: On average, testers using LLM support discovered 6.5 defects—nearly double the previous study’s 3.71.
• False positives increased: While AI-assisted testers found more defects, they also produced more faulty test cases—5.1 false positives per participant (on average) compared to 2.7 for those doing manual testing.

Perhaps most interestingly, the best testers using AI tools found more bugs than even the best manual testers in earlier studies. The top 10 AI-assisted testers collectively detected 31 of the 38 defects, while the top 10 manual testers in the previous study only found 24.

Why Does AI Boost Testing Performance?

The success of AI in test creation comes down to automation and speed. LLMs can generate test cases almost instantly, reducing the time it takes for a developer to think through different scenarios manually. They can also identify patterns and potential failure points faster, leading to more comprehensive coverage.

But, there’s a catch: LLMs aren’t perfect. One major downside is that they can sometimes generate incorrect unit tests—leading to more false positives and adding noise to the debugging process. However, the study found that this increase in false positives wasn’t due to AI making more mistakes—it was mostly because AI generated more tests overall, and with volume comes a higher chance of errors.

Practical Takeaways: Should Developers Rely on AI for Unit Testing?

✅ When AI Works Well for Testing:

• Boosting productivity: If your goal is to test a large amount of code efficiently, AI accelerates the process.
• Extending test coverage: More tests mean better code coverage, increasing the likelihood of catching hidden bugs.
• Helping junior developers: LLMs can assist less experienced coders by suggesting solid test cases and guiding them through proper test writing.

⚠️ When to Be Cautious:

• Beware of false positives: AI won’t always get it right—review the generated test cases critically.
• LLMs depend on good prompting: If your prompts are vague or unclear, the generated tests might be low quality. Knowing how to craft better AI prompts can improve test accuracy.
• Don’t blindly trust AI: While AI can automate testing, a smart human tester is still needed to validate outputs and fine-tune the testing process.

In essence, LLM support can be a game-changer for software testing, but it’s not a silver bullet. The best approach is a hybrid one—leveraging AI while applying human judgment to refine results.

Key Takeaways

• LLM-assisted software testing significantly outperforms manual testing in terms of test volume, code coverage, and defect discovery.
• AI helps testers write more tests faster, providing broader test coverage and detecting almost twice as many defects compared to manual testing.
• However, false positives increase with AI use, meaning testers must still review and refine results manually.
• The study suggests that LLMs mark one of the biggest advancements in unit testing efficiency over the past decade.
• Developers looking to integrate AI into their testing workflows should balance AI use with careful manual validation for the best results.

The Future of AI in Software Testing

This study shows that the way we do software testing is evolving. With AI-powered tools becoming increasingly accurate and sophisticated, the role of testers may shift toward reviewing and refining AI-generated tests rather than manually writing every test case from scratch.

Looking ahead, future research may explore: – How different AI models compare in testing performance. – The best prompting techniques for generating high-quality test cases. – How AI-human collaboration can be optimized to reduce false positives.

With AI reshaping software development, testers who learn how to effectively collaborate with AI will have a major edge. The future of quality assurance isn’t about replacing human testers—but empowering them with smarter, more efficient tools.

What do you think? Are you ready to let AI help with your testing, or do you still prefer the manual approach? Let us know in the comments! 🚀

If you are looking to improve your prompting skills and haven’t already, check out our free Advanced Prompt Engineering course.

This blog post is based on the research article “Unit Testing Past vs. Present: Examining LLMs’ Impact on Defect Detection and Efficiency” by Authors: Rudolf Ramler, Philipp Straubinger, Reinhold Plösch, Dietmar Winkler. You can find the original article here.