Navigating the AI Frontier: How Developers are Using ChatGPT on GitHub

When robots start writing code and fixing bugs, you know you’re living in the future—or at least walking very close to its edge. ChatGPT, OpenAI’s conversational large language model, is one of those futuristic gadgets that’s becoming a handy companion for developers worldwide. But what’s really happening when developers plug ChatGPT into their workflows, especially around the internet’s most-talked-about development space, GitHub Issues? Well, buckle your seatbelts because we’re about to take a deep dive into the captivating world of AI-assisted software problem-solving!

Understanding the Landscape: ChatGPT and GitHub

If you’re a developer, GitHub is probably one of your daily stomping grounds. It’s a massive platform where code is shared, tweaked, and where haggling over bugs is more common than ordering tacos on a Tuesday. But here’s an eye-opener: 92% of GitHub developers are now experimenting with AI coding tools like ChatGPT. This trend comes just as everyone and their grandma are talking about ChatGPT’s potential for writing and debugging code.

Yet, the big question is: What exactly are developers doing with ChatGPT? This study analyzed a whopping 1,152 conversations between developers and ChatGPT on GitHub Issues to find out just that. Let’s break down what they discovered.

Rolling Up the Sleeves: How ChatGPT is Being Used

ChatGPT as a Brainstorming Buddy

So what are those geeky developers and ChatGPT chatting about? Mostly: brainstorming! Developers are using ChatGPT to come up with fresh and innovative ideas. It’s a bit like having a clever friend over your shoulder, ready to help with brainstorming how to tackle a problem or design the user experience for your app—even if you’re a little stuck. This role stood out as the most popular, snagging 25.26% of all cases reviewed.

Keeping Testing on the Back Burner

Interestingly, discussions around testing are way less common. It’s like when you have broccoli collecting dust in the fridge, while the pizza gets gobbled up. Backend dev issues like API management dominated the conversations, yet for all you testing fanatics out there, it seems ChatGPT isn’t your go-to just yet.

Code Validation Takes a Back Seat

Further down the line is code validation—developers don’t often rely on ChatGPT to proof-check their code. Perhaps it’s because ChatGPT sometimes mixes up facts or isn’t the most reliable when it spits out code that’s about as trustworthy as a weatherman in a hurricane.

The Million-Dollar Question: Does ChatGPT’s Code Actually Get Used?

ChatGPT isn’t just talk; it offers code suggestions. Yet, the researchers found that only 12.78% of GitHub issues were resolved using ChatGPT-generated code. Out of that, merely 5.83% of the code was used ‘as-is’. Most took ChatGPT’s code, gave it a good polish, and then used it. Just goes to show how humans have not yet surrendered their keyboard scepter to a machine!

How Happy are Developers With ChatGPT’s Help?

Mixed Emotions Abound

Feeling fuzzy or frustrated after chatting with ChatGPT? Well, you’re not alone. Developers expressed positive sentiments in only 22.2% of developer-ChatGPT exchanges. For tasks like refactoring—making code cleaner and more efficient—developers gave a thumbs-up nearly half of the time. On the other hand, debugging got a measly smile from only 7.84% of participants.

The Devil’s in the Details

Fun fact: ChatGPT does reasonably well in data analysis, sprouting positive feedback about one-third of the time. Why? It’s all about context. Developers who provided rich details got much better, more tailored responses—something think about next time you’re framing a question.

Real-World Implications: What Can We Learn?

The study’s findings are a bit like Rorschach test results for AI in coding. There’s heaps of potential but several rough edges to smooth out. Here’s a peek at the broader picture:

Improving AI Tools

For researchers and software designers, it’s high time we dig deeper into developing AI tools that better meet developers’ needs. Whether it’s tailoring solutions to specific tasks or nudging ChatGPT to ask better follow-up questions, improvements need to happen to bridge that satisfaction gap.

Better Prompting for Better Results

Developers: a tip for you! Providing a little extra context helps ChatGPT greatly, avoiding its notorious generalizations—that might be less helpful than your Uncle Bob’s vague life advice. This means specifying your problem and including enough background to ensure you get the most valuable feedback.

Tackling Back-End Complexity

While ChatGPT shines in the data analysis arena, it tends to stumble when dealing with the complexities of backend development—those convoluted dependencies can trip it up. Teams and individuals working in these areas may need more specialized guidance and experience to sidestep common pitfalls.

Key Takeaways

• Developers mostly use ChatGPT for brainstorming: It’s a great partner for generating ideas, steering clear of mundane tasks like testing.
• Chat-composed code is often modified before it is utilized, emphasizing the current AI-human teamwork dynamic in coding.
• Satisfaction with ChatGPT varies, largely depending on how refined the user prompts are. The more details you give, the more helpful ChatGPT becomes.
• There’s room for improvement: Specific areas like backend complexity demand further developments in AI to make these machine conversations more fruitful.

In the grand scheme of software engineering, it seems ChatGPT is the friendly robot we’ve invited into our coding kitchen. While there’s still some fine-tuning needed to spice up the dish, it’s undeniably a helping hand that promises to redefine our culinary code creations in exciting new ways. So, why not take that leap into the future and let AI lend a hand in solving your development dilemmas? At the very least, it’s bound to make the journey a bit more interesting!

If you are looking to improve your prompting skills and haven’t already, check out our free Advanced Prompt Engineering course.

This blog post is based on the research article “Why Do Developers Engage with ChatGPT in Issue-Tracker? Investigating Usage and Reliance on ChatGPT-Generated Code” by Authors: Joy Krishan Das, Saikat Mondal, Chanchal K. Roy. You can find the original article here.