ChatGPT becomes ChatRepair to automate bug fixing for less Boffins at the University of Illinois Urbana-Champaign have enlisted ChatGPT, the OpenAI chatbot that responds to written instructions, to repair software bugs without breaking the bank. Chunqiu Steven Xia, graduate research assistant, and Lingming Zhang, an associate professor of computer science, give away the surprise …
To be fair, frequently whoever wrote the original code didn't understand the problem either.
Testing assistance can be useful. That's what fuzzers do, after all. If an LLM-based system can generate valid unit or functional tests, that's one of the few areas where I'd be happy to have it. But suggesting fixes? Sure, let's avoid making developers learn anything, or think about why the code is wrong in the first place and perhaps find occasions for remedying anti-patterns, refactoring duplicated code, or providing better defensive mechanisms.
The vast majority of papers and comments I've seen in favor of using LLMs in software development (from people across the AI enthusiast-skeptic spectrum) boil down to "I'm a lousy developer and I don't want to think about what I'm doing". Of course they typically use the euphemism "more productive", which is the scripture most often quoted by the devil in this field.
Unfortunately, it's an approach I've seen in humans as well. It's exemplified by this quote:
"We observed that including useful information such as test failure error or even the failing test name itself can provide additional information like the type of bug (e.g. null pointer exception) and the expected correct behavior of the code,"
I've seen people take this and fix a bug, but never change what actually happens. Sure, they've put in something that checks for null and doesn't crash the program, but they haven't asked questions like "where did the null come from", "is that null behavior we accept or not", and "could anything else result in a null ending up here which isn't the case I've just added in". Debugging isn't a process of looking at what it is doing and making it stop doing that, but looking at what it isn't doing that it's supposed to and making sure it does that.
Only now you'll have to go back to ChatGPT to get the fixes for the bugs the previous fixes introduce which may also introduce bugs or break the original code further. That's going to push the costs up.
Then you have the question as to what you submit to ChatGPT for it to try and tix the bug: Do you submit all your code to ChatGPT? And in doing so, might you be exposing your code to others?
And will ChatGPT reply with code in the correct language version - or even the correct language, and where will it get the source from as it may not have anything to work from...
I know I've tried ChatGPT and Bard for code examples, and ChatGPT returned what might have been the correct answer, only it was in the wrong version of the language where as Bard... said it couldn't do that, Dave. Not sure who Dave was, but... it couldn't give me an answer.
However, the idea of using ChatGPT to bug fix is akin to putting all your eggs in the same basket. And exposing your code to the world: Don't forget that. Alternatively let those who will understand the response do the work and PR the ChatGPT response. Just to be sure.
"In order to do this, future work should definitely focus more on the dynamics between powerful LLMs like ChatGPT and human developers to additionally add human intuition and understanding of the code base for better combined bug fixing."
not the whole human! way to expensive and in-efficient, just the bits that do the work and a support mech.
then you would need to bug test the human part, bet that's going to take some time.
brain machine interface, come on Elon, plug me in!
Meta space? nah that will never work.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
