AI coding tools are like that helpful but untrustworthy friend, devs say Software developers largely appreciate the productivity improvements they get from AI coding tools, but they don't entirely trust their output, according to a survey conducted by AI coding biz Qodo. As a result, some potential productivity gains get lost to manual reviews deemed necessary to check the AI's work. Qodo offers " …
Isn't this a dream team for those 76%? A "junior" doing the grunt work and an experienced staffer doing a 100% code review.
That said, it's limiting the development of "human" juniors that one day would become the experienced staffers.
Not really, depending on what the "junior" is doing. If they're putting in prompts until the thing compiles, then sending it to the reviewer, then the reviewer is picking out lots of bugs and not getting any real work done because they're cleaning up after the LLM. Of course, the junior could be doing a better job, actually writing some code that works and having someone reviewing it, and if they can manage that by using the LLM then more power to them. Unfortunately, in my experience, most usage of the LLM ends up being counterproductive because it introduces bugs that no human would write, meaning whoever ends up having to remove those bugs is going to take a long time and the reviews have to be much longer because we're searching for things that would never be an issue in the real world. Some of the time, that's the original user of the LLM whose productivity suffers as they try to use something that should save them time but doesn't, and sometimes, it's the reviewer who receives things that obviously haven't been tested and either needs to review it and report the problems, review it and fix the problems, or throw it away and write it from scratch so the patch can go out now (the most dangerous option because they may not have another reviewer to send it to).
In traditional code reviews, there were various pieces of code that didn't have to be reviewed too strongly because you could reasonably assume that they had been written correctly enough and tested enough before it got to review as long as the person who wrote it was someone you knew and trusted. Of course, that's not the most professional approach, and ideally, every line would be ruthlessly scrutinized, but in many environments, that didn't happen because speed was an important factor. LLMs have introduced the ability to fail in places where professionals would not. That means the really detailed code review is being done, which would be an improvement if the code quality was the same or better, but it isn't because we're now dealing with the possibility of ridiculous errors having been written in by the LLM which even an incompetent human wouldn't have done. There are probably some people who have found ways to use an LLM to write functioning code, but unfortunately not as many as people who think they have but are wrong.
Reading between these lines:
"There's a small minority of power users, who tend to be very experienced developers, who are seeing massive gains – these are the 10Xers. The majority of developers are seeing moderate gains, and there’s a group that’s failing to effectively leverage the current AI tools and is at risk of being left behind." --- Itamar Friedman, CEO and co-founder of Qodo
Mr. Friedman is cleverly trying to create the (mis-)impression amongst potential client MBAs with top down planning power that for the best coder drones (formerly doing humans), AI writes perfect code the first time, while for 2nd rank or lower coder drones it doesn't write good code. It's clever of Mr. Friedman because he knows that those MBAs want to believe that scenario.
how Google boosts search relevance by incorporating contextual signals and personal info.
Despite Google's processing of massive amounts of personal information, Google has not boosted its search relevance. Contrarily, it clogs my search results with unwanted links, and shoves what I want to the bottom of the results, or omits it altogether.
If "AI" code helpers perform as badly as Google, then they are not worth the trouble, regardless of what any surveys report or white papers say.
Depends on a lot of specifics. If it's something Google either invented (e.g. Go), had a strong influence over (e.g. Python) or merely represents a lot of answers on Stack Overflow (e.g. Java), then Gemini will often provide accurate answers. For Google technologies, it will have been trained on carefully curated datasets.
"One good method for dealing with the inherent flaws is to start a session by prompting the agent to review the codebase structure, documentation, and key files, before then giving it the actual development task"
Am I alone in reading that as "pour your IP in to the AI that belongs to someone else"?
It depends whether you trust whatever system you're using to not collect it. Most of them do claim not to store or train on the data you send to them, but I don't trust that. Neither do my employers, who have decided that AI coding tools are useful* but that they need a contract to ensure that their code is only processed by one where there is a legal agreement not to take the code fed into it. I have some more trust that that agreement is being followed. You do have the option of local models if you're worried about that and still want to run one.
* I'm still not sold on them. They do not work well when making changes to anything large, and that's pretty much all of the things I do. The only situation where they're capable is writing the code for small functions you don't want to write, except that I've had to clean up from times where someone did that and it still went off the rails, for example when a colleague used some LLM to write a function to escape characters in a URL which happily mixed HTTP escaping and HTML escaping, and this was Python, so the right thing to do would be to just call the standard library function that someone already wrote to do HTTP escaping.
It is a good idea to start every sesion by feeding in your entire codebase, again - because then you are burning up input tokens as fast as the machine can eat them. And pay for the privilege, of course.
If the LLM was retaining your code then you'd not need to pay for all those lovely expensive tokens EVERY session.
So you can trust the LLM not to permanently ingest your data because that would lower your payments to the model provider (assuming you aren't trying to do it on the cheap with a "free" service).
anything more than a useful tool for doing autocomplete or finding examples or refactoring, and you think you'll be able to use it for developing mission-critical, tested, maintainable line of business software then I have a Tower Bridge to sell you.
My cow-orker's use of a machine-learning coding tool, which is basically regurgitated copypasta from Stackoverflow or stolen open-source code, often results in dubious code without sufficient error checking.
He often also has no real understanding of what it does and so is reliant on the tool the next time for a similar but different task.
I'd like to see less machine learning and more human learning.
They can program, and they do a have a lot of domain-specific knowledge.
However due to time pressures from above which they lack the personality attributes to resist, plus a bit of laziness (which is admittedly one of the three Wallian virtues), they have a tendency to resort to cra-poliot when they don't know the solution instantly.
For the record I've had a former coworker who was quite capable of writing non-robust code without assistance. The problem being it works 80% of the time. Until it doesn't.
Using an LLM do write code. You need a specific AI which is specified in code writing. Instead they built generalist LLMs which everyone tries to use as expert systems. At least the medics tend to take an off the shelf one then teach it what it needed to know. Though one I read about recently was not very good at spotting cancers. it wasn’t subtle enough. Hunan eyes were much better. In medicine there’s always a human in the loop on those.
I have used AI dev tools and good & bad points.
It can speed up "grunt work" such as writing unit tests as it quickly learns the pattern, so can produce boiler plate code you just need to tweak slightly.
However hallucinations are an issue, it does like to invent non existent functions etc..
I find it better for simple tasks rather than anything too complex.
It also does not seem to cope well with evolution of a language.
e.g. I have to support code running on different versions of SQL Server (in the real world lots of people do not always upgrade to the latest & greatest)
Even when stating in the prompt what version of SQL Server is to be used, I often find it creates code using functionality inappropriate for that version.
e.g. TRIM did not appear until SQL Server 2017 (before that you had to do a combo of LTRIM and RTRIM to remove leading & trailing space on string data "tidy up" code)
However it will happily produce code samples using TRIM (when requested task includes "tidy up" of text data) even though you specified a SQL Server version predating 2017.
Which is odd as syntax / functionality changes per version of SQL Server are well documented (I know because I occasionally have to check what version a certain function was available from!).
So I am guessing that although the version specific data is there in the AI somewhere it is not always being fully referenced when code samples generated
> So I am guessing that although the version specific data is there in the AI somewhere it is not always being fully referenced when code samples generated
Or perhaps there is a misunderstanding of what goes on inside an LLM - this belief that it has stored those language specs in some neatly checkable form and then actually runs checks; as though it was looking things up in any way that is analogous to your deliberately re-reading the function lists.
The AI is trained on code written by me and people like me. It's not some genius, it's the average of all of us. We leave it behind every time we write some new, original code.
It's not terrible as a code review tool - running it across a PR it will often catch one or two problems alongside making up some completely nonsensical suggestions, and it doesn't get tired or bored with reading code as most humans do - but I wouldn't leave it unsupervised for anything because the useful stuff is way less than 50% in my case.
I often suspect the people these reports call "10X" developers were bad to start with so the LLM can do more lifting for them and they don't have the discriminating eye required to recognise the bugs it is introducing. I'm sure a lot of us have worked with people like that.
All code needs reviewing regardless of who wrote it and regardless of unit and e2e tests passing.
I guess I shouldn’t be surprised that there seems to be a trend to entirely automate this or skip it altogether.
Depending on the project and the task, a good code review doesn’t just look for the smaller issues but can question code that passes muster in terms of doing the job and being syntactically robust, but could be ; for example; very inefficient.
A lot of these more advanced reviews require intimate knowledge of the entire stack, external services, obscure business logic etc.
Whilst it should be noted careful planning and smaller tasks can mitigate this to some degree, shipping code that hasn’t been reviewed by experienced humans is not a good idea.
Just cast your memory back to 2011 when they tested IBM Watson by humbling Ken Jennings on Jeopardy.
Anybody can do an easy test of coding proficiency anytime by simply reviewing "Advent of Code", the coding challenges go back to 2015.
December 1 2015: Not Quite Lisp
https://adventofcode.com/2015/day/1
"Santa was hoping for a white Christmas, but his weather machine's "snow" function is powered by stars, and he's fresh out! To save Christmas, he needs you to collect fifty stars by December 25th.
Collect stars by helping Santa solve puzzles. Two puzzles will be made available on each day in the Advent calendar; the second puzzle is unlocked when you complete the first. Each puzzle grants one star. Good luck!
Here's an easy puzzle to warm you up.
Santa is trying to deliver presents in a large apartment building, but he can't find the right floor - the directions he got are a little confusing. He starts on the ground floor (floor 0) and then follows the instructions one character at a time.
An opening parenthesis, (, means he should go up one floor, and a closing parenthesis, ), means he should go down one floor.
The apartment building is very tall, and the basement is very deep; he will never find the top or bottom floors.
For example:
(()) and ()() both result in floor 0.
((( and (()(()( both result in floor 3.
))((((( also results in floor 3.
()) and ))( both result in floor -1 (the first basement level).
))) and )())()) both result in floor -3.
To what floor do the instructions take Santa?"
My experience (very experienced developer):
I use AI (in particular, a couple of LLMs, not coding agents) every day, but as an “AMA desk.”
It’s like having a really, really effective personal assistant nearby. I can ask for coding suggestions, documentation assistance, debugging help, etc.
I have to review any code that I get, VERY carefully. It tends to be fairly “naive,” and can often contain silly bugs (like hallucinated API calls).
But it’s like having a direct Stack Overflow feed, from about eight or nine years ago, when people rushed to answer any question, within seconds. The first answer was often not perfect, but might have the information I need.
The important thing, is that I can ask ANY question that I want, no matter how silly, and get an IMMEDIATE, highly relevant, answer. The fast response is the important part.
