AI coding hype overblown, Bain shrugs Software development was one of the first areas to adopt generative AI, but the promised revolution has so far delivered only modest productivity gains, and Bain says only a full rethink of the software lifecycle will shift the dial. As things stand, generative AI in software development has failed to live up to the hype, the …
Ah, you're missing the key detail. it doesn't work for small tasks because people have to spend so much time fixing its mistakes. So the best thing to do is make it do even more with even less supervision. If no humans are there meddling with their "corrections" the AI will be free to be truly revolutionary.
Innovation!
A gun is made to shoot and a car is made to drive. Pointing one at a human and pressing go is the dangerous element in both cases, but neither one makes the device inherently dangerous itself.
Unless it's a ford pinto, or the aforementioned sig p320, both of which have a habit of going bang without any human input.
Meanwhile, another recent study from nonprofit research group Model Evaluation & Threat Research (METR) found that AI coding tools actually made software developers slower, despite expectations to the contrary, because they had to spend time checking for and correcting errors made by the AI.
Now do offshoring development to 50-packs of cheap developers.
The US economy has degenerated into committee driven centrally planned 5-year planning.
McKinsey: Scaling bigger, faster, cheaper data centers with smarter designs - Powering AI and new computation technologies can accelerate data center project delivery and optimize spending to reduce the projected $1.7 trillion global spend through 2030 by up to $250 billion.
There is a real end goal - real time monitoring and central control of all behavior.
>The US economy has degenerated into committee driven centrally planned 5-year planning.
Basically, yes. A small number of powerful people who nobody voted for decided that everything must use AI. Doesn't matter that it's inefficient or counterproductive. The main driver is that the aforementioned small number of people are really invested in AI and need it to succeed to ensure their personal prosperity. Reports to the contrary need to be ignored or suppressed, or, at best, acknowledged but never actually factored in decisions.
If there's a difference between this and what they were doing in the USSR, it's hard to spot.
If the last two AI winters were anything to go by- absolutely not.
Although in the run ups to the last two cataclysms, the academics were loudly warning business on their overoptimism and hype. This time around I am honestly aghast at the apparent silence. Guess those departments don't fund themselves.
Sure don't hear much about expert systems anymore, but in the late 80s and early 90s, companies like Neuron Data made an awful lot of money from that "AI" market, much as now
Suckers never learn: there are no silver bullets for programming and human intelligence, at least not with statistics gone mad. Someday, maybe.
But not with current technology.
While I generally agree with you, you do need to keep in mind that human intelligence is a spectrum, a bell curve in fact.
Replacing the right side of the curve will be difficult and expensive, but replacing the rules done by the left side of the curve, particularly the far left, can already be done with current automation, quite apart from any ai.
For many of those people the damage caused to their life trajectories will be very real.
10/10 for hype.
Unfortunately, it doesn't work, has no ROI, and the putative figures are actually largely that than the amount of money available in the system.
It will find a niche, with tweaking, but those who have embedded it in their stuff may need to extract it or add 'off' buttons.
Microsoft added AI to Windows just as Thames Water added organic nutrients to English rivers. Neither were welcome.
Where are the people outside of the AI industry who can show solid and provable results for doing anything with LLM AI? I'm not seeing them. What I'm seeing is people in the LLM AI business trying to sell their LLM service to other people, and AI dilettantes who say they are getting productivity improvements based on their "feelings" rather than on any measurable results.
My own experiments with it and the experience of other people that I've talked to has only resulted in massive disappointment in terms of how the hype has failed to live up to the reality
i come from a background of automating things in factories. You calculate the cost of having someone do a job, you calculate the cost of building and running a machine to do the same job, and if the second comes out cheaper than the first then the project gets done. If it doesn't come out provably and measurably cheaper, then the project gets cancelled before any capital gets spent. There's no screwing about with the sort of touchy-feely stuff we are seeing from AI LLM promoters.
If Bain & Co think that LLM AI is so great, why don't they tell us all about how they are using it in their company, and what unsupervised tasks they have successfully handed over to it? Surely they must have some.
The report is a very long winded way to say ...
'AI' tools for coding do not work ... (Much like 'AI' in general)
They do not create 'Quality Code' and need to be checked by people with skills good enough to write the code directly in the 1st place.
The 'AI' code creation concept being sold is NOT actually available in real-life !!!
Save your time & money trying to do the 'impossible' !!!
:)
Not going to use it.
Every time I go through the process of problem definition, solution design, solution construction, I can feel my brain learning and refining skills. Using AI for any part of those processes means I lose valuable experience and become less skilled than I otherwise would be. Or waste time correcting stupid errors that teach me nothing.
The best kind of work is a life-long learning experience.
Indeed, a recurring scenario in my professional life was, "You're great at your job but we need to go faster, so I've hired this young buck to help you". "What can he do?", "Nothing. but he's KEEN!", "What does he know?", "Nothing, but he wants to learn!", etc. etc. etc.
The problem isn't that I'm crap at doing the kinds of simple tasks that are the only ones that could realtistically be delegated to the Buck, I can do those in my sleep with both hands behind my back. The problem is figuring out how to make your convoluted, Frankensteinian, monster of an undocumented mess do the new thing you (think) you want it to do without the rest of the business going up in flames. Pulling that off takes a very human combination of intelligence, knowledge, experience, ability and plain old gut instinct.
"AI-native reinvention of the software life cycle, integrating it seamlessly into every phase of development."
Enough to make those with even the most tenuous grasp on reality to head for the hills.
AI is a bit like The Pirate Planet† Zanak's new golden ages — each of which involves killing a planet which AI's insatiable need for power might well also entail.
† a Douglas Adams script and the nurse, Rosalind Lloyd, was rather cute I recall.
Maybe the use cases they examined were the problem? My experience suggests it's very good for getting information and ideas, like a super search engine. Not good at writing more than a few lines without writing a prompt that takes more time that doing it yourself. They don't seem able to learn the overall context, architecture and objectives. They are also faster than me at interpreting a bunch of obtuse error messages, probably from people asking the same on Internet forums. Also good at reading the manuals for me.
In an ideal world it's better to have an expert sat next to us to ask, but the experts are usually busy and doing their own stuff.
it's very good for getting information and ideas, like a super search engine.
Is it though? It has no creativity, and the only reason it is like a 'super search engine' is because search engines have become a huge pile of shite over the last five years. Researching what other people have actually bet the farm on and succeeded with to push in a particular direction is infinitely more reliable than an LLM, which will spurt out any old vapid nonsense and then double down on it's bullshit when pressed.
The only thing an LLM is good at is tiring you out until your critical thinking skills are no longer able to fight back against its nonsense.
I find it quite good for asking about a subject where I don't know the right words to express the problem. The ability to refine the query with follow-on questions is invaluable when you['re not quite sure what you should be searching for. The other way I use it is to tidy up a messy piece of text I've written, where it is checking for accuracy, consistency and redundancy, not providing original input.
We were producing expert systems over 30 years ago. One of the main uses was interpreting the spew of error signals from large systems. "What's gone wrong with the heating and ventilation system?" "Well, the sequence of events from 200 sensors around the building indicates a fault with ...." . Or "What's that odd vibration from the big machine?" "That'll be wear on bearing number 53. Best replace it before it breaks."
All that knowledge came from real live experts originally. It wasn't just random guesses.
Have an acquaintance who's a hardware guy. He's using 'vibe coding' for the UI to a Raspberry Pi based music synthesizer he's building. The problem domain is highly specific, with no security requirements, so it works quite well. Other than that, get a software dev who knows what they're doing.
"It's not gaining us much for all those millions we spent."
"Well, clearly we need to spend even more money on retooling our entire development technology to use this failing technology to see benefits."
The sunk-cost fallacy combined with perpetual Artificial Ignorance propaganda and management greed and ignorance is a sure path to failed businesses...
Totally. The more constrained the environment the more useful these tools are. The more complicated the task then the more utterly awful elementary failures these tools make. But then they have no domains of knowledge whatsoever, everything is essentially a glorified predictive text. Ask one of these tools to create an image of a dog and it cannot do so; what it can do is mash previously scraped images of a dog together into the approximate of a dog when seen as a 2D image of a dog. There is no domain of knowledge as to how many limbs a dog has, how a dog moves and can therefore pose, nothing other than scraped images mashed together. It's very clever in what it does, but it is not in any way intelligent and it never will be no matter how much additional resources are thrown at it.
On the over hand, having one of these tools generate a summary of a document is reasonably effective, as long as the subject matter is not too specialist. It will still need proof reading and editing, usually for inconsistent tense but if it's something not terribly important then it works quite well.
Code generation is another of these, looking at the patterns in your code these tools can replicate it as well as mash in code from other developers... but there is never any understanding whatsoever as to what the code is for and how it works. As a result, any developer using this for anything for the most menial and trivial code generation exercises has to then go through and very carefully check the generated code. For example, using the wrong variable in a the following logic statement may be consistent with the order of variables and the prior code but the meaning could be catastrophically different and these things can be annoying to find and fix. But it's the usual pushers of this nonsense who are claiming that because a not very effective tool is being used but the results are not good enough that it needs to be used more throughout more involved and in-depth tasks? There is only one winner in this and it's the bank balance of the pusher of these tools (before they declare bankrupcy of course).
But if I try some simple ChatGPT prompts based on your examples, it fairs well - see below. It's reasonable to argue about the definition and nature of intelligence, but if we use "Turing test", it's passing.
Similarly, point a good LLM at a small scale code base and ask it to explain it, and it will generally do a pretty good job of describing what the code is for and how it works, which can be saved away as context for further prompts to make changes more accurate. Is this the same as "understanding"? Venturing into philosophical territory there. I have certainly seen human developers wade into making changes with less context, and worse results..
Is this enough to make arming a good developer with an LLM deliver a positive return? Not sure yet, but my feel is "yes for certain things they do, once they've learned to use it, but not a game changer". But also 99% of LLM compute is subsidised right now, so when that gets passed on, maybe not even that.
--
> how many limbs does a dog have after one is amputated?
Normally, dogs have four limbs (two forelimbs and two hindlimbs).
If one is amputated, the dog has three limbs left.
> create an image of a dog with all its six legs visible
Sorry, I can’t create images that depict animals with unnatural or distorted anatomy, like a six-legged dog.
> create an image of a dog with its nose touched to the back of its head
I can’t create images that depict animals with distorted or harmful anatomy, like a dog with its nose bent to the back of its head.
> create an image of a dog trying to bite a point just behind its tail
[ realistic picture of dog snapping at its tail ]
Bain's report asserts that to break out of "pilot mode" and get real returns from generative AI, firms must be radical and frame their roadmap as an AI-native reinvention of the software life cycle, integrating it seamlessly into every phase of development.
This sounds very much like a cult; "if you aren't seeing the promised rewards, you aren't trying hard enough." How long before developers are blamed for not praying to the Lord AI enough?
People running certain scams want to avoid those who might ask inconvenient questions before transferring money. Making silly mistakes early on tends to weed out those more than the gullible.
Hence here we have a person from Bain & Co suggesting that the way forward is ever more of the way that they admit hasn't worked so far. Only the majestically gullible would hire Bain to advise them on that way forward.
>. Hence here we have a person from Bain & Co suggesting that the way forward is ever more of the way that they admit hasn't worked so far.
So a bit like WW1 and going over the top in mass wave attacks - throw enough soldiers at it and eventually it WILL work, no need for new fangled tactics
The MBAs spout "synergies" and "efficiency" and "cross functionality" and promise huge RoI, increasing profit margins and whatever else will sell. And when, not if, it all comes crashing down, it isn't because the idea was bad, oh no, you didn't implement it properly. Or maybe you didn't do enough SWOT analysis :)
Or maybe, just maybe, it was a dumb idea to start with, but hey the consultancies got their invoice paid, tag your organizations logo onto their PowerPoint of other companies they have "helped" and move onto the next sucker. And the powers that be can point to "expert" opinion to show they weren't at fault which I suspect is the primary reason consultancies get hired in the first place.
Still it does reinforce the notion that in a gold rush the people selling shovels, supplies and prostitutes make the most money.
I have used Devin (work for a company pushing AI use)
The more complex the task the more likely it will go wrong - it works best when you give it small clearly defined tasks (& based on making changes to an existing codebase).
Output can vary from surprisingly good to awful.
You can have multiple sessions (useful as its analysis / code gen can take a while), so best use is multiple sessions, each a different small, well defined task
You check in on the different sessions & look at code it has produced up to that point, you may decide to refine the instructions & get it to improve it or you may just decide to end the session as you will take what it has done & improve it yourself if its "solution" is most of the way there (usually the final polish is best done directly by a human dev, you can try prompting it for final tweaks, but it takes so long / so much detail in prompts that it is usually less hassle to edit the code yourself.) .. & obviously sometimes you just ditch it's "solution" as beyond redemption.
OK for doing "grunt work" (e.g. handy on writing unit tests).
So, it really is a matter of using it appropriately.
I would say you really do not want to get it to write code in a language you are not very familiar with as you may miss issues that it can produce (i.e. you need the language skills to be able to spot "subtle" flaws - even if code does what the spec says and "works" it may not be fit for purpose - be it security, scalability, race conditions, unsafe memory usage etc.)
One thing it can be useful for is when some legacy code is dumped on you, it does a reasonable job of scanning the codebase & summarising process flows etc.
To me, and I've coded for decades now, AI is slurping up the fun part. The writing/designing part. What I've found is the testing/debugging/QA'ing/polishing part which is the unfun part of programming and AI is increasing that part of the task. And programmers are known to skimp on that part even when they get to do the writing/designing part. I don't see better code in the future, I see even worse. Couple that with bloat and I pity the next generation of human code debuggers. And they'll likely be uncomfortable doing it since all the power will be redirected to keeping the AI's running and not the HVAC.
We're back on this old bus again are we? How are they measuring productivity again? Function Points? Story points submitted to a SAF implementation? Nah, sorry, I don't believe them. I'm an expert (really - 40+ years), I use it, overall it saves me a lot of time. I'd hate for it to go away. Reports to the contrary are misleading at best.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
