Devs are frustrated with AI coding tools that deliver nearly-right solutions

According to a new survey of worldwide software developers released on Tuesday, nearly all respondents are incorporating AI tools into their coding practices — but they're not necessarily all that happy about it. The report, part of an annual study conducted by developer help site Stack Overflow, reveals that 78.5 percent of …

[Anonymous Coward]

I needed a simple parser for something (just a few lines of regex and some manual character checking, really) and was too lazy to do it myself so I thought I'd give vibe coding a chance. I don't know if I just got unlucky with the task I'd chosen or what, but it took ages to get Copilot to generate working code, I ended up having to read and understand the code it did output just to formulate my own theories of what wasn't working so I could guide it, and in the end there was a very easy problem it simply couldn't fix (but pretended to by generating the exact same code that was already in the file, without changes), which I ended up having to implement manually (literally just adding an extra `c == ","` in an existing check. Yes, that simple.) Overall it was an arduous, grueling task and I felt like I was playing in a three-legged race. I don't know where people get the patience to vibe code regularly, I was miserable the entire time. Ultimately I'm glad the option exists and I'm glad it's something I can reach for when performing tasks I don't want to do (like mucking with regex or SQL), but it really is a last resort for me for most tasks.

[Anonymous Coward]

Well I have used it for regex and it got 90% first time. I needed to do the last 10%. I will get down voted for suggesting this but currently I am finding Grok good for short specific pieces of code. But the problem I find is in describing what I want. You have to tell it what a human would just know. Even if I use something like Q in Vscode it doesn't seem to take account of what already exists beyond what is in the editor window, although I think they have been working on this as it has been improving and reads more of the files now.

[PerlyKing]

Re: But the problem I find is in describing what I want

This sounds a bit like... requirements gathering? Which sort of makes the AI prompt a higher level language, but with non-deterministic output.

I wonder how LLMs would do in the International Obfuscated C Competition? ;-)

[martinusher]

Re: But the problem I find is in describing what I want

Got it in one, there.

Any programmer will (should?) tell you that the problem isn't writing the code, its writing the specification for that code. Specifications are written in the same human language that people use to create poetry, literature and other works of art, its very powerful and expressive but also potentially really imprecise. Writing precise specifications is a bit of an art form itself and is often a lot more work than actually writing code. (If it were that easy then much of the legal profession would be out of a job.) There are obviously lots of relatively trivial situations where formal steps can be shortened or even skipped entirely but part of the programmer's skill is knowing how far you can push this safely. (I've spent a lot of my career cleaning up after often very clever people who never quite understood where that line was, or even if there was such a line.)

Testing code is also a big part of making software (at least for embedded software). There needs to be a formal test plan and often some kind of harness to stress the product. Here you'd be very unwise to use the same system -- or even people -- to test a piece of code that you used to write that code.

[Anonymous Coward]

It would be interesting to run this survey from a C-Level perspective. I'm sure the answers would be very different.

Where i am there is a 'Use AI or prepare to be shown the door' style policy. The problem is, no-one has shown us how to use AI effectively, so we're kinda stumbling around in the dark. I use co-pilot as a fancy autocomplete mostly, but when I have tried it for serious tasks it's either led me down the wrong path, or has taken me so long to prompt it properly I may as well have written the bloody thing from scratch!

I'm on the older end of the team age range. Watching my younger contemporaries using AI reminds me of how my grandparents used to watch me program the VHS....

[Wang Cores]

I use it to tart up my resume.

Way I see it, if all you need to escape the consequences of straight up lying is a disclaimer, you can just hide it in white text saying: "experimental. Items on this resume may not be accurate."

[LybsterRoy]

It seems to have worked for Rachel from accounts!

[Anonymous Coward]

Rachel just has her strings pulled. I don't think Rachel actually comes up with the ideas or policy. That is probably true for the entire cabinet.

[Smeagolberg]

"Rachel just has her strings pulled. I don't think Rachel actually comes up with the ideas or policy. That is probably true for the entire cabinet."

Indeed, and for most politicians, come to that.

[JpChen]

“ The problem is, no-one has shown us how to use AI effectively,”

This is what I see at my place. Loads of people playing with it, but no real idea how to properly drive it - and no consistency on how people use it. FWIW, I’ve had some training, done my research, and put the hours in to figure out how to get the best out of it. Context is key - with good instructions files, decent prompts, and choosing the right model for the job, I’ve found the output to be quite impressive and a massive timesaver. Personally I just see it as yet a higher level programming language.

I’ll caveat all that with the fact that I’ve spent nearly 30 years in the game so have the ability to properly assess and evaluate its output. I do worry about the next generation of devs who won’t even be able to read the code AI produces, let alone understand it. Perhaps the paradigm will shift so we stop caring about the actual code and treat it more as a black box, the emphasis shifting more towards specification and verification.

[Anonymous Coward]

"I do worry about the next generation of devs who won’t even be able to read the code AI produces, let alone understand it."

Very, very true. But that problem already exists in general life. People read things in papers, on the net or on the news and believe it without thought. Well, it's all going to come back to bite, code and news. AI like humans lies. Often the AI regurgitates our lies. But what else would it do? It's not AI it's LLM selecting the most probably responses according to training.

[Anonymous Coward]

> Watching my younger contemporaries using AI reminds me of how my grandparents used to watch me program the VHS....

One wonders if your youngers are actually getting productive, effective use out of the thing, or if they're just putting on a show for upper management; not quite malicious compliance, but at least "just following orders" and collecting the paycheck. I'd hardly blame them.

Your grandparents' VCR may have been flashing 12:00 since the last power cut, but I'd expect they also knew a lot of other things you didn't at the time. Certainly they knew enough to have a more experienced person handle it. A lesson, there.

[Anonymous Coward]

HR and other bureaucracies are using it and loving their extra "productivity" I'm sure. They can now easily compose an essay complete with references to 2 years old emails, that totally waste everyone's time instead of answering a question.

[Anonymous Coward]

Older end of team range here - company massively pushing AI use.

With Copilot need to put effort into the prompts.

Sometimes results are good, sometimes dire.

Often on dire results, you can give it a nudge & it does better on next attempt (e.g. you tell it what it got very wrong (e.g.non existent interfaces, methods) & send it off to have another go )

Once it has produced something roughly workable you then tidy up & tweak the remaining faulty code.

It is OK for (after a few prompts) getting somewhere near the code you need.

In some easy cases it can get it right first time, but its on more complex or obscure stuff that it often goes badly wrong, so I find it best to prompt it on subsets of a task rather than the whole complex task in one go.

I mainly treat it as a way to generate a rough & ready code scaffold, that can then be manually refined - essentially a flexible templating tool.

Sometimes it saves time, other times it probably ends up longer than the coding from scratch approach - after a while you learn when not to use it & when it's worth a punt.

Treat it as an over enthusiastic error prone suggestions aid & you should be OK.

[X5-332960073452]

so 'clippy' on steroids then !

[vogon00]

Code vs Codebase

Full disclosure : I am NOT a fan of AI, and haven''t even bothered to try it for code generation. That said:

I can see the attraction of having AI write your code for you, but what most people appear to have forgotten is that you still have to debug it. IMO, debugging real human code (aka 'Actual Intelligence' code) rather than the artificially generated stuff must be easier, as there will be fewer errors or 'hallucinations' to deal with¹.

I can see that AI generation may be a way to accelerate production of a codebase, but not to a trusted codebase. Getting a codebase to trusted and mature status requires commitment and effort (AKA test and/or debugging). This may be a 'old-skool' attitude, but AI ain't anywhere ready for use until it can write the code AND reasonable unit tests. Why do I say this? Because you have to understand the code, or the intentions behind it, before you can write good unit tests.

I'm not even going to look into the repeatability of AI generated code. I bet there will be subtle or not-so-subtle differences every time you ask for the same result with an identically-phrased request.

¹_{Where does this come from? I'm porting some legacy code (cpp, bash) over to python and it's hard to keep the code working when you're inexperienced with python like wot I is.}

[rafff]

Re: Code vs Codebase

"Because you have to understand the code, or the intentions behind it, before you can write good unit tests."

In my experience* good unit tests depend on understanding the intent and the interface, not the code. The tests should be blind to the code; change the code and the tests should still be valid.

* About 60 years of it.

[Mk10]

Re: Code vs Codebase

I asked CoPilot to generate a COBOL copybook for a file record layout. I had to type in all the field names, provide each field length and the data format. Some fields needed to be redefined for sub-fields, etc. But prompting CoPilot to do that exactly as I wanted was challenging. Finally, it produced a copybook to my liking. I could have simply typed all that directly to create the copybook. And would have finished much sooner.

[Dan 55]

In my case it takes my prompt, somehow comes to the most obtuse interpretation possible but one which still could be tenuously called correct, and then presents me with a code fragment which is utterly worthless. I then have to prompt it a few more times to get it to interpret my original prompt correctly. If it hasn't gone off the rails by that time, I have a code fragment that I can mess around with some more to make it fit the rest of my code.

If it could manage to get it right first time it would aid productivity, but after all the prompting I doubt there's much difference between using a LLM and finding the solution myself... apart from the data centre's electricity, water, and arbon footprint and my atrophying brain.

[JLV]

Here's a funny one I noticed, on repeat, today. I read somewhere that at least one thing they are good it is spewing up reasonable HTML UIs. And this a subject that I suck at, I typically have to read a lot, experiment a lot, before having something semi-serviceable that a CSS person can look at later. I any case, I typically immediately forget about it and feel like an idiot. So LLMs can shine polishing up 10 lines of HTML.

OK, here goes. I have a div - let's call it #parent that surrounds 2 other divs. I want to do something CSS-y via Tailwind, but I figure I can be lazy and ask CoPilot.

So I have 4 lines: parent, child 1, child2 and the tag closure for #parent on line 4.

Me: Copilot, make that red text bold with Tailwind.

Copilot: -- adds the some Tailwind class, saving me much time.

screen: no bold, no red text either.

me: ???? And then I notice that Copilot in all its wisdom has added a </div> closure for #parent online on line 1. Giving me malformed HTML and nuking any CSS effects on child1 and child2.

You'd be miffed at an intern for getting this wrong and my success rate getting it to do Tailwind has been pretty much limited to it styling forms correctly. Which was semi-impressive until I realized it cribbed the entire styling direct from the Tailwind doc on input tags... Which I would likely have managed just as well. For the rest, it tends to very self-assuredly adds a mish mash of Tailwind classes that typically do nothing on every prompt and I learn even less than if I refreshed my Tailwind knowledge, which really isn't that hard to do.

[breakfast]

Given that people talk about it being good with context, AI's fierce appetite for leaving brackets and markup blocks either unclosed or pre-emptively closed is surprising to me. Really shows that there is no actual logic to it.

[JLV]

Yeah, surprising to me too. I may have just selected just line for #parent or all 4 the two times it happened. This was just a playground bit of code, however, if #parent was on bracketing a large amount of lines then that would have more difficult to do anyway.

Quite possibly, with a better prompt approach - I just started this html playground as a testbed - I'd have gotten better results, but at some point you have to wonder at the time tradeoff between a) learning to prompt and not learning to code and b) learning to code.

Keep in mind: all this is fluid - what shortcomings are true now might be solved 2 years from now. My bigger concern is that I have an idiosyncratic coding style, partially out of habit, partially from the problem domain, and I don't feel all that comfortable trusting the LLM to stick to it rather than going back to everyone else's "best practices". In my case I use Django but eschew using its ORM because I mostly deal with introspecting "arbitrary" databases with very high table counts and their own custom metadata tables. I can't have everyone else's "best practices".

Writing 10-50 lines of code under tight supervision, especially on syntax I am unfamiliar with (I'll review it later) is my preferred use case and it's achieved little with that. The one thing it has been good at is filling in the gaps of my shell knowledge, except that I am on zsh and it never seems to remember that either, which sometimes bites like when it uses bash array syntax.

But I admit my prompts are undisciplined and I will need to be more deliberate with them. Copilot on vscode has things like context flags like #codebase and pre-rolled "prompt files" that look promising.

[breakfast]

The thing I find is that firstly I tend to think in code, so I can probably code a solution faster than I can write a prompt for most of the platforms I work with.

The situations where it takes me a while to get them right, I get that I could use a generative approach and get a script that does it, but if I spend a little longer and learn how to do it myself I've got a lot more than a script out of the endeavour - I have a bit more of a handle on a new tool or platform. To me that's worth a bit of time.

At some point somebody will build a small deterministic AI tool that works somewhere between an LLM, a linter and a documentation search, and that could be really useful, but the randomness of current tools and the fact they're wrong more than they're right (when I use them, at least) really makes it a gamble whether or not they're useful for any given scenario, or even useful for the same scenario twice.

[Headley_Grange]

I get that the examples here aren't in the context of large multi-disciplinary projects, but the C-suites might not get that and will just read "told AI what to do and it gave me code". Some of what I'm reading rings true for human coding too - spend time getting the spec right and you'll get a better product and possibly faster. What it all ignores is that coding is a small fraction of the development process and the quality, usability and supportability are built in at every step - specification, design, review, unit test, integration, system test, acceptance test....etc. If you're not writing code with these in mind then it will be a problematic to say the least to get it into production and support it. I guess the AI proponents will say that given time AI will learn much of the other stuff but I'm a bit sceptical.

[Anonymous Coward]

C Suite

The C suite will find out the hard way a few years too late. They will lose developers and in a couple of years find their code quality has declined and they are spending the money on debugging and have disgruntled customers. Followed by disgruntled shareholders, who were disgruntled previously when the company wasn't moving into AI as fast as they thought they should. Someone in the C suite will be the scapegoat after scapegoating a couple of managers first.

[Headley_Grange]

Re: C Suite

Microsoft has been using its disgruntled customes to debug its declining code for years, its gruntled shareholders are happy enough with the results and the C-suite suits take home a decent salary and bonus.

[Doctor Syntax]

Re: C Suite

C-suite suits take home an indecent salary and bonus.

FTFY

[Anonymous Coward]

Hype

The problem is the usual one; the technology was over-hyped. Had it been left to developers to pick up organically it would've been better but of course we had management types pushing it in order to impress the big bosses by telling them how productivity was going to be improved by 80% or some high number. Then the usual spin to pretend they met their stated objectives. Reality is a much lower number. But ... it does help.

[Dan 55]

Re: Hype

... sometimes. For small values of help.

It'll do great for your technical interview questions though, which have little to do with the real world.

[Anonymous Coward]

So this article was timely...

Being fairly new to salt I'd been struggling to get some automation put together, and this article popped up.

I've been skeptical of LLM output for useful work, but figured I'd give it a try on this specific problem. After all it was a well defined, small, problem which I can be pretty confident has been solved many times before.

I was pleasantly surprised at how well my prompts were answered, with working examples of *most* of what I needed. I could then add on an extra request and more working snippets of config were generated.

I could read, understand, and verify that the code snippets did what I wanted (total end product was under a dozen lines) and it certainly took less time than I had already spend on documentation.

It even suggested a workaround when salt refused to use the correct end command, and that workaround config was also correct.

So for a simple problem which I expected to already be solved, and probably documented somewhere, it worked quite well.

Part of my frustration is that salt's documentation is generally missing the woods for the trees.

Does that mean I am now a stochastic parrot evangelist - no, but it might have a place. And when what you need is a parrot (or a rubber duck) it might well fit the bill (or beak).

[DrXym]

Well yeah

AI is a handy tool for samples and throwaway code. I've used it to knock together simulators that need to handle some response or whatever for testing purposes. It doesn't matter if the code is a bit broken or needs fixing up because I'm not going to throw the world at it. At the same time, AI code is generally terrible - verbose, mangled, inefficient and insecure. Sometimes it mashes together the APIs from projects with breaking API changes between versions. E.g. use it with Jetty or Mongoose at your peril. I would *never* trust the code unless I could understand what it was doing, review it for mistakes and fix it. That requires knowledge and expertise. I cannot understand the mentality of companies who think "vibe programming" is going to work out well for them because it won't.

[herman]

It will prolly all go the way of Rational Rose and Rhapsody eventually - into the trashcan of good intentions.

[Scene it all]

I heard that people were using AI to write software so I thought I would give it a try. I asked a well known AI chatbot (NOT ChatGPT) to write a bitwise OR routine for a DEC PDP-8, a machine from 60 years ago that I know very well. Now, the PDP-8 only has a Two's-complement ADD instruction and a bitwise AND, along with some things like complement, etc. The correct answer should take under a dozen instructions, taking advantage of DeMorgan's Rule.

It got it wrong. It just said to use the Two's Complement add which is "close enough" to an OR, through in certain cases a more complex algorithm would be required.

[JavaJester]

I'm sorry Dave. I'm afraid I deleted your production database

Beware trusting AI agents too much lest you lose your production DB because the AI agent deleted it.

[frankvw]

It taught me a lot

I have always sucked at JavaScript development. So as an experiment I turned to ChatGPT and let it write me a few simple website features to point me in the right direction.

Boy, did I learn a lot.

The first version of the code was OK. Then I asked it to add another feature to it. That was OK up to a point and then the whole thing started to behave erratically. It turned out that the Great Magic AI Oracle had used the same variable name for the original functions and for the additions. Uh-huh.

Once I got it working, I tried to use it on a smart phone (I'd been testing on the desktop so far). That opened up another whole can of worms; on the smartphone Javascript appears to run in strict mode; on the desktop it doesn't. Issues like scope, scope leakage and race conditions that didn't present on the desktop now suddenly materialized. What ChatGPT had given me did run on the desktop, but it was still bad code, in that it included a lot of the bad, sloppy coding practices that are typical of an inexperienced and incompetent coder.

Bottom line: I learned a lot about Javascript with this silly little exercise. Takeaways: code that runs can still be badly written code; debugging is more educational than coding; and ChatGPT appears to be better than I am at JavaScript development because it knows more about the syntax than I do at this poinnt in time, but the logic of the code it produces is flawed more often than not.