Re: I really wish "hallucination" hadn't stuck.
To put it in plain language, the bullshit generator is working a little too well.
Given that humans have serious difficulties in agreeing, writing down, and verifying requirements, and it is the human intelligence of human developers that can unravel the mess and (sometimes) produce software that does what a client wants, which usually involves some degree of domain knowledge, and common sense, the odds of any sort of LLM that has been "trained" to produce human-like output, but lacks the fundamental essence of actual humanity, actually producing such useful work, beyond the task of producing boilerplate code (which can usually be automated anyway, if it's a tedious enough of a job to make it worth doing so), approaches zero.
(tl; dr; - AI has no useful use case)
it was the great Larry Wall who coined the three tenets of programming - laziness, impatience, and hubris:
Laziness: If we can't be bothered to do those repetitive tasks (such as producing boilerplate code), then we write a tool to do it for us. Asking "AI" to do it leaves us with the job of checking and verifying it, which is even duller a job than writing the boilerplate in the first place, so the correct solution (not "AI") is to specify and write a tool that does it deterministically.
Impatience: If the code takes too long to run, then the programmer will find a way of making it run faster. Sure, you could get AI to do it for you, or to optimise your algorithms, but again, you then need to verify the results. You'd be better off reading Knuth, and doing it properly. It's not like efficient algorithms have only just been discovered, and ones that AI invents are likely to either not be faster, or to cut corners, and produce incorrect or approximate results. I'd be astounded if "AI" can ever come up with any new categories of algorithm that are improvements on existing ones, based upon the inputs of existing algorithms, because that would imply a category of mathematical proofs that humans have somehow missed. Doubtful. Again, if, as a developer, you are unable to optimise your code, you need to go and learn how to. "Get gud, scrub".
Hubris: Excessive pride, achieved by writing well organised, readable, maintainable, well-commented code. Yeah, "AI" ain't doing that. "ChatGPT, write me some gibberish to comment how and why this code does what it does and how it solves the domain problem". Yarp, narp.