You wins some you lose some
Having recently gotten myself a Claude sub as well, I thought I'd give Claude Code a serious try.
First, a little project I hacked in years ago to extract info from Steam. It worked well, until one library was no longer supported, attempts to fix it failed in the horror that was the original source, Steam change its API and I left it with a shrug of regret. Enter Claude, which "figured out" a work-around and got it to work again (because it was able to scour the insane amount of HTML and JS and embedded div blocks and all the stuff I had no clue about, because I never got into the web dev and though HTML should have stopped at b1 and a tags)
It worked so well I had it integrate a number of external scrips I used to mangle the output, and then a few nice-to-haves. Wow! Great!
Emboldened, I thought I'd go for something more involved: and actual RAG and document analysis system for our ancient archive of statements, bills, what-not. I didn't want that out there, so use Claude to build the code for the local running of one of the freely available LLMs. Might make my old GPU melt, but let's see what happens.
Oh dear. Turns out that just reading in and scanning/OCR-ing the thousands of docs was the easy bit - there's a great CUDA enhanced tool for that. OK, shove that into a RAG and let's run queries over it, just like they advertise is sooooo simples.
The results were laughable. And as of writing this, still not working, after days and days of "vibe coding". Code that worked suddenly regresses. Queries returning information that isn't even close to correct - despite it "telling me" that it found and is accessing the right information. It's just doing LLM things which is random prediction and which is about as likely to be correct as me picking the Powerball numbers. It forgets instructions and prompts, so I have to constantly inject them again, because it doesn't have enough "memory". Ingestion of data can take an entire day with the hardware I have available - and it will happily make some minor change of a few lines of code and have me do the whole thing again, no matter what I prompt it. (and gods I want to murder a techbro when I get that "you're absolutely right, I'm sorry" nonsense. You're a fucking machine! Not Marvin the paranoid android!
So, small chicken feed? Yeah, not bad. Anything serious? Tell em, they're dreamin! Not without an army of elves to hand-massage the output. Oh wait, you just fired all your elves, did you? Good luck, then! And even more good luck to the customers who paid money for that vibe coded swamp.