Usage-based pricing killing your vibe - here's how to roll your own local AI coding agents

With model devs pushing more aggressive rate limits, raising prices, or even abandoning subscriptions for usage-based pricing, that vibe-coded hobby project is about to get a whole lot more expensive. Fortunately, you're not without cost-saving options. Over the past few weeks, we've seen Anthropic toy with dropping Claude …

[DaemonProcess]

good

That's a great start. Ideally you want a model which has been tuned(trained) for a specific purpose such as they are much smaller, faster and don't make so many stupid errors. Don't forget you can still make a good start with a public model then copy/paste it into the local one. Choose a model that fits inside your graphics card memory, ideally. If you can't afford a big graphics card then you can still get going with a tiny tuned model. It's amazing what progress you can make with OpenCode and Big Pickle in under 4GB RAM and no GPU acceleration. The LLMs know the main language constructs so you don't have to keep looking up module APIs and syntax, which saves weeks work straight off the bat. If you are an experienced developer even from last century (like me) you will have sufficient knowledge and experience to be very productive. I've found that adding features/functions one at a time works well, if it starts hallucinating then simply quit and re-start with a new session. Now all I need is a killer app idea, I've been searching for one of those for 30 years...

[Bebu sa Ware]

"all I need is a killer app idea"

Seemingly variation on Mrs Glasse's (apocryphal) "first catch your hare."

"I've been searching for one of those for 30 years..." Sounds a little like Pellinore's pursuit of the Questing Beast. :)

[TheMaskedMan]

Re: good

"Now all I need is a killer app idea, I've been searching for one of those for 30 years..."

Haven't we all?? I'm in awe of the twisty minds that come up with neat ideas that seem so obvious after the fact!

I suppose we could try asking ChatGPT ... :)

[bitwise]

Eh?

Don't we all have more ideas than time ?

A few of them might be good.

[Anonymous Coward]

temperature=0.6?

[b0llchit]

It is in Kelvin and is required to get the quantum computing part of your computer to develop a concious mind.

[Richard 12]

Randomness

"Temperature" means how much random entropy is injected into the model.

In theory zero temperature should make them repeatable, giving the same output for the same input.

[bitwise]

Re: Randomness

In practice zero temperature is not quite repeatable (you do get repetitions but also some variation).

I think this is because the operations are done in parallel and so the order they are combined is not stable.

[Anonymous Coward]

That’s some risky network advice…

Telling users in a how-to to set “-host 0.0.0.0” seems dangerous as that has the server binding to any IP — which for some of the more fortunate could be publicly-accessible.

[O'Reg Inalsin]

Re: That’s some risky network advice…

Yes. To recap the instruction

If you're planning on running Llama.cpp and accessing it on another machine, you'll also want to add --host 0.0.0.0 to the command, which will expose it to your local area network.

Even if not publicly accessible its better to be safe in case routers, cellphones, or some co-habitants of of date Windows, on the same network become compromised.

It's better to open an SSH tunnel from ones local client to the remote host with

/usr/bin/ssh -N -L ${PORT}:localhost:${PORT} ${REMOTE_HOST}

(assuming ssh keys and ~/.ssh/config are properly set up) and then add

--host 127.0.0.1:${PORT}

to the server invocation command. "127.0.0.1" is localhost. And keep the server fire walled except for the SSH port.

[elsergiovolador]

Local

Local coding agent is the new chaos monkey.

[frankyunderwood123]

cost of the hardware?

If you don’t already have a 24gb gpu , we’re talking £1000.

I have tried LLMs on my 8gb gpu and it’ll surprise nobody that I could look up coding problems online faster than a model can assist!

I can absolutely see the attraction of local control, but the economics only work out if you already own the hardware required or are in the market for a new gpu for other reasons, perhaps gaming.

For the rest of us, £1000 gets you an incredible amount of tokens and access to the top of the slops.

[Richard 12]

Re: cost of the hardware?

For how long though?

The Github $39pcm plan would have cost £1000 in about 34 months. So including electricity and assuming a five year replacement cycle means the local hardware is pretty similar cost.

But the $39 per month plan is gone, replaced by something roughly double the price. Now the local hardware is cheaper after two to three years.

Yet Microsoft are still losing money at the new price. The cost per token is going to rise further. In a year that $39 pcm usage is going to cost much closer to $400 pcm, and your local hardware will be cheaper within six months.

[elsergiovolador]

Re: cost of the hardware?

£1000 until the rug pull.

Better spend that money investing in own hardware.

This is the same trap as what "cloud" companies did. Promise savings, once customers got hooked they can't get out of it and it is magnitudes more expensive than self hosting.

[steviesteveo]

Re: cost of the hardware?

It'd also have helped if we didn't have first crypto and then AI doing awful things to the supply for years

[werdsmith]

Re: cost of the hardware?

£1000 for a device that:

Is not only useful for running the local model.

When running a model it can be shared by multi user.

Does not hit token limits.

Can be used offline with sensitive information.

Can be used RAG/MCP with less security problems.

Can be tweaked and specialised to your own use case.

I’m sure there’s plenty more but local is the way that I am going, for most things there is no need to use big frontier models, but they are still there (for now anyway) if needed.

I would be more interested in the cost of having the same level of GPU and RAM in a service from one of the big cloud providers for those who have already committed everything to the cloud. Of course that doesn’t stop people running a Mac Studio on their desk.

Or there are some used Mac Mini M3 on EBay right now. 64GB/2TB for £399. That’s about the price of 2 x 32GB DDR5.

[vtcodger]

Color me skeptical

If, as Ed Zitron and others tell us, LLMs are not remotely economically viable for the big guys, why would one believe LLMs are economically viable for anyone? And it looks not only possible, but likely, that any online facing vibe code produced with today's technology will be yet another security nightmare. Something we don't remotely need,. Why not wait a few years for more insight into actual costs and risks before embracing what looks to be a costly, disruptive, and possibly risky technology?

I'm not 100% against use of LLMs. For research. Or hobby products. ... fine, have at it. But for serious business? Color me skeptical.

[Bugsy11]

Re: Color me skeptical

The Pentagon would beg to disagree with you. So color me skeptical about your post.

[jake]

Re: Color me skeptical

Have you not noticed who the dipshit is who runs the Pentagon these days?

The mind absolutely boggles ...

[steviesteveo]

Re: Color me skeptical

If you treat it as expensive software engineering device then having it on prem could change the calculation. You could find it justifies experimentation outside business hours better than the usage based models do. The economics of selling executions don't change but you're actually trying to get value out of your TCO instead of trying to find profit after the provider's fee

It's the same argument as databases- a serverless database costs you more the more you use it which is an interesting exercise in whether a query was worth the candle, as opposed to amortising queries across the life of the hardware - you simply wouldn't interact with systems on the different pricing models the same way

[Anonymous Coward]

agent sandbox guide please!

i haven’t dared to use slop machines in an agentic capacity yet, too much potential for things to go wrong.

[Anonymous Coward]

"Providing you have enough local resources to run them".

Yup. That right there is the actual problem.

[IvyKing]

Hmmm, sounds like my M4 Pro Mini with 64G of RAM should work nicely. Also explains the run on Studio's with the 512G memory.

[elsergiovolador]

I now wonder how much of the memory shortages are artificial. AI companies figured the pleb will run the models locally meaning they will not have access to precious data. Solution? Buy all RAM.

Regulators should step in, but they are still in wine and steak coma.

[steviesteveo]

There absolutely is ordered hardware with no place for it to go. We're way out beyond just in time with these orders

[jake]

Yeahbut ...

... what does it do for me?

Seriously, even if I temporarily ignore the non-zero chance of "hallucinating"[0], what does this do that is useful? Why do I want it?

_{[0] "Hallucinating" my left nut ... call it what it is. It's an automated bad results generator, NOT something I need or want running on my computers.}

[PM.]

I would like to see another tutorial. Thanks!

[hardboiledphil]

Why so dense?

The dense models code well on my M1 mac but they are horrendously slow. The Mixture of Experts I find pretty close for coding but have much more acceptable speed on Mac hardware. One big advantage of PI is the context by default is pretty small - in the hundreds of tokens. Even Opencode is ten thousand+ tokens before you've even typed anything and that pre-fill speed although better on M4/M5 is still going to be painful once you get a decent sized context.

[sedregj]

16GB GPU + 4 bit quant

Works nicely.