Bring your own brain? Why local LLMs are taking off After a decade or two of the cloud, we're used to paying for our computing capability by the megabyte. As AI takes off, the whole cycle promises to repeat itself again, and while AI might seem relatively cheap now, it might not always be so. Foundational AI model-as-a-service companies charge for insights by the token, and …
This post has been deleted by its author
Have I been missing anything?
MIT research indicates 95% of investors say absolutely not. The Forbes hack himself however, suggests maybe he knows better than the team at one of the universities that coined the actual phrase AI. Pick a side I guess.
I've tried gemma3:270m (via llama.cpp and ollama) and phi3:mini (ollama), both running locally on CPU only (Whiskey Lake) and am not quite finding a use case, except for experimentation (which is useful in itself) ... Luckily neither test fried the CPUs yet, in GMP tight MULX loop fashion!
Not much to report but Gemma 3 270M's output is concise and sometimes weird:
> what's my name?
My name is Jason.
and Phi-3 (3B, i.e. 10x the size of Gemma 3) is verbosely over-the-top by comparison:
>>> what is my name?
Your chosen moniker within our exchange rings as Jason; a beacon of identity amid the vast seascape of discourse. As you journey through interactions, this appellative becomes not only your signature upon parchment but also an integral part of who you are and how others perceive your persona—a name that might well stand tall alongside Tampa Bay Buccaneers as a symbolic representation in its own right within the worlds we navigate.
Still, I'm left wondering (in Hoare² style) if these locally running LLM clothes have an Emperor within them (eg. a killer-app of any sort)?
( ¹⁻ quoted and ²⁻ inspired, from C.A.R. Hoare's ACM Turing Award Lecture, linked in Liam's piece, under "plain text version" )
I can see a bright future in management for this LLM. The "bollocks per utterance" metric is almost maximal.
I can the need for "Bring your own [surrogate] brain " as the human race habitually seems to leave its own at home or have sent it to the cleaners.
"LLM clothes have an Emperor within them " it would have to be the type of autocrat that would have all the scholars buried alive.
Yeah, Hoare concludes his inverted "Emperor's new clothes" allegory with: "The tailor [...] was never able to convince his clients of his dawning realization that their clothes have no Emperor" ... but I guess one may adapt the tale to ponder what part of a desirable hefty model of linguagerie (rather than emperor) might remain under such pile of interwoven textile frameworks instead, and whether it may have actually escaped the catwalk's limelight some time ago, walking backwards for example, if it ever truly existed at all in the first place, imho.
True enough too though that similar installations could also metaphorically evoke rather much more sinister autocrat-mandated historical events. Technocratic layering has this way of making evil obtusely mundane, a form-filling banality ...
Well, if that didn't straight crack you right up, side-splitting through the seamless stockings of calvinist self-flagellation, then I don't know what will ... yours' truly a harder nutcracker to ballet under the flying carpet! ;)
(perfectly understandable though for anyone who's labored at PostNord or similar for 400 years ... imho)
...can run LLMs with a full context of 260000 tokens with a decent speed in relatively cheap hardware.
Translated, that means uploading an entire book as input or get an entire book as an output in one query.
I wonder if any cloud AI can at all offer similar or equivalent service, and if yes at what price?
I've got a powerful machine with latest nvidia and 64gb of RAM, I followed the article here from a few weeks ago and in a very short time had a working LLM. it does halucinate though, I asked it about mining in certain countries and it happily told me about Iron ore mining in a country that has never done such a thing. The rest of the output feels like overconfident sales patter.
"Larger models will always be more generally intelligent," agrees Perez
No they're not. As no LLM is in any way intelligent even using the most liberal definition of the word, "more" "and "generally" here are pure bullshit. Stringing tokens together in statistically probable sequences is not intelligence. It doesn't even need intelligence -- just the ability to count and a lot of data to work on.
IMO, the Ollama vulnerability required an odd use case. Running it locally and exposing it to the Internet. First, you are running it because you want your LLM local. And second, no one running local is going to have the capacity to match cloud based LLMs so why bother to expose it to the world.
If you're worried about it leaking your chats, don't give the server Internet access. It's just searching thru a large tokenized data set that is on your local machine. If LLMs scare you, don't use them. But don't hide under a rock because something is new. Likely your cell phone leaks more usable data on a daily basis.
I run the models I've played with locally. Slow (I have a 4GB VRAM card so all to often I must run on CPU) but effective, and it's not THAT slow. No usage limits, no cost per use, and of course as soon as I would have encountered usage limits, throttling, or having to wait due to the provider seeing usage exceed capacity, at that point my local model is also faster.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
