'How do I reset my router' isn't in LLM corpuses. An alliance of telcos wants to change that A joint venture comprising carriers SoftBank, Singtel, SK Telecom, Deutsche Telekom and e& Group plans to develop a large language model (LLM) they will use to automate customer service for the telco sector. The Global Telco AI Alliance, as the entity styles itself, wants to create a specialized LLM because the participants …
I think Singtel also owns Australia's Optus so new lows in support can be expected.
After last year's Optus router outage fiasco I probably wouldn't let them reset my router. :)
...based on my - admittedly limited - understanding of LLMs and how they work.
So, as far as I understand it:
- To train an LLM it has to be able to make statistical inferences with a high degree of accuracy.
- To improve the accuracy of the statistical inference, you have to have a large dataset.
- The larger the dataset the more links the LLM can make between words and phrases and the higher the likelihood that some outcome is "correct" based on the fact that this outcome appears very often in the training model and other outcomes do not.
- Small datasets lead to weird behaviours from the LLM because there's not enough signal for them to start selecting for meaning instead of noise.
Given these facts (and feel free to correct me if we have any data-scientists in who know better) it's unsurprizing that if you ask big LLMs questions that are highly technical that have very few examples to draw from in the dataset they're likely to come up with silly answers. A fairly generic question like "how do I reset an *thing that needs resetting*" will probably have hundreds or thousands of examples of "You reset a *thing that needs resetting* by..." but if the thing is sufficiently rare it might have none at all for the specific thing you're asking for.
It doesn't seem to be in the nature of LLM's to go "Fuck if I know." but instead they tend to come out with something plausible sounding based on the data that is in their training set, but which actually bears no relation to reality.
So, ok, to try and fix that problem, we train the model on some very specific data that does contain all the information about your specialist field.
...but that's a small dataset. Which leads to weird behavior from the LLM because there's not enough in it for it to build useful statistical inferences.
Sounds to me like what they actually need is the operators manuals for the equipment they buy. Which I have to assume they have already, because they'll need it to train the model.
Perhaps they should try having their employees read them.
True, but after you've done that, you need to log in and reconfigure it. I don't envy the customers who don't know that the experience of being told by a chatbot mangled instructions of configuring an ISP-supplied modem. I pity their technical friend or relative even more. Having recently spent hours on the phone to customer service for a friend's ISP to get them to allow the modem to talk to their network, even though I could see that it had already managed to communicate and be issued an DHCP lease, I know the chatbot won't be doing that for anyone.
"Sounds to me like what they actually need is the operators manuals for the equipment they buy"
The printed manual for my Multitech modem (1990s vintage) was over 100 pages long, including a troubleshooting section. By 2008, the equivalent manual for an ADSL router was a 230 kB HTML file just containing a blow-by-blow of what each config dialog did, with no explanations or troubleshooting section. Sadly, except for some enterprise level kit, operators manuals are now a thing of the past -- supplanted by "user forums" where you can't trust the information to authoritative. So replacing tech support with a statistical parrot bot may not actually make that much difference.
And the willingness to admit it's out of ideas it part of the training and weighting. Most of the chat-bots are heavily incentivized to produce a credible sounding response to every prompt, but that isn't a universal trait of LLMs.
The QUALITY and specificity of the "small" training set is more critical. Overtrain with a small set of even slightly inaccurately tagged data and your results will get "weird" as you put it. But plenty of the smaller models outperform the bigger GPT-x families when provided targeted and high accuracy training material.
Realistically though, I question if an LLM is the right tool for these applications. I suspect the Telcos will have to pour more work, money, and compute hours into this than a non-ML framework that just provides a guided list of options and answers. You might be able to "copilot" the people programming the rules in to speed things up, but plugging a chatbot into your non-technical users is going to annoy them, and LLMs are notorious for only being able to reach partial utility before you hit a wall of diminishing returns.
The smartest response from an LLM to this kind of problem is to direct the questioner to the website of their router manufacturer. That would a perfect (and universal) answer if router manufacturer websites were designed so that a person could easily find the answer from there.
But no, let's try and teach the LLM about every kind of router ever made (or rebadged) by anyone.
Oh, and for added brownie points, the instructions for a factory reset should include a clear description of how to change the router access password away from the default and a warning of the dire consequences for the owner's bank account if they fail to do so.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
