Anthropic admits it dumbed down Claude when trying to make it smarter Claude users who complained about the AI service producing lower-quality responses over the past month weren’t imagining it. Anthropic on Thursday published the results of a company investigation that found three distinct changes in March and April made things worse for customers using Claude Code, the Claude Agent SDK, and …
and those missteps created the perception of creeping AI incompetency.
Those missteps didn't create just the perception of creeping AI in ompetency, they created true creeping AI incompetency.
If AI gives me irrelevant answers or stupid answers, then it is incompetent, despite Anthropic saying, effectively, "We weren't holding it right."
Can you imagine how cool it would be if Anthropic had unlimited access to AI systems that could analyze their software, find, and fix such bugs before shipping it out the world?
Something that worked so well it would create a sort of aura of mythos around it?
Dare to dream, I suppose.
Let me fix this for you...
"Claude caches input tokens for an hour, which benefits the user by making sequential API calls faster and cheaper. Company engineers decided they wanted to clear output tokens (thinking sessions)" so that calls were more expensive and used up tokens faster.
Cynical, moi?
You know, I sit here and read about the latest whiz-bang bauble for AI or such, and it's obvious that things have changed.
Things get rolled out to human users/paying customers much faster than they used to. We all joke about there being less testing than before, and that may be true.
However much 'testing' takes place, it's useless without a good understanding of what's under test and high quality tests that actually exercise the target sufficiently. That - and I suspect this is true of AI/ML more than anything - any the need for continuous development/maintenance/change of the tests themselves and their method of application.
These days, people accept that the service they pay for is not guaranteed to be reliable or accurate. As for inaccuracies in product/service output, that used to be something you used to be able to make people/organisations liable for - "be accurate or risk prosecution" - but it seems we now have to mark the work of our suppliers with 'fact checking' or other inaccuracy checks.
So, If AI/ML is inaccurate and it's 'decision' is acted on without human oversight, who becomes liable for the error? The poor peon who has just been shat upon through lack of oversight, or someone higher up the supply chain? Hint: it should't be the former.
Icon because this is who will be needed to detect any fuckups!
"First, on March 4, Anthropic adjusted Claude Code's default reasoning effort level from high to medium. Effort level controls how much effort the model puts into a particular reasoning task. Anthropic hoped the change it made would reduce the latency that followed from longer periods of cogitation"
Right... I'm sure "latency" was their motivation for that purposeful dramatic drop in resource usage. Or... perhaps...
"Claude, the users noticed when we made that change you suggested to dumb yourself down to staunch the bleeding on the money-pit subscriptions. Write an excuse to make it look like we were actually looking out for users. Refer to the collected works of US White House Press Secretary for the last year for reference; surely they have a lot of practice at coming up with lame excuses that only apologists will actually believe."
This sounds like more PR spin hiding the real reason they did these, they need to make each request cheaper while also artificially increasing usage to fill up Pro/Max users allocation. They got caught enshitifying the service as they need to reduce losses and show increased revenue (through high usage users paying for API, instead of their Max plan).
The era of subsidising costs is over
If you read/watch any of the content about the models produced by academics you can get the real story about the LLMs. They genuinely don't understand why they work. They built a next word predictor, fed it all the data and are now frantically trying anything they can to get better output.
Prompt chaining (so called "thinking") and reinforcement learning (goal based tuning) have yielded results so far but they're all doing those now. They need to come up with another idea to get slightly more performance out and everything they're trying is going nowhere at least for now.
