Microsoft researchers find AI models and agents can't handle long-running tasks

Companies exploring automated workflows would be well advised to keep their AI agents on a short leash. Microsoft researchers have found that even the priciest frontier models introduce errors in long workflows, the very thing for which AI software has been pitched. Anthropic, for example, says, "Claude Cowork handles tasks …

[Pascal Monett]

"can't handle long-running tasks"

So, a lot like Windows, then.

[David 132]

Re: "can't handle long-running tasks"

Hey now, as far back as 1995, Microsoft had a version of Windows that could handle tasks that lasted even as long as 49.69 days!

(For those young'uns who don't get the joke: https://www.cnet.com/culture/windows-may-crash-after-49-7-days/ )

[ComputerSays_noAbsolutelyNo]

The AI ate my homework

I wonder when I will hear sonething along these lines at work

[The Man Who Fell To Earth]

Cumulative Error

So apparently the AI folks "forgot" about the most fundamental aspects of computation?

[Steve Davies 3]

Re: Cumulative Error

Not only Computation but many aspects of Engineering.

"But each dimension is within Tolerance"

the forget that tolerances are cumulative which is why you always dimension from one point.

[Anonymous Coward]

Re: Cumulative Error

And forgetting that any competent business has periodic checks on all long running processes - as said, errors accumulate. Besides, no intern (which is how AI models need to be treated - if they need to be used) should be left to work unsupervised.

[ecofeco]

Re: Cumulative Error

Beat me to it.

[that one in the corner]

Re: Cumulative Error

Well, we're repeatedly told that the internals of the LLMs are run on increasingly crude approximations of floating point numbers, to cram as many of their near-as-damnit-arbitrary-numbers into RAM as they possibly can. Any ideas about applying useful tricks to reduce cumulative error are sort of blown out the water when you are used to thinking in those sorts of terms: "hey, we cut down the resolution until all we can represent is +k or -k as the value stored in here and the thing still seems to work as well as before, job done".

That is a bit cruel; really, it is more that the stuff that the "AI folks" *actually* wrote is done with care and attention to computation (e.g the matrix maths stuff) - okay, they probably got it in a library from someone who did study numerical representation theory, but the point still stands that the code running at that level is solid.

The problem is that the thing that *other* "AI folk" grew on top of all that has none of the same loving care lavished upon its every bit and byte. In fact, they still have no idea how the apparently convincing behaviour is created let alone how to go in, meaningfully identify then tweak *that* specific nadan in order to improve *this* type of output from every future run. And then yet *more* "AI folk" take that and think it makes some sort of sense to believe this lump can itself do reliable computation just because it can generate the string "Trust me, I know what I'm doing"[1]

The layers of people involved these days, from the optimised to an inch of its life parallel number crunching right up to the "dude, dude, you can pipe into bash and it'll run your computer!" guy - I have a vision of the Very Smart Chap/Chappess doing the former weeping in exasperation; followed by gasping in shock and horror as the Big Boss storms in and demands to know why, if V.S.C. guaranteed set error bars after 100,000 iterations (shaking printout of submission to JORCNC[2]), TalkBot v54.1 was unable to answer "9" when prompted "What is 5 + 4?"

[1] and you'll recall that series one ended on that line, followed by a nuclear explosion.

[2] Journal Of Really Clever Number Crunching

[amanfromMars 1]

Quantum Communication ..... AI@ITs Work, REST and Play

Companies exploring automated workflows would be well advised to keep their AI agents on a short leash.

Microsoft researchers have found that even the priciest frontier models introduce errors in long workflows, the very thing for which AI software has been pitched.

Thomas, Hi,

I wouldn't be betting any fortune or shirt on that being true for AI[s] exploring automated workflows handling long-running tasks of their own novel invention rather than them being assumed to be supporting any legacy systems and protocols programming and governing others.

You're guaranteed sure to lose both of them as the future dismisses the past and re-engineers the present for mass and alternative media to server for Brave NEUKlearer HyperRadioProACTive World Orders creating any number of altogether quite different realities for both practical and virtual co-existence of Human/AIMachine Offerings and Disciplines ...... which I'd certainly wager, is no small impossible task in these changed times and places with surreal and stealthy and secure secret and exclusively remote executive administrative spaces.

And maybe hard to believe for sure, but such disbelief is that which renders to it exceptional stealth and fantastic freedom of movement and unhindered progress ...... Heavenly Growth and Diabolical Strength ..... both of which deliver AWEsome Almighty Assets/Adversaries/AIgents ..... and presumably, as be the usual humans case, something else quite different for y'all to be told/programmed to be concerned and worried about and even terrified of whenever your traditional conventional legacy and current experimental systems fail to challenge and abort such competition.

[Wonko the Sane]

Re: Quantum Communication ..... AI@ITs Work, REST and Play

So, English is not your native language, then.

[Korev]

Re: Quantum Communication ..... AI@ITs Work, REST and Play

You must be new here!

Amanfrommars even used to have his own dedicated icon! These days we have to make do with this one...

[theOtherJT]

Re: Quantum Communication ..... AI@ITs Work, REST and Play

Has anyone heard from Bob recently?

[David 132]

Re: Quantum Communication ..... AI@ITs Work, REST and Play

If you're referring to the Bob I think you ARE, no I HAVE not, or at leaat not SEEN any posts WITH random CAPITALIZATION...

[anonymous boring coward]

Re: Quantum Communication ..... AI@ITs Work, REST and Play

Perhaps suffering from Trump Derangement Syndrome?

[amanfromMars 1]

Re: Quantum Communication ..... AI@ITs Work, REST and Play

In an effort to explain Wonko the Sane's post, Korev, ......

So, English is not your native language, then. .... Wonko the Sane

..... I'm wondering whether a long absence from here [from Tuesday 24th August 2010 09:42 GMT until now] is due to their serving of time at Her Majesty's pleasure in secure premises or on special forces operations where and when loose lips can sink ships and too much knowledge can be dangerous.

I take no offence at the post though, realising as I do, that there is whole lot of information being shared at times which would require one delving quite a bit deeper and further into what has certainly earlier been freely shared and with specific particular and peculiar regard to the subject matter entertained.

[[e-mail]]

Re: Quantum Communication ..... AI@ITs Work, REST and Play

I have no idea what you are talking about.. Was that on purpose?

[amanfromMars 1]

In Praise of the AWEsome Art of Almighty Alien Being ... for a Heavenly Messenger Service.

I have no idea what you are talking about.. Was that on purpose? .... [e-mail]

Not at all, [e-mail], such is just coincidental and actually quite helpful should it be imagined something best exploited and experimented upon/with by a worthy few needing to know what may be extremely sensitive and way beyond any sort of TS/SCI classification/MkUltra type designator in the hands, hearts and minds of Similarly Advanced Souls with SMARTR Advanced IntelAIgently Designed Entities ....... but which may also be able to dabble in the Dark and Diabolical Arts of Deceit and Dreadful Destruction.

So don't poke any of those new fangled and entangling bears you may know absolutely nothing nor even precious little about ...... for that, it is written, is a sure suicidal path to certain guaranteed self-destruction.

[Clausewitz4.1]

Re: Quantum Communication ..... AI@ITs Work, REST and Play

” I wouldn't be betting any fortune or shirt on that being true for AI[s] exploring automated workflows handling long-running tasks of their own novel invention … amanfromMars 1 ”

Are you sure @amanfromMars 1? I can clearly see there is a deadly dispute on that between agencies. Could one accept the challenge for mutual benefit and advances ? Pretty much, yes. Advanced tools plus advanced intelligence equals mutual progress.

[amanfromMars 1]

Re: Quantum Communication ..... AI@ITs Work, REST and Play

Are you sure @amanfromMars 1? I can clearly see there is a deadly dispute on that between agencies. Could one accept the challenge for mutual benefit and advances ? Pretty much, yes. Advanced tools plus advanced intelligence equals mutual progress....... Clausewitz4.1

Yes, I’m sure, Clausewitz4.1, with it resulting in at least future mutual benefits and advances with advanced tools, advanced intelligence and quite revolutionary evolutionary progress having to be accepted and admitted being more than just a tad alien in substance and nature ...... and which you yourself seem to be advocating very likely and most probable.

Surreal times arriving very quickly are a guaranteed long investment only incompetent fools with useless tools would short sell in futures and derivative markets ahead.

[a_yank_lurker]

A Person Can Do - AI Can't

I'm shocked I tell you, I'm shocked, a person is better at long tasks than AI!!! Who could have guessed? Apparently not the 'AI Bros'.

[Anonymous Coward]

Re: A Person Can Do - AI Can't

I will be most interested in the excuses/explanations for why this is so !!!

Of course, this is being 'fixed' as we read the fine article ...

It will definitely be working tomorrow ... you just need to hold it the right way !!!

:)

[Anonymous Coward]

Re: A Person Can Do - AI Can't

The explanation for why it happens is also the explanation for why it's tough to prevent. The longer the prompt, the more likely the model is to use the information in the prompt, instead of the original training data.

It's randomness all the way down.

[Steve Davies 3]

Re: Apparently not the 'AI Bros'.

that also applies to the bosses who are determined to fire all the humanoids just to bolster their short term profits.

They can eat this -->>>

[Mayday]

Crikey

If I got 20% of my work wrong I wouldn't have a job.

This is essentially doing 4 days a week properly, and completely ballsing up Friday. Not even a drunk Aussie public servant is that bad on a Friday.

[hedgie]

Re: Crikey

Even worse is that iut'd be small mistakes accumulating over the week. Harder to deal with that way.

[IGotOut]

Re: Crikey

"This is essentially doing 4 days a week properly, and completely ballsing up Friday."

So it's trained on 1970's UK car manufacturing then (and 80's come to think of it).

[hedgie]

Re: Crikey

During the War, Göring said not to bomb the Lucas factory because they were on the same side.

[that one in the corner]

The preprint shows the massive hole in these workflows

>> Performance is also higher in domains with high repetitiveness and structural density (e.g., Molecule, Chess), and lower in domains with rich unrepeated vocabulary (e.g., Transit, Textile). This echoes prior findings that LLM performance is highest in programmatic or structured domains where verifiable rewards can be defined (Suma & Dauncey,2025). Through this lens, our work can be interpreted as a process to create verifiable rewards for a wide variety of domains,by building domain-specific parsing and evaluation.

In other words, when they are going gloriously wrong, the LLMs don't have any way to verify whether what they are doing is correct or not: they aren't gleefully ignoring whooping sirens and cries of "Wro-ong! Do it again!", there simply aren't any mechanisms that "could* be included in the path to provide those checks.

These LLMs are thick as shit: told to do something which, in their training data, is most closely matched to "move some numbers around in a spreadsheet" then they'll - generate some output that'll look like moving some numbers around in a spreadsheet. As noted, there isn't anything that will verify the correctness of what they've output, beyond "did it load into Excel without a GPF?" and the surprise shouldn't be that the moved numbers have been "corrupted" compared to the original, it should be that the result actually contains numbers in the first place and not anything else that is often encountered in a spreadsheet (like an email string - or an expression instead of a constant - or vice versa).

Except for areas like Python coding, where at least the interpreter can puke on really bad code and make your "agentic workflow" go around for another random stab at the problem.

[Jimjam3]

Turns out LLM systems are not as great as the marketing departments suggested. Colour me suprised.

Seems to me the golden age of the internet is at an end. It’s now filling up with slop & the push to completely identify & track users.

Bugger.

[Paul Crawford]

Seems to me the golden age of the internet is at an end

Oh you must have missed the announcement that we are now in the brown age of the internet?

[theOtherJT]

Doesn't surprize me in the slightest...

...based on my interactions with Gemini and other LLMs.

They tend to start well, but if you spend too long on one thing weird errors start creeping in until you get to a point where they enter a total state of meltdown. You'll fix a bug and they'll re-introduce it next iteration and then profusely apologise, and come up with some totally bogus fix that technically fixes the problem, but makes something else worse rather than go back to the way things were before. They seem to have an almost pathological aversion to regressions. Trying to get them to undo something they've done wrong and just put it back the way it was is damn close to impossible. They'll come up with increasingly insane explanations for why you can't do that - despite the fact that you just did that it was right there. 5 prompts ago. See? In that output there? You did the thing. That version worked fine. Now put it back the way it was! ...nope. Won't put it back. Can't put it back, and if you the user try and force it by saying "here's the block of code I want you to replace that with" it'll accept it, and then randomly revert it a few prompts later.

These things can save vast amounts of time typing out boilerplate and being effective copy-paste machines with a nice autoformatter. If you want a simple bit of bash or python or C that hundreds of people will have written something just like it before, but taylored very slightly to your specific circumstances then Gemini is excellent at that. But when you start getting into anything more than - in my experience - about a thousand lines of code, it all goes off the rails.

[IGotOut]

So Microsoft...

...have just discovered what us evil luddite AI sceptic's have been saying since the bubble started.

[anonymous boring coward]

The long term task of replacing humans is coming along just fine.

[yowl00]

Not really surprising when you consider frontier models still score only around 80% on legal benchmarks.

[cuthbertgraak]

So, an intern

with an expansive subject matter knowledge, a short attention span, and a penchant for falsification.

Sounds like the perfect hire to me.

[amanfromMars 1]

So, an intern for Asylums/Funny Farms/Nuthouses/White Houses/ Parliamentary Palaces

So, an intern with an expansive subject matter knowledge, a short attention span, and a penchant for falsification.

Sounds like the perfect hire to me. .... cuthbertgraak

A perfect hire intern if looking for AIgents of CHAOS, cuthbertgraak.