Microsoft: Why not let our Copilot fly your computer? Microsoft will soon let Copilot agents drive computers through the GUI just like humans – by clicking buttons, selecting menus, and even completing forms on screen. On Wednesday, the Windows empire said it plans to enable computer use from within Copilot Studio - Microsoft's platform for building and deploying AI agents. This …
Let's say I run a shopping website. One day I decide to add an extra checkbox on it, automatically checked. Text label "pay £2000 extra". Now, do you want to gamble that when Copilot is automatically ordering something from it, it will understand that this is going to quite legally take lots of extra money from you unless it decides to uncheck the box before completing the purchase?
I had a shock recently when I put something into a shopping basket on a site that happens to use Stripe, on my desktop pc. A site, incidentally, that I had never visited before. I went to the Basket, typed in my email address and, just as I entered the final letter, I received a text with a code.
The form itself asked me to enter that code, which I did. Not only did that populate my postal address, but it filled out my credit card details!!! WTF? I tried this test on another, completely clean pc, with the same outcome.
I abandoned the cart and emailed Stripe that they have somehow connected (1) My email address (2) my mobile phone no. (3) My postal address (4) My Credit Card details. How was that possible? Isn't this a flagrant breach of data protection?
Who remembers when people thought that Cookies were dangerous? Well, this is similar, yet on an entirely new level, where your Credit Card No is stored remotely, then presented to anyone with the incentive and expertise to do so.
To their credit, Stripe responded immediately saying that I had bought something from site xyz, x months before (hmm, that's a bit dubious as well, social engineering anyone?), and I clicked on a link on making that purchase, something like "speed up future purchases" which maybe I mistakenly pressed (I don't remember so), but even if I did, my expectations were "speed up future purchases ON THIS WEBSITE", not everywhere on the f'ing planet. Stripe informed me how to "opt out" of this behaviour, which I then did.
Since then I've encountered similar behaviour with Shopify. The difference between them and Stripe being that the parent company there seems to not understand what customer service is - ignoring my attempts to "opt out". Have a look at their TrustPilot ratings, they are about as popular as companies that issue PCN's. Presumably there's no redress as they are in Canada. (If anyone can help with this problem, please let us know, surely I can't be the only one?).
Ok, rant nearly over. What relevance is this to Copilot flying my computer? Well, as Howard says, that extra checkbox might help the likes of Stripe and Shopify leverage themselves into your Browsing Experience in unintended ways.
So, not only would I not touch Copulate (thinking of it thus might help), but I am now telling potential suppliers that use Shopify that I will not be entering my details into their Baskets, due to privacy issues. Some are decent enough to accept orders via other means. If enough of us did this then it might make a few companies think they are losing business due to the checkout system they are using.
Because Copilot is staggeringly incompetent at anything I have asked it to do (out of morbid curiosity). It really is the new Clippy.
I would rather have my 90 year old mother who clicks things at random run my computer. At least she wouldn't be 100% sure she is doing the right thing when 100% doing the wrong thing.
We tried for decades but could not get Excel to figure out what data type and format to use when importing data from simple formats like comma separated values.
Now why are you sceptics doubting our agents will be capable of moving data from the web to different applictions in complex and mutable work flows?
Yet more clutching at straws in a desperate attempt to find something for ClippyPilot and co. to do.
Given that none of these bullshit generators so far can be trusted in the slightest to not hallucinate and fill documents, emails and webforms with innacurate crap - if anything it will decrease throughput due to the amount of cross-checking required.
Of course, that'll only get put in place after the first time things go wrong.
The continuing march of Microsoft to force unwanted 'features' advances to the stage where one critical program is the only thing keeping me using windows. The frustration of having detritus that cannot be removed permanently may not bother corporate users but it annoys the hell out of me Bing, Edge, Copilot, Gamebar,mixed reality portal and AI I have no use for but like the cat they come back the very next day. I do need portability for the program so perhaps I will just turn off updates now on one of my windows 10 laptops before the October deadline and before further unecessary features are added.
Is this the real reason for the constant enshittification of the Windows UX over recent years? It's actually a deliberate ploy to ensure that only the MS AI is capable of finding the control you need without resorting to a hour of web searching and three pointless YouTube videos where the narrator gives a 45min preamble telling us what the Windows UX has meant to them and their family before finally pointing out where MS has hidden the "rename file" option this week.
This post has been deleted by its author
"the agent can adapt on the fly when it encounters obstacles or unexpected changes in the interface"
Hmm, unexpected changes, eh? Now who would go around doing that?
In theory this could be useful, in reality it will be a nightmare. Presumably, this means that copilot is running as you, with your permissions? Not good.
I guess it sees what's happening on screen via screen shots. But how does it respond? Is the CBT hook still available? Didn't that allow button clicks etc? Or will it just send wm_lbuttondiwn to the individual button windows? Or maybe physically move the mouse to the right place and insert button down and up messages into the queue? Frankly, the mechanics of how are a lot more interesting than the product itself.
In addition to all the other flaws mentioned upthread, this is entirely the wrong way to do automation. A much better approach would be to provide APIs to the underlying functionality that can be directly invoked by an automation agent - preferably an agent that does not purport to be AI - and ideally, upon which the human-centric UI is also built.
Driving the UI directly is the software equivalent of creating a self-driving car by taking an ordinary car and replacing the driver with a humanoid robot.
“Computer use enables agents to interact with websites and desktop apps by clicking buttons, selecting menus…”
But this is not “AI” AB’s does not in any way need any “AI” shit. Apple did something similar about 30 years ago. And I’m sure there are other examples about. I think even MS did this many years ago. Obviously MS’s effort was shit and so nobody used it but still
Microsoft is making itself irrelevant with these stupid A.I. gimmicks. No one's asking for some stupid A.I. to take over their computer and "do things" for them. The only one who touches my computer is ME, thank you!
Said company shouldn't be surprised if users jump ship at the first opportunity, and that will be October 13th, 2025. To Linux Mint.
Ooh... I might leave mint for win 11 just for the fun of watching my AI agent going bat shit crazy waiting for windows update to finish whilst I make a coffee!
On the other hand I'm doing nothing to encourage whose MS fuckwits to continue living.. Whatever reality they're in.
Given Microsoft's history of lax security and total disregard for privacy of Windows users, this is no doubt going to launch even more remote and more automated hacks, data mining and infections. The repeated re-working of Recall is a prime example of this. Microsoft disabled registry backups in Windows 11 claiming it takes up too much disk space, but no problem having Recall gobble up gigabytes of space in storing of everything the user has done.
Are Microsoft's A.I bots going to wake them up to the environmental disaster they are inflicting upon the planet with their forcing of millions and millions to buy new hardware to run this bloat?! No, just ship it off to some third-world country and contaminate their soil and water with toxins and heavy metals. Run out and buy an electric car so these battery packs can be added to this e-waste nightmare, CO2 is much more dangerous! Who needs CO2 when we can grow our food in toxic soil and drink water polluted with heavy metals and toxins? Sounds yummy!!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
