Google's AI-enabled mouse pointer understands 'this' and 'that'

Google doesn't design mouse traps, so it's trying to design a better mouse. Google DeepMind announced a research effort to transform the standard computer mouse cursor into a context-aware, AI-powered tool, marking what the company described as the first major rethinking of the cursor in more than 50 years. The project by …

[elsergiovolador]

What

What could go wrong?

[Like a badger]

Re: What

What could go right, though? The whole WIMP interface is "of its time", and that time was about 1968.

Ideally I want my office desk to feature a wall of monitors and that hand-wavy interface from the film Minority Report, no effing keyboards, no cheap clunky mice, and no OS that seems to think it is the star of the show. Our IT team told me none of this featured on the approved options list though.

[Dan 55]

Re: What

Hand-wavy interfaces make their users suffer from gorilla arm.

There's still nothing better than a desktop UI from the period where it was known as a WIMP interface or a GUI instead of UX.

[Like a badger]

Re: What

Hand-wavy interfaces make their users suffer from gorilla arm.

I'm one of those animated gesturing types, so I can live with that. I'm not certain which colleagues would go in the tank, or what we'd then use the predictions carved on wooden balls for. I reckon Giles would be good in the tank, as he is super bright and has negative social skills. Ilyana is clever and irritating, so it's into the dry-suit for her as well, and she'd be lead pre-cog. I've still got a third pre-cog position available - anyone?

Imagine: LA Badger relaxes at his desk, then waves his hands across his magnificent field of displays. "Ilyana, what's my best option today?" Scene cuts to the water tank where Giles, Ilana and another float in water, jerking in small convulsive movements. Back to Badger: A whirring sound is heard, and then a high quality hard wood ball trundles on to the desk. Badger picks it up, confident that the wood is sourced in full compliance with the UK Forest Law Enforcement, Governance and Trade regulations. The camera zooms to the ball, and the lettering says "16:18 card at Newmarket this Friday. Expressionless to win at 10:1, no minority report".

[PB90210]

Re: What

It was bad enough on Win8, when you drag a little too far and bam you are in split screen mode.

It was a new machine but luckily Win10 was out by then and I was able upgrade to something more usable

[AVR]

When the AI guesses wrong about what you want to do with your mouse untangling it will take longer than a simple mouse click, possibly much longer. I'm sure it'll still be possible to do other actions and follow other paths but this software will make it harder. If all the usual right-click menus are still there, then learning the use of this kind of desktop takes longer because you have to know the AI procedures and the prior procedures; if not, definitely go with the 'much longer' to disentangle misunderstandings.

Plus all the problems about expense if a remote AI (once it's fully priced in) or a fairly massive overhead if it's local, and Moore's Law is not current.

[hugo tyson]

WIMP "undo" is nonexistent

Exactly that. I hate having very many things on the desktop, especially using Windows at work because it would freeze and when it unfroze after some manic clicking and swearing, sometimes a whole folder or two was just gone - inside some other folder, or the bin, or off the screen where you can't pick it up. And there's no "tell me the recent few actions you did" so you can guess. So it's out with the command line and ls -altr to try to guess what folder was most recently inadvertently changed. The whole thing stinks....

And when IS/IT added a "shredder" icon to everyone's desktop, like it or not, which is like the bin but writes random over all the content so no-one can see that oh so important confidential file, and wondered why I begged them to remove it....

So, IMHO the "wtf did it just do" problem's not new - and I totally agree it'll be far worse when an AI mouse can "help" you without a record of what and why it did....

[Anonymous Coward]

Calm down

It's just another "research project" (aka someone trying to make a name for himself in the company); those are a dime a dozen.

What surprises me is that they still insist on voice input. That's hardly scalable outside a certain demographic, as already seen with their attempt at goggles. What I'm saying is, they're not even trying to make it look like a credible project.

[Dan 55]

Re: Calm down

It's going to go places, the users will have to set a limit which can be arbitrarily raised by Google. Imagine a future where you have to pay $2000 a month to move a mouse.

[Adair]

Exactly what I need ...

... Sir Humphrey driving my mouse.

[Simon Harris]

Re: Exactly what I need ...

... who would then explain the difference between this and that in Latin and go on to decline hic and ille.

[Pulled Tea]

In a demonstration website, a user can hover a cursor over a crab and say “move this here,” and the system understands enough context to grab the crab and move it to where the cursor indicates.

So, basically shades of “put that there”, but, like, with 1) your mouse constantly polling a remote server and probably slowing down the system when your network connection gets janky, 2) relying on a constantly-spying and eternally-unreliable chatbox that 3) does the equivalent of burning down a forest the size of Montenegro every day just so you can generate tung tung sahoor to send to your buddies.

I mean, as an experiment, that's cool, especially if it was being done by an academic research group or a sufficiently deranged individual. A large megacorporation? Nah, I'm good, thanks.

[ComputerSays_noAbsolutelyNo]

Open plan office from hell

I guess this is all nice, when you're researching this in the lab and you can always go back to the peace and quiet in your office.

Imagine 20 people chatting with their computers in an open plan office hellscape?

"No! That one!"

[Mishak]

Exactly...

Might work for WFH, but not in an office where the manager insists on silence "or else you're not working".

I also think that "likely intent" is going to be very frustrating for those who have other ideas.

[My other car WAS an IAV Stryker]

Re: Exactly...

But but but I thought manglement wanted everyone in the office because "collaboration"... which involves talking -- lots of it, at random times when inspiration hits (or a coworker just happens to be up and about like a restroom or coffee run).

You want silence, except for meetings? Let me (continue to) work from home and you won't hear a thing, especially me not complaining about the office and dumb manglement "rules".

[Bebu sa Ware]

Re: Open plan office from hell

Imagine 20 people chatting with their computers in an open plan office hellscape?

I don't have to. A room full of devs, support etc all using their headsets and talking with WFH colleagues, customers, totties and whatnot — for me a Circle so horrible that Dante mercifully chose to omit.

Apart from formal speeches or lectures, I don't see that the rate of informational transmission is much higher than noise - conversational speech in particular. AI might excel at speech recognition (ie word recognition) but given it is not exactly shining in extracting meaning from far more structured and informative textual material, I can easily imagine this type of system is going to be completely at seas trying to determine the intent of the speaker.

Imagine: "Where there are two or more crustaceans next to each other, move them to the bottom of the screen but place the crabs with the crabs already there and likewise the prawns with the prawns and so on for the other crustraceans. Any remaining unmatched crustacean should be placed in a miscellanous group."

Good luck with that.… you are just as likely to get a pile of molluscs for your trouble.

[Like a badger]

Re: Open plan office from hell

A room full of devs, support etc all using their headsets and talking with WFH colleagues, customers, totties and whatnot — for me a Circle so horrible that Dante mercifully chose to omit.

Coming next from Google and Meta, a full VR helmet for office workers!

Something like a riot cops helmet, integral speakers and mic, brainwave monitoring to ensure the victim is thinking about dull things. Able only to hear what the IT believes they need to hear, unable to see anything the IT chooses is non-relevant, perhaps even a collar lock that ensures they have the device on for the entirety of working hours. Think of the benefits! Increased output because no need to interrupt work for a toilet break; unable to ogle attractive colleagues meaning low harassment cases; erm, and some other bullshit benefits believed only by senior managers.

[PB90210]

Re: Open plan office from hell

Me: "And delete to end of document..."

Kevin from 2 cubes down: "Where did my work just go?!?"

[Paul Herber]

If the AI mouse were the norm ... and someone then invented the right-click, we would be over the moon! Wow, peace and quiet for a change. Doesn't this make things so nice and simple!

Some research just needs to be locked away in a nice softly padded room.

[Bebu sa Ware]

Some research just needs to be locked away in a nice softly padded room.

Possibly even more research of this ilk ought to be locked away in some STASI run institution noted for its celebrated testicle stimulators.

[steelpillow]

"Pointer says No!"

Yes, the off button is there because legals. But you have to click it with the AI-enabled pointer...

[Mark #255]

"meeting you where you work"

And all that extra data to hoover up and send back to base is just a side-bonus for Google, I guess

[lukewarmdog]

Isn't this what the Neuralink chip is supposed to do?

Because I'll wait for the implant and save the Amazon Rainforest at the same time, thanks.

[Wade Burchette]

If it ain't broke, don't fix it!

There is a reason why the mouse interface hasn't changed in decades: because it just plains works! What is it with these companies that want to change things that well understood. Muscle memory is real and important. This research idea shouldn't have even made it out of a room. The person who proposed it should have been told that they were stupid and an idiot.

But let us be honest, the only reason this dumb idea even made this far is because companies have some kind of fetish for everything AI. Throw the acronym AI into an idea and corporate will laud your genius, no matter how stupid and retarded the idea is.

[andy gibson]

Re: If it ain't broke, don't fix it!

There was that brief period when Apple released the round "puck mouse" with the iMac.

[DS999]

So the AI will be training the user

It has an idea what sort of mouse wiggling it believes signifies "this" and "that" and if you wiggle it wrongly it won't do what you want, when you wiggle correctly it will. So it isn't the AI that is "understanding 'this' and 'that'" it is the AI training the human what sort of wiggling it wants you to do.

[TeeCee]

Motivation

...the system can understand where a user clicks, what they are clicking on, and the likely intent behind the interaction.

Data that advertisers will pay handsomely for, I'm sure.

(Entirely non-evil Satan icon)