New patent will give iPhone screen interactive 3D

iPhones of the future could have virtual 3D interfaces that will detect and respond to the movements of your eyes, revealed an Apple patent granted today. The newly patented technology would also allow for a world-behind-the-screen experience with the user's fingers appearing to reach into a space behind the glass. Mashing up …

[LarsG]

NOT EVERYONE....

Can see properly in 3D, so will it also be produced in a 2D version.

Or is this just hopeful thinking?

Can you patent the impossible?

[LarsG]

Apple can patent anything they want to in the US

Including the inose picker attachment and the sneeze to dial app!

[miknik]

They patented interactive 3D??!!??

Oh noes, I use interactive 3D all the time! Does that mean I have to pay licencing fees now or should I just gouge out an eye?

[bobbles31]

then just maybe this device isnt for them?

some people don't have fingers.

[Blank Reg]

Not really a 3D screen

I think this is just a 2D. This appears to be an adaption of an existing method of making screens appear 3D by changing the perspective and lighting of objects on screen according to the orientation in which the device is held. This has been done before on phones and handheld game machines. There is even a simple example in Killzone 2 loading screens. If you move the controller around you will see the effect I'm talking about.

What Apple have added is taking into account ambient light in creating shadows and the position of the users eyes in highlighting certain objects on screen. I question the value of either.

Trying to use ambient light can have all sorts of unhelpful side effects, especially if you are somewhere like a night club for example where the lighting is undergoing frequent and rapid changes. And the eye tracking will have to take into account that your eyes actually move around quite a lot even when focusing on a particular object.. That may mean the UI isn't as responsive as you would like it to be, otherwise it will be jumping all over the place as your eyes dart around the screen.

[stizzleswick]

Seeing what has been patented...

...in the U.S. in the past, despite copious prior art and, often enough, no way of providing a working example at the time of application, yes, in the U.S., you can patent everything. You can even register a generic term for a hole in the wall as a trademark and sue others for using that generic term in a similar application.

Over the last decades, I have more and more gotten the impression that the U.S. patent offices exist only as a means for lawyers to generate money by filing ludicrous patents and then suing the pants off anybody producing anything even remotely similar.

</flame>

'nuff said before I go overboard. My coat is the one with the patent writ in the pocket for an on-line medium embedded in a news site, allowing readers to share their opinions.

[Dale 3]

Good

Just remember to hold it right.

[NoneSuch]

Well then.

I guess they have a working prototype if they are actually patenting it.

(Insert disbelief here)

[paulc]

yet again...

they're patenting an idea... without having to make an actual working model like they used to have to... and it's software being patented again, which should NOT be allowed...

[ed 22]

so f'in correct

but freakin cool nonetheless

[Anonymous Coward]

Ah! You must be one the "Cool trendy young" types!

Or gullible and easily impressed, baaaaaah!

Your kidding if you think this is cool. Kids, tisk, they know now't.

[Blank Reg]

There are too many prior examples of 3D UIs for even the US patent office to ignore. But Apple did add some extras that may make it patentable, though they may not actually be useful in practice.

Sun had a working 3D UI (Project Looking Glass) about 10 years ago, and I doubt it was the first. But they did also show it running on a phone long before the iPhone was even a twinkle in Stevie's eye. It may have been the first on such a device.

[wayward4now]

Project Looking Glass

One of the Sun spin-offs of that project still exists today as OpenWonderland. Just google on that. It's a GPL'd Open Source project that is still very much alive, even after Oracle closed it down.

[paulc]

yippeeee!!!!!

I loved Looking Glass... just didn't have hardware then that could do it justice... and that I think was the real downfall... it was ahead of it's time... and Microsoft et al have lifted ideas from it for their flashy UIs

[EyeCU]

Prior Art....AGAIN

Just a quick example from LG

http://news.cnet.com/8301-17938_105-20079160-1/lg-d2000-glasses-free-3d-monitor-tracks-your-eyes/

Just changing the patent to read phone does not make it novel or innovative. This should be thrown out as it is an obvious extension of an existing product

[Synonymous Howard]

I'll match your Prior Art and raise you ...

http://weblogs.baltimoresun.com/news/technology/2010/01/apple_tablet_3d.html

which appears to date back to research done by Apple in 2007 ...

http://www.macrumors.com/2008/12/11/apple-exploring-3d-desktop-and-application-interfaces/

Anyone get LG's patent numbers covering that telly?

[EyeCU]

Not quite

Your first doesn't mention tracking. Nor does it mention Apple researching it, in fact the telling bit in the article is ths : 'Apple obtained the rights to this patent application from three French citizens: Fabrice Robinet, Thomas Goossens, and Alexandre Moha' and 'searches under Apple's name in the patents database doesn't retrieve this patent, because the names of the original French inventors are still on it.'

Tracking is mentioned but only in the comments by MikeWise 'using some sort of head-tracking system, it could simulate a 3D experience, like these researchers have done using a Wii remote.

http://www.youtube.com/watch?v=Jd3-eiid-Uw'

So it looks like Apple had nothing to do with this research, they saw a patent they liked and bought it. Add to that someone had already demonstrated a working prototype using the Wii and you see yet again Apple have no clue how to be innovative. No research, no working prototype, no product planned for release, sit on it till someone comes up with something similar = PATENT TROLL

Your second attempt is not 3D. It is a 3D representation of your desktop but that had already been done. Look up Bumptop or Project Looking Glass. Yet again Apple are looking at other peoples work to try and patent it and pass it off as there own. No wonder they don't spend much on R&D, they let other people do the work and steal it when it suits them.

PS: Here's a working prototype demo of a minority report type interface in 2006

http://www.flixxy.com/minority-report-interface.htm - again, nothing to do with Apple and well before the patents you mention.

Apple - the new Microsoft

[ThomH]

@EyeCU

You're conflating unrelated issues. The post you reply to, and the one it replies to, are about whether such a patent can be valid, given the prior art. Yours is primarily about to what extent Apple can be credited for research it has bought.

On the topic you want to pursue, buying intellectual property isn't the sin you seem to believe. Apple fans give the company credit for commercialising new markets - they spot an emerging technology and give it that final polish to make It really work as a mass-market product. Only in the imaginations of flame-happy comment-board drones does anyone argue that there was no GUI before the Mac, no MP3 player before the iPod, no smartphone before the iPhone, etc.

Apple's presentation of new products can be overly smug and cause a deserving discussion of the antecedents but this isn't that sort of situation. It's a granted patent application released by the patent office.

[EyeCU]

Read what I was replying to

'which appears to date back to research done by Apple in 2007 '

What were you saying about Apple fans crediting the company with the commercialisation and not the invention?

Embrace - Buy in new technology from elsewhere

Extend - Give it a bit of a polish

Extinguish - Sue the arse off everyone even if it has no relevance and hope something will stick, the iPod and the iPood or the shape of a rectangle for example

Like I said, Apple are the new Microsoft

[ThomH]

@EyeCU

"What were you saying about Apple fans crediting the company with the commercialisation and not the invention?"

Your logic: someone gives Apple improper credit. Ergo they must be an Apple fan. Ergo Apple fans give Apple improper credit.

Can you honestly not see any flaws in that?

[EyeCU]

Yes I see flaws

If you take a sample size of 1

Search these forums, search macrumours, search anywhere that the Apple faithful may have posted and time and time again they spout the same crap. Apple invented..., Apple were first to.... Even when corrected and shown evidence that they are wrong they still don't accept it. Whenever Apple do something that is blatently wrong they are there ready to defend the indefensible. Look at the downvotes people get for complaining about yet another frivolous lawsuit or a patent application so vague it could apply to thousands of different things.

There is a reason sterotypes become sterotypes. There is a reason that the Apple RDF has become part of popular culture. That doesn't mean all Mac users are fanboys but all fanboys are Mac users willing to defend the company at all costs and would do so even if the next itunes EULA said they must hand over a percentage of their annual income and their first born child.

[ThomH]

@EyeCU

The sample size of one was a direct response to your comment and I accept that it isn't a reasonable to assess generalities. But I maintain that you're overgeneralising and that confirmation bias explains how things like this become stereotypes irrespective of any particular truth.

For a suitably non-hyperbolic example — albeit which is still significantly more moronic than anything you've argued — see the things many people will argue are true about all men because, you know, everyone says that they're true and, anyway, in their entire life they can think of three or four examples where the thing they're talking about has happened so that proves it.

My honest belief is that around here the blinkered Apple fans get more down votes than the blinkered Apple detractors, but I'm quite possibly suffering bias of my own.

[James O'Brien]

i cant believe i can finally ask this question.....

How well does the eye tracking work in the dark seeing the typical iPhone user has their head stuck so far up their arse its darker than sin there?

[Michael Thibault]

So, like any good skeptic...

you've checked for yourself, have you?

[James O'Brien]

Nah

I have people who do that for me :P

[LarsG]

WHATS WITH ALL....

The deleted posts, or isn't the Anonymous check box working....

No it's not!

[Doogie1]

Ha ha, so that's why your usual anonymous reply to yourself above isn't anonymous. Someone at El Reg has a sense of humour!

[Chris Weatherby]

Holographic Phone Screens

definately stating the obvious

[Anonymous Coward]

oooh shiny..

With a bit of luck it will be so realistic and immersive all those fanbois will disappear into it and never bother us again, after all Apple have already given them all a free friend with Genuine People Personality (It's not a coincidence that it's called Siri surely?) to talk to, saving them the indignity of having to chat to proper friends.

[scarshapedstar]

Well then

I'm sure Johnny Lee feels very guilty about ripping off his legendary 3D Wii hack from Apple, six years ago.

[jai]

ignoring that it's an apple patent for a moment...

...but this seems like a very nice idea for a UI on a small screen.

If there's all these prior art examples, why hasn't anyone gone ahead and produced a phone with a UI where, instead of sliding left and right to get to more app icons, or background apps, etc, you move through the depth dimension. I think it'd be a very cool to see, and very intuitive to use interface.

[A 31]

without tracking the eyes ...

make your own :D

http://videos.howstuffworks.com/ted-conferences/7611-hacking-the-wii-video.htm

[Anonymous Coward]

why - because they can? - hmm.

I'm trying to picture this kind of 3D interface in usage to start applications, or locate files and I guess it depends how cleverly it's implemented?

If there's icons of some sort at different 'depths', at what point do the icons 'closest' to you obscure the ones behind to such a point that in order to reveal them, you would have to look past the ones in front without actually know exactly what's behind?

Or would you tilt the screen?

Seems to me that swiping to another screen of icons is just a whole damn lot easier that trying to operate an interface with your eyes.

The software would have to try to interpret your eye movement, which could lead to all sorts of irritating glitches. You may be tired, or get distracted and the system would misread that.

This also leads to simple bit of logic in terms of structure, order and finding what you want.

Scenario - you have 30 spanners of different sizes and you want to pick a number 12 spanner.

They are grouped in two ways.

The first way, the spanners are in a small box, you can only see the top 5 spanners. Now try to find the one you want. You have to rummage around in the box to find the spanner you want, because you can't quickly see all the different sizes - lets call this one the 3D interface

The second way, the spanners are all laid out flat, wall mounted. You can scan all 30 of them to get the one you want - lets call this one the 2D interface.

No prizes for guessing which one is faster and more intuitive.

In essence, a 3D interface of this nature is creating a 'box' with layers of icons in.

That seems to be exactly like the analogy above, with the spanners in a small box.

So, perhaps, as a interface to find stuff or to interact with files and launch applications, this has fail written all over it... BUT ... for other uses, gaming perhaps or maybe viewing maps in 3D, that's a different kettle of 3D fish.

[Tim Bates]

Already a box

I work in a computer shop that also happens to be a dealer for a large telco, so we get a lot of iPhones wander in. Usually they'd like to know how to .......

Of course with the way Apple does their "intuitive" icon listing of everything installed, but in no particular order, it can take 2 or 3 minutes to find things like "settings" or "messages" among the pages of unsorted crap.

My Android phone on the other hand has this little feature where it sorts thing in a way we technical people call "alphabetically". A bit geeky, but I think Apple could probably patent it still.

[Sid_the_Kid]

However...

...the search screen does a great job of finding apps that are stashed away on the seventh screen of icons or in a folder. I support a fair few iPhones and it is by far the easiest way to fire up apps which aren't immediately obvious.

[Toothpick]

Which bar ...

... do I go to to see a prototype lying around,

[ACZ]

Just a patent application - *not* a granted patent

This one is just a published patent application, *not* a granted patent.

Here's the publication:

http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PG01&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.html&r=1&f=G&l=50&s1=%2220120036433%22.PGNR.&OS=DN/20120036433&RS=DN/20120036433

Getting the images is a bit of a pain - they're TIFFs, so your browser will probably need a plugin like AlternaTIFF. Should be available as a PDF from the link below ("Original document" link on the left) in a week or so:

http://worldwide.espacenet.com/publicationDetails/biblio?FT=D&date=20120209&DB=worldwide.espacenet.com&locale=en_EP&CC=US&NR=2012036433A1&KC=A1&ND=4

For those that are interested in what happens to the case, use Public PAIR (http://portal.uspto.gov/external/portal/pair) and search for 20120036433 as document/publication number.

:)

[Robert Forsyth]

How do you touch an icon behind the screen

without breaking the screen?

[Morrie Wyatt]

iWonder

(a) How it will have cope with someone like Marty Feldman?

(b) Would Steven Hawking's eye and cheek operated wheelchair computer interface count as prior art?

[Anonymous Coward]

WHY?