back to article LIDAR in iPhones is not about better photos – it's about the future of low-cost augmented reality

For the past six months I've been staring at the backside of my iPhone 13 Pro wondering what possessed Apple to build a Light Detection and Ranging (LIDAR) camera into its flagship smartphone. It's not as though you need a time-of-flight depth camera, sensitive enough to chart the reflection time of individual photons, to …

  1. Major N

    Whilst I'm sure apply sticking these into their devices probably helped with the economies of scale required, how about all those 'automated' cars pouring R&D dollars into the same field? Pretty sure apple wouldnt have done it if the units cost hundreds of dollars each at the beginning...

    1. ClockworkOwl
      Boffin

      Mouser jogged my interest with a new product line:

      Osram Multizone ToF Sensors. (TMF8820, 21, 28) Tiny little SMD LIDAR units (2mm x 5mm approx.)

      Upto 5m range and upto 8x8 zones. Multiple targets in each zone, you just get an I2C signal giving you the range of each target in each zone... They even have 2 GPIO pins!!! All for under a tenner.

      However I don't presume Apple had much to do with their development.

      1. anonanonanonanonanon

        I think the ones in the current Apple devices are manufactured by Sony.

        Apple have been in the AR game for quite a while now, ARKit was already very good before LiDAR sensors came along.

        The LiDAR has other uses in an Apple device too, rapid Auto focus for the cameras, doing fancy portrait shots blurring the background etc.

        1. nijam Silver badge

          > ... fancy portrait shots blurring the background ...

          I can think of some photos where I wish the foreground had been blurred.

          1. SteveK

            Teams does that sometimes, leaving the background in perfect clarity but blurring the meeting participant...

      2. GlenP Silver badge

        They could be interesting for a project I have in mind in the future, as I'm not into soldering SMD hopefully someone will stick them on a board with the minimal additional electronics.

        1. imanidiot Silver badge

          AMS has designed a few different dev kits and breakouts for these sensors (I think you'll want the TMF882X-DB)

        2. NoneSuch Silver badge
          Joke

          Oh Apple...

          That eternally budget friendly corporation that looks out for its customers with reasonably priced goods and comprehensive self-repair options.

          Joke icon for the truly brain washed.

          1. Lord Elpuss Silver badge

            Re: Oh Apple...

            You get a downvote from me because your comment is utterly irrelevant to the content of the article.

      3. Anonymous Coward
        Anonymous Coward

        Precisely. If you also consider all the venture capital that has gone into solid state LIDAR, also for industrial, automotive and drones (https://www.dji.com/at/ronin-4d), I d say Apple is a net beneficiary of the trend.

        1. RM Myers
          Joke

          Sorry to disagree, but giving Apple credit for bringing another product to the consumer market seems only appropriate. After all, this was the company that invented the personal computer, the graphical user interface, the portable music player, the tablet computer, and the smart phone among many others products for individual comsumers.

          Besides, their founder was rightly famous for augmenting reality from Apple's earliest days, so adding Augmented Reality to the list is only to be expected.

  2. Ali Dodd

    Why bang on about Apple and LIDAR

    when at the end he just mentions intel and camera inside out tracking?

    It's also on the market now and a lot cheaper than an iphone plus (the one with the sensor) - the Meta Quest 2 does it quite well for £300!

    If you dislike Facebook there's a really nice open source product on it way shortly the Lynx platform https://www.lynx-r.com/

    I'd love Lidar to become really popular as it's very accurate but also expensive, I think more will just use normal cameras as its a fraction of the cost. But yeah Tracking happens!

    1. anonanonanonanonanon

      Re: Why bang on about Apple and LIDAR

      The article does seem to imply there was no AR until the LiDAR, for sure the LiDAR adds some really cool features to ARKit, but all iPhones with an A9 chip or later support AR. The newer and fancier phones have improved tracking over earlier models, but earlier models are still very good.

      I think the headsets are a completely different kettle of fish. You can't carry one in your pocket.

      Other manufactures of mobile devices also offer AR, and have done for roughly the same time as Apple, I just happen to be an iOS dev who works on an AR based app, so I'm biased.

      Android phones also have Tof sensors, but IMO, Apples implementation of LiDAR in their dev kit far outperforms the Android equivalents

  3. DS999 Silver badge

    The use I wish someone would put the iPhone's LIDAR to

    Is a launch monitor for golf. The cheaper ones are already photo based, the addition of LIDAR would improve upon that but you'd use the iPhone you already have instead of paying to replicate all the sensing hardware and computation ability in a separate device. If someone wrote software to do this and sold it for $100 they'd have made enough to retire on by the end of the week!

    It doesn't have to be as accurate as a $20,000 Trackman, if it is a few percent off (which IIRC is the case with the cheaper photo based units) or misses the readings once in a while that's good enough for everyone for whom golf is just a hobby not a career.

    The only downside is that the sensor is in the rear so you couldn't see the display after each shot, but a $5 mirror would fix that issue.

    1. anonanonanonanonanon

      Re: The use I wish someone would put the iPhone's LIDAR to

      How would LiDAR help? Is it just to get scale of the scene? I don't think the sensor could track the golf ball, it isn't very high resolution.

      Are these apps interested in just the speed of the ball, or the direction too?

      I'm not a golfer, but I'm interested in making this app for the hundred bucks you speak of.

      1. DS999 Silver badge

        Re: The use I wish someone would put the iPhone's LIDAR to

        The sensor alone cannot track it, but photo based launch monitors already work with a camera only. A LIDAR would hopefully provide some enhancement, but even if not the camera in an iPhone is almost certainly better than that in a $499 launch monitor.

        They track speed, direction, launch angle and spin rate of the ball as it leaves the clubface.

        1. anonanonanonanonanon

          Re: The use I wish someone would put the iPhone's LIDAR to

          There some other nice features in the vision framework for AI analysis of images, including trajectory tracking. Combined with video at up to 120fps, it could have some interesting possibilities. Might go an buy myself a golf ball.

        2. Brewster's Angle Grinder Silver badge

          Re: The use I wish someone would put the iPhone's LIDAR to

          My inner astronomer thinks this should be easy - give me half a dozen stills with timestamps and I'll give you the code for everything except spin. But stuff inside the atmosphere can be a bit of a drag, and commercial ball-tracking using six camera, so maybe it's not quite as easy as I think.

          The trouble is the app won't stay at $100 once people realise you can do this effectively from a phone - especially if all the app needs to do is be a thin wrapper round an API call; it would rapidly drop in price.

          1. DS999 Silver badge

            Re: The use I wish someone would put the iPhone's LIDAR to

            That "Hawk Eye" ball tracking shown in the photo at a tennis court is tracking the ball from a long way away, which is a different problem to solve. A golf launch monitor is placed within a couple feet of where the ball is being struck, and really only has to track it long enough to get the desired information so it may only need to track it for a few inches, or perhaps a few feet at most.

            I doubt it could possibly be a "thin wrapper around an API call". If it was, there would already be such apps out there. My guess is the API helps, but only solves part of the problem. The difficulty of solving the rest would avoid the "race to the bottom" you suggest.

            Spin is probably easier to detect with radar since it can "see" the dimples in the ball more easily than a camera can (since a ball that is struck by a driver is traveling at 100 to 200 mph after impact depending on whether it is struck by an average woman/senior or a tour pro. Since it is only used during practice, if you had to add some lines to the balls you are hitting (balls on a driving range often already are striped) to "help" the cameras better see the spin that wouldn't be an unreasonable ask.

    2. Tom 7

      Re: The use I wish someone would put the iPhone's LIDAR to

      Photogrammetry is what your looking for.

    3. Missing Semicolon Silver badge

      Re: The use I wish someone would put the iPhone's LIDAR to

      Doesn't the LIDAR for cars need to be rather more powerful? I seem to remember one self-driving VC-sink being untroubled by the reporting journalists' cameras all acquiring dead spots on their sensors....

    4. Roland6 Silver badge

      Re: The use I wish someone would put the iPhone's LIDAR to

      A useful application of this tech is to enable real-time translation of sign language.

      Some years back Microsoft did a demonstrator on the Xbox 360 using the then new Kinetic sensor for American sign language.

      Given the performance increase in phones (particularly iPhones - which seem to be widely used by the deaf community and seem to give good results with cochlear implants), it should be possible to do.

  4. jdiebdhidbsusbvwbsidnsoskebid Silver badge

    I'm sure there's an element of "build it and they will come". I suspect Apple have partly fitted lidar sensors simply to show they can, and to encourage others to come up with novel uses for their products - and thus sell more.

  5. Anonymous Coward
    Anonymous Coward

    Letter to Elon

    Dear Elon Musk...

    So you want to cut costs and remove LIDAR from your cars? We here at Apple have added LIDAR to our iPhones at a minimal cost.

    We are already in talks to supply our technology to other car makers around the world. Are you still sure that LIDAR is not needed for your FSD/RoboTaxi dream? We think not.

    But please carry on penny-pinching Mr Musk. You have clearly taken that crown from us and to be honest, we are delighted.

    Yours

    Apple inc

    The above is very tongue in cheek and is not meant to be taken seriously and has no factual basis.

    1. Gordon 10

      Re: Letter to Elon

      Thing is I'm not sure its funny. LIDAR for Phone <> Lidar for a car.

  6. Efer Brick

    Lie-dar?

    Hany for when at the House of Commons

    1. adam 40 Silver badge
      Happy

      Re: Lie-dar?

      Handy for when at the House of Commons

      Just point it at the Labour Front Benches!!!

      There - I made it funny.

  7. nijam Silver badge

    Isn't reality bad enough without augmenting it?

    Be careful what you wish for...

    1. Brewster's Angle Grinder Silver badge

      Reality distorition field

      Give it 25 years and it will be offensive, if not down right criminal, to view someone without the AR filters their headsets are transmitting to yours

      1. Lord Elpuss Silver badge

        Re: Reality distorition field

        True. It'll be the 2050's version of misgendering. Sigh.

    2. Michael Wojcik Silver badge

      Tired: reality. Wired (or, rather, wireless): ubiquitous fiction, without even a smidgen of mental effort.

      My first hands-on experience with VR and, to a limited extent, AR was at SIGGRAPH in Boston in ... 1989, I think. I wasn't impressed then, except for a few niche applications. I'm not impressed now.

      "Rendering happens" indeed. For various meanings of "render".

  8. adam 40 Silver badge
    Big Brother

    Mandated by the NSA

    The new sensors in the phones are so the NSA can spy on you, in new ways you can't imagine!

    What next, millimetre wave passive radar using Starlink illumination?

    1. Binraider Silver badge

      Re: Mandated by the NSA

      24/7 3-d model world building in populated areas. Of course, the data overhead might be a tad of a problem to take advantage of such a capability!

  9. Morrie Wyatt
    Stop

    It's a safety feature.

    After all, someone/something needs to keep looking where the user is walking while their attention is glued to the screen.

    1. anonanonanonanonanon

      Re: It's a safety feature.

      You jest, but that could be really useful for someone who is blind

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like