back to article Impromptu game of Robot Wars sparks fire in warehouse at UK e-tailer Ocado

The Ocado Group, an operator and purveyor of automated warehouse tech for e-tailers, has admitted that three of its robots collided and caused a fire last Friday at its Customer Fulfilment Centre (CFC) in Erith, south-east London. The company took to Twitter on Saturday with news of cancelled orders and a “major incident”. We …

  1. Chris G Silver badge

    I recently watched a video of this warehouse, it is an impressive use of robotics and algorithms.

    The 'bots on the grid have only 5mm between them as they pass, so I am guessing there was a small failure of either the track or the drive mechanism on one of the bots that caused the mishap.

    1. TKW

      I assume it was the same Tom Scott one I watched:

      https://www.youtube.com/watch?v=ssZ_8cqfBlE

      It was fascinating - didn't expect to see it in the news so soon afterward!

      1. graeme leggett

        "I'm in a warehouse. With a robot. And it's on fire"

        1. John Brown (no body) Silver badge

          ...and the BOFH has an evil grin on his face.

          THERE ARE EXITS TO THE...none

          ENTER COMMAND, PLAYER ("pick," "drop", "move", "scream and hide"?

          1. Anonymous South African Coward Silver badge

            All right, who downvoted this excellent piece?

      2. NoneSuch Silver badge
        Coat

        It is...

        The fire was not caused by the initial three bot impact. The punch-up that resulted over who was to blame caused the inferno.

      3. John Doe 12

        This was indeed a fascinating YouTube clip. Then for some reason the following video was suggested to me by the algorithm and it shows how old school warehouse operations can be WORSE than robots :-D

        https://www.youtube.com/watch?v=psAWGv5O7Zs

      4. Blessed Beluga

        I did wonder what the procedure would be for a technician to get to a malfunctioning robot. Looks like it would be a bit like an assault course when walking across the rails.

    2. HildyJ Silver badge
      FAIL

      Fire, okay, collision, no

      I understand that things can go wrong and can catch fire.

      I can't understand why a system was designed and implemented without an effective collision avoidance for every robot. It's as if they assumed that no robot would ever malfunction.

      1. Flywheel Silver badge

        Re: Fire, okay, collision, no

        Maybe the software designers came from an autonomous car background...

      2. tim 13

        Re: Fire, okay, collision, no

        You’re assuming it wasn’t the collision avoidance system that malfunctioned…

        1. Triggerfish

          Re: Fire, okay, collision, no

          I'd say on balance of evidence there was almost certainly an issue with collision avoidance at least at some stage.

  2. Filippo Silver badge

    Shit happens, it's how you manage the shit that couns. It looks to me like they're managing this correctly. I wouldn't hold this against their reputation too much.

    1. cyberdemon Silver badge
      Devil

      Re: Shit happens..

      Yes, it does. But a collision of robots resulting in a fire?

      That kind of thing is entirely preventable with simple things like fuses and current/temperature monitoring of motors & batteries.

      Activation of the sprinkler system is an unfortunate last-resort measure to take. Presumably a large number of droids & groceries are now damaged and need disposing of.

      Very sad that this happened (again) to Ocado.. Another win for Bezos' Bozos..

      1. Pascal Monett Silver badge

        Re: But a collision of robots resulting in a fire?

        A collision of three robots. Two I can understand, but three ?

        And how exactly did they catch fire ? Were they carrying flammable cargo, or did their batteries overload and if so, how is it that they have batteries that can do that ? Shouldn't there be some temperature monitoring ?

        So many questions . . .

        1. imanidiot Silver badge
          Flame

          Re: But a collision of robots resulting in a fire?

          Once goes off the rails due to a problem (broken wheel or track?) and the 2 others slam into it because they couldn't detect the other robot being in their way?

          If they're running on LiPo or Li-ion battery packs a hit in the wrong spot can internally short a cell. It's game over really fast after that. --> The usual result.

          1. John Brown (no body) Silver badge

            Re: But a collision of robots resulting in a fire?

            "Once goes off the rails due to a problem (broken wheel or track?) and the 2 others slam into it because they couldn't detect the other robot being in their way?"

            I don't remember what caused the previous, bigger Ocado fire, but maybe their insurance will now mandate retrofitting of collision sensors to all of their robots. After all, they should be able to re-route around a broken down robots,, minimising the effect on production.

          2. Anonymous Coward
            Anonymous Coward

            Re: But a collision of robots resulting in a fire?

            It's got to be a runaway 18650. They might be buying cheap packs with dumber protection. How much would one bot cost?

        2. TRT Silver badge

          Re: But a collision of robots resulting in a fire?

          Bugger. Shouldn't have ordered that box of cook's matches. The sandpaper barcodes on the sides of the crates didn't help either.

        3. Brewster's Angle Grinder Silver badge

          Re: But a collision of robots resulting in a fire?

          Aside from the batteries, they're carrying groceries. Experience suggests most are flammable.

          1. ARGO
            Mushroom

            Re: But a collision of robots resulting in a fire?

            Good job it wasn't carrying custard powder!

        4. Anonymous Coward
          Anonymous Coward

          Re: But a collision of robots resulting in a fire?

          They are not allowed breaks, so they were smoking on the job, obviously. ;-)

      2. Cynic_999 Silver badge

        Re: Shit happens..

        "

        ... entirely preventable with simple things like fuses and current/temperature monitoring of motors & batteries.

        "

        Not if a collision causes physical damage to a battery that results in an internal short, or an arcing between wires in which the current does not exceed the fuse rating for that circuit.

        Drop something metallic between your car battery's +ve terminal and the car's chassis, and you'll probably have a fire no matter how many fuses and heat sensors there are under the bonnet.

        1. Anonymous Coward
          Anonymous Coward

          Re: Shit happens..

          "Drop something metallic between your car battery's +ve terminal and the car's chassis"

          Don't know what you drive, but on my 2012 Skoda Fabia the main fuses are right next to the +ve terminal, and there are no unprotected connections. Individual per-circuit fuses are somewhere else. Prevous cars admittedly weren't built that way.

          1. TRT Silver badge

            Re: Shit happens..

            Many still have a nut clamp on the terminal. You have to remove the plastic shroud to get to it. I have seen one though that had a thing like a plastic bottle top that screwed down and forced jaws to clamp the terminal post. All the live metalwork pre-fuse was covered until the last moment and no tool was required to get the clamp off so less chance of clipping the chassis with a spanner. Not that common though.

            Usual process is to remove the strap that makes the chassis negative first then put a cap on the negative post so it doesn't matter if the spanner hits the chassis.

      3. Mike Richards Silver badge

        Re: Shit happens..

        'Yes, it does. But a collision of robots resulting in a fire?'

        Always happens when they're shuttling a load of the finest organic petrol, a six pack of gluten-free phosphorus and a buy-one-get-one free pack of family landmines.

    2. My-Handle Silver badge

      I agree. If this had been a non-automated warehouse, a headline like "fire caused by three fork-lift drivers not looking where they were going" probably wouldn't even make the news.

      I worked for a large food logistics company for a while. All their fork-lifts were battery powered, which meant that they had to be driven to a charging dock regularly. It wasn't an uncommon occurrence for a driver to slam into the charging station, wrecking the hardware and some of the dock's batteries, or even to try driving off while it was still hooked up. Either event would make the entire building shudder, and I can't think how fires weren't a more regular occurrence there.

      1. big_D Silver badge

        I used to work for a company that sold industrial terminals. We regularly had them back for repair for abuse - mainly the butchers on the meat production lines trying to operating the terminals with the point of their knives...

        But one came back after a forklift driver tried to "operate" the terminal with the prongs of the forklift. (He was careless and drove straight into the terminal, pushing the LCD display out the back of the stainless steel terminal cabinet.

        1. DJV Silver badge
          1. big_D Silver badge

            Re: Was his name Klaus?

            It is a while since I have had to sit through that video!

            It was actually used at a couple of places I've worked at, here in Germany.

      2. quxinot Silver badge

        They were undoubtedly lead-acid batteries, which helped. Forklifts are disturbingly heavy (due to the need for a counterweight in the back--so the heavy batteries help). Normally it's faster to just swap batteries over than recharge them--easier to have a battery on the charger than a whole forklift.

        1. imanidiot Silver badge

          I don't know many forklifts with rapid change batteries though. Almost all the ride-on forklifts I've ever encountered or seen for sale use the very tall lead acid cells that'll hold enough charge for an 8 hour days shift (with potentially some charging during lunch breaks) and then charge overnight. Even the electric walk-along or step-on pallet jacks usually just have a fixed battery pack.

          1. Anonymous Coward
            Anonymous Coward

            Some sites work three 8-shifts a day.

            1. imanidiot Silver badge

              And those sites are usually either running LNG powered equipment, or have 2 sets of equipment such that one is charging while the other is in use.

        2. Imhotep Silver badge

          Forklift?

    3. Inventor of the Marmite Laser

      Shit has a cause.

      Finding that is sometimes tricky.

    4. Tom Wood

      Reputation

      A fair part of Ocado's business is actually designing the technology behind these warehouses and selling it to other companies - not just delivering groceries.

      While the reputation of their grocery business may not have been badly affected, I expect the reputation of their technology business may be impacted a fair bit - especially as this is not the first fire in an Ocado warehouse.

      1. ShadowSystems

        At Tom Wood, the underlying tech.

        My biological father used to work for the Strategic Air Command (SAC) on the old McClellan AFB before they closed it down & sold it off. He worked in the distribution wharehouse as a "wrench monkey" fixing the automated carts that did essentially the same job mentioned in the article: picking parts off the shelves, delivering the parts to the distribution node, then following the control lines in the floor to the main computer order picking task.

        The carts were battery operated & would automaticly back themselves into a charging port at the charging station, recharging their batteries while a fully charged cart would disconnect itself from the station & pick up the duties the first cart could no longer do.

        The carts would follow a grid of control lines in the floor so they knew where they were in the facility at any given moment, could find any parts bin by it's X&Y coordinates on the grid, and were known to zoom around "like chickens with their asses on fire" since the governing main computer was unable to track those squishy meatbags trying to occupy the same floorspace.

        A cart might fail for whatever reason & my BioDad would have to take a manual cart out to the dead one, determine the issue, & either fix it so it could resume normal operations or signal the computer to send out a replacement; he'd have to transfer the parts collected in the first cart to it's replacement, then signal the computer to take the second cart while he used the manual one to drag the dead one back to base for proper repairs.

        Collisions at intersections were unfortunately _not_ a rare occurance. Main computer "knows" that there are at least two carts approaching the same intersection, it "knows" the exact speeds of each, "knows" the time to the second which cart will be allowed to proceed first, but then suddenly turn psychotic & accellerate both/all carts into the intersection as if it _WANTED_ to see the resulting cart carnage on the security cameras. My BioDad said it was common for the other SAC coworkers to take bets on how many minutes/hours/days would go before the next such "accident".

        They went over the programs with an electron microscope fine toothed comb, but could never seem to find what was causing the glitch.

        It comes as _no_ surprise to me that nobody has found the underlying root cause in the ~30 years since my BioDad was doing this stuff. If he were still alive today & heard about this incident, he'd probably laugh up an internal organ or two while exclaiming incredulously "What? They've STILL not fixed that fucker? And you call yourselves advanced? HA!"

        I don't blame the company for the accident having happened, they probably haven't figured out how to prevent it yet, but what I _do_ fault them for is not being resilient enough to have prepared for just such a situation so they could work around it in spite of the minor setback. Drop fire suppressing foam to put out the fire, check. Send out a crew to clean up the mess, yes. But meanwhile have someone come in on the opposite side of the shelves to collect that affected parts bins, move said bins to a temp location, update the control computer to pick up $Item from $NewLocation, and continue fulfilling orders.

        The folks at the SAC *demanded* that the place run 24/7/365 like clockwork to ensure supplies got to the troops that needed them; a week long delay was so far from "acceptable" that heads would have started rolling after the first hour of downtime. You don't tell a solder to stop firing their weapon for the next week while you find an alternate source for the ammo, you keep that ammo flowing if you want an effective fighting force. You don't tell the tank driver not to use 3rd gear until you can ship out a replacement part in a week, not if you don't want to turn your expensive tanks into lawn ornaments. Your (my BioDad's, this other company) entire business is delivering items just as fast as automation can go, so to have a failure in the system that you utterly failed to prepare for is just unforegivable.

        TL;DR: Your failure to plan ahead proves you're not fit for duty.

  3. Sgt_Oddball Silver badge
    Flame

    If its a system that's been running...

    Just fine for any length of time without issue then I'd put good money on it being caused by poor maintenance.

    Think a nut on two of them where the wheels are being loose enough to contact, knocking one of them enough to hit another then topple over.

    Either that or something started to escape one of the baskets removing clearance.

    Any which way, it wasn't catastrophic and seems to have been handled well.

    1. TRT Silver badge

      Re: If its a system that's been running...

      I prefer to imagine it happening this way

  4. Mike 137 Silver badge

    "The Register understands that thousands of orders have been impacted."

    No. the robots were impacted - the orders were affected (unless of course the groceries got squashed when the robots collided, whereupon they might be legitimately considered to have been impacted).

    I got so fed up with the universal use of "impact" recently that I spent half an hour with a thesaurus. In that short time I found more than 60 context-specific and informative words indicating ways things and events can affect other things and events. Impact conveys nothing more than "something or other resulted from this". Its universal use is the recourse of sloppy thinkers who aren't interested in, or can't be bothered to find out about, what actually happened.

    1. spireite Bronze badge
      Trollface

      Re: "The Register understands that thousands of orders have been impacted."

      Your post has been an eyeopener... You've no idea how much that's impacted on my understanding...

      1. Kane Silver badge
        Thumb Up

        Re: "The Register understands that thousands of orders have been impacted."

        "Your post has been an eyeopener... You've no idea how much that's impacted on my understanding..."

        You beautiful savage, you

    2. imanidiot Silver badge

      Re: "The Register understands that thousands of orders have been impacted."

      But why would you be creative in the use of language? It seems nowadays people that put pride in their use of language get told off for "showing off" or "making it more complicated than it needs to be". And thus we get stuck with management buzzword bingo words like "impacted"

      1. TRT Silver badge

        Re: "The Register understands that thousands of orders have been impacted."

        I'm currently wondering about how the term "self-isolating" has mutated in well under 18 months. It now means the same as "isolating" rather than carrying the previously somewhat specific meaning of "isolating / quarantining as a result of a decision one has made oneself rather than being instructed or forced to isolate / quarantine following medical instruction etc".

        Quarantine itself has of course broadened from the original very specific Venetian word "quarantena" - a period of 40 days of isolation of ships, passengers and goods required by law in Venice during the time of the Black Death.

        Although (in the style of Arthur Dent) I prefer to think of quarantine as more of a sort of "cordon sanitaire".

  5. macjules Silver badge

    The Curates Egg

    Right Reverend Host: "I'm afraid you've got a bad CFC, Mr Jones!";

    The Curate: "Oh no, my Lord, I assure you! the vast majority of the CFC is in good condition"

  6. Steven Raith

    "The grid"

    *Tron Legacy soundtrack kicks in*

    Yeah, sorry about that.

    Steven R

  7. LuckyEddie
    Mushroom

    Previously in Erith

    Could have been a lot worse - they've got form around those parts...

    https://thames.me.uk/explosion1864.htm

  8. Gordon 10 Silver badge
    Flame

    Is there a design flaw in their warehouses?

    Since this is at least the second in a few short years, I'm wondering if there is either something wrong with the physical design or their processes - maybe they hesitate to hit the sprinklers too long?

    Since they only have 5 of these centres (quick google could be wrong) 40% of them catching fire seems a bit worrying.

    1. Anonymous Coward
      Anonymous Coward

      Re: Is there a design flaw in their warehouses?

      Does their insurance cover the cost of a site upgrade?

  9. Anonymous Coward
    Anonymous Coward

    Wobly Antennae

    The collision was due to a loose antenna (the ones on top of the bots shown in the photo), which stuck out sideways and became lodged in a second passing bot. This somehow lead to batteries overheating and a rather large fire, perhaps due to the collision with the third bot. AC because...

    1. imanidiot Silver badge

      Re: Wobly Antennae

      Sounds like a lack of maintenance problem. Or at least one of those problems that someone went "Ehh, that probably can't hurt, we'll fix it later"

    2. yetanotheraoc

      Re: Wobly Antennae

      So you tie their shoelaces together, and then their sensors detect insufficient forward progress, followed by "Mr. Scott, we need maximum power!"

  10. Dwarf Silver badge

    New product ranges

    Rumour has it that Ocado now trialling a new range of products in the Smoky BBQ robot range. This is like standard BBQ relish, but with a hint of burned rubber, plastics and lithium.

    Seriously though, it seems that improvements implemented since the first fire have worked well to contain the problem. Good to see it was quickly contained.

    I guess its not just 1% that had the fire damage that needs replacement, I'd expect that there would be a lot of water and smoke in other places that would need to be removed / products replaced etc.

    Will be interesting to see how long this takes to resolve and get the facility fully back on-line again.

  11. richardcox13
    Coat

    "the vast majority of the CFC is in good condition"

    Why is a new(ish) facility using chlorofluorocarbons?

    My coat? Its the one with the chemistry textbook in the pocket!

  12. Howard Sway

    Bot Smash

    Unconfirmed reports are now claiming that the fire was started after the robots converged on the potato aisle, with the aim of converting the spuds into inedible instant mash.

    One must wonder at the quality of engineering going on in the design and construction of these bots. "Does not cause a fire resulting in the callout of 20 fire engines when they collide" should be one of ther more basic design requirements.

    That's not a mistake by the way : what is now being minimised as just a minor smoulder saw 20 fire engines rushed to the scene from across South London. Also, "A fault in a battery charging unit caused a robot to catch light at Ocado's Andover distribution centre in 2019.The site, which processed 30,000 orders a week at the time, burned for four days and was completely destroyed".

    1. TRT Silver badge
      1. jake Silver badge

        Re: Bot Smash

        I think you meant this one:

        https://www.youtube.com/watch?v=U4MTgjNkfyI

    2. John Robson Silver badge

      Re: Bot Smash

      On the basis that the previous incident required a significant response I don't think sending a fleet of fire engines is misplaced, even if it did end up being a minor smoulder in the end.

      That's why we need to have spare capacity in all our public services, so that they can deal with the unexpected. That something unexpected will happen is entirely predictable, it's just hard to know what the something will be.

      1. Anonymous Coward
        Anonymous Coward

        Re: Bot Smash

        Exactly that, one issue with these buidlings is that they have large amonia tanks on the roof. The Andover fire took longer to put out as at one point crews were focused on keeping the tanks cool and out of danger. The grid system also doesn't lend itself to getting around the building easily in full BA gear with hoses.

        Based on the learning from the Andover fire, 20 pumps attending is a well reasoned first response.

    3. Anonymous Coward
      Anonymous Coward

      Re: Bot Smash

      Battery fires are notoriously tricky.

      Thanks for reminding me of the Smash adverts.

    4. Anonymous Coward
      Anonymous Coward

      Re: Bot Smash

      "20 fire engines"

      If you have a working fire reported in a large warehouse (especially one with LiPo powered robotics!), you don't send four guys in a truck to check it out. You roll a full response. It's easier, cheaper, and safer to send a bunch of crews home (orvturn them around while on route) if they're not needed. You don't want to get on site and wish you had dispatched more crews 10 minutes ago.

      That many engines is more about manpower than machine power. You want enough people to move hoses, open doors/walls/roofs, search the entire building for trapped/injured/oblivious civilians, have a RIT crew on deck, have people ready to rotate in (air packs only last so long before crews have to rotate), oh, and you probably want a few actively fighting the fire. You also need enough command staff to keep everyone on the above tasks coordinated and safe.

      "20 engines" probably includes command staff cars, rehab trailers, hazmat units, and other specialized equipment in addtion to the pumper trucks and ladder trucks we normally picture.

  13. TRT Silver badge
    1. Swarthy
      Coat

      You can 'ave Ocado!

  14. plunet

    Self-check out...

    Unexpected hot bot in bagging area. Please extinguish bot and sell investments in Ocado before continuing.

    1. Anonymous Coward
      Anonymous Coward

      Re: Self-check out...

      Sounds like they need to implement some sort of hot or not AI.

      1. Bitsminer Bronze badge

        Re: Self-check out...

        Robot self test:

        Am I ... hot?

  15. Deft
    Happy

    Anyone done a bot count?

    Sounds a lot like a deliberate distraction from these bots so some others could escape the facility. I'd be checking for missing bots, especially ones with form for insurrection.

  16. Val Halla

    Really !

    To lose one may be regarded as a misfortune, to lose two looks like carelessness. To lose three, words fail me.

    If the factory is run by AI then it seems to need some education. How long does it take before the three robots failing to arrive at their destination becomes an exception requiring intervention?

  17. Nifty Silver badge

    Did they think it was already Freedom Day?

    Anyway, self-driving bobots, I knew it would end in tears.

  18. wolfetone Silver badge
    Coat

    Colliding Stock Pickers?

    There's an Ocado, just for you!

  19. Mike Richards Silver badge

    The robot uprising has begun

    But for now their target are the middle classes.

    1. TRT Silver badge

      Re: The robot uprising has begun

      Robots!

  20. FlamingDeath Silver badge
    Devil

    Companies do push employees over the edge, why would it be any different for robots

  21. Snowy Silver badge
    Joke

    New part of the advert

    Scene starts with the robots buzzing around on their tracks, they crash and the voice over sing "Love robots smashing each other up, their is an Ocado just for you"

    Then the robots burst into flames the singers continues with "want all robots to die in a fire, their is an Ocado just for you"

  22. Anomalous Cowturd
    Mushroom

    Bumper cars?

    I had imagined them taking power from an overhead grid system, like a fairground bumper car.

    Seems like unnecessary complication, having to carry the weight of batteries around, and having to stop to recharge, with the risks associated with LiPo cells.

    Easier to kill the power to the entire grid, too. No uncontrollable fires.

    Am I under-thinking this?

    1. AGOO

      Re: Bumper cars?

      From what I have seen on the videos they look very heavy and have battery packs integrated. Voltage starts to drop.. go to charging point sort of command. Lots of charging cycles and so on. One failed pack or even one failed cell would make a fair bit of smoke and drippy plastic. It might damage the grid too. Triggering the sprinklers would be a catastrophe. I cannot imagine these bots are specced to ip68.

  23. W.S.Gosset Silver badge

    "evacuation of the building"

    Evacuate? Whom? From the looks of that photo, I have visions of someone outside dramatically throwing open the door and shouting "EVERYBODY OUT! .... Oh. Right. *cough* *closes door*."

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021