Impromptu game of Robot Wars sparks fire in warehouse at UK e-tailer Ocado The Ocado Group, an operator and purveyor of automated warehouse tech for e-tailers, has admitted that three of its robots collided and caused a fire last Friday at its Customer Fulfilment Centre (CFC) in Erith, south-east London. The company took to Twitter on Saturday with news of cancelled orders and a “major incident”. We …
I recently watched a video of this warehouse, it is an impressive use of robotics and algorithms.
The 'bots on the grid have only 5mm between them as they pass, so I am guessing there was a small failure of either the track or the drive mechanism on one of the bots that caused the mishap.
Yes, it does. But a collision of robots resulting in a fire?
That kind of thing is entirely preventable with simple things like fuses and current/temperature monitoring of motors & batteries.
Activation of the sprinkler system is an unfortunate last-resort measure to take. Presumably a large number of droids & groceries are now damaged and need disposing of.
Very sad that this happened (again) to Ocado.. Another win for Bezos' Bozos..
A collision of three robots. Two I can understand, but three ?
And how exactly did they catch fire ? Were they carrying flammable cargo, or did their batteries overload and if so, how is it that they have batteries that can do that ? Shouldn't there be some temperature monitoring ?
So many questions . . .
Once goes off the rails due to a problem (broken wheel or track?) and the 2 others slam into it because they couldn't detect the other robot being in their way?
If they're running on LiPo or Li-ion battery packs a hit in the wrong spot can internally short a cell. It's game over really fast after that. --> The usual result.
"Once goes off the rails due to a problem (broken wheel or track?) and the 2 others slam into it because they couldn't detect the other robot being in their way?"
I don't remember what caused the previous, bigger Ocado fire, but maybe their insurance will now mandate retrofitting of collision sensors to all of their robots. After all, they should be able to re-route around a broken down robots,, minimising the effect on production.
"
... entirely preventable with simple things like fuses and current/temperature monitoring of motors & batteries.
"
Not if a collision causes physical damage to a battery that results in an internal short, or an arcing between wires in which the current does not exceed the fuse rating for that circuit.
Drop something metallic between your car battery's +ve terminal and the car's chassis, and you'll probably have a fire no matter how many fuses and heat sensors there are under the bonnet.
"Drop something metallic between your car battery's +ve terminal and the car's chassis"
Don't know what you drive, but on my 2012 Skoda Fabia the main fuses are right next to the +ve terminal, and there are no unprotected connections. Individual per-circuit fuses are somewhere else. Prevous cars admittedly weren't built that way.
Many still have a nut clamp on the terminal. You have to remove the plastic shroud to get to it. I have seen one though that had a thing like a plastic bottle top that screwed down and forced jaws to clamp the terminal post. All the live metalwork pre-fuse was covered until the last moment and no tool was required to get the clamp off so less chance of clipping the chassis with a spanner. Not that common though.
Usual process is to remove the strap that makes the chassis negative first then put a cap on the negative post so it doesn't matter if the spanner hits the chassis.
I agree. If this had been a non-automated warehouse, a headline like "fire caused by three fork-lift drivers not looking where they were going" probably wouldn't even make the news.
I worked for a large food logistics company for a while. All their fork-lifts were battery powered, which meant that they had to be driven to a charging dock regularly. It wasn't an uncommon occurrence for a driver to slam into the charging station, wrecking the hardware and some of the dock's batteries, or even to try driving off while it was still hooked up. Either event would make the entire building shudder, and I can't think how fires weren't a more regular occurrence there.
I used to work for a company that sold industrial terminals. We regularly had them back for repair for abuse - mainly the butchers on the meat production lines trying to operating the terminals with the point of their knives...
But one came back after a forklift driver tried to "operate" the terminal with the prongs of the forklift. (He was careless and drove straight into the terminal, pushing the LCD display out the back of the stainless steel terminal cabinet.
They were undoubtedly lead-acid batteries, which helped. Forklifts are disturbingly heavy (due to the need for a counterweight in the back--so the heavy batteries help). Normally it's faster to just swap batteries over than recharge them--easier to have a battery on the charger than a whole forklift.
I don't know many forklifts with rapid change batteries though. Almost all the ride-on forklifts I've ever encountered or seen for sale use the very tall lead acid cells that'll hold enough charge for an 8 hour days shift (with potentially some charging during lunch breaks) and then charge overnight. Even the electric walk-along or step-on pallet jacks usually just have a fixed battery pack.
A fair part of Ocado's business is actually designing the technology behind these warehouses and selling it to other companies - not just delivering groceries.
While the reputation of their grocery business may not have been badly affected, I expect the reputation of their technology business may be impacted a fair bit - especially as this is not the first fire in an Ocado warehouse.
My biological father used to work for the Strategic Air Command (SAC) on the old McClellan AFB before they closed it down & sold it off. He worked in the distribution wharehouse as a "wrench monkey" fixing the automated carts that did essentially the same job mentioned in the article: picking parts off the shelves, delivering the parts to the distribution node, then following the control lines in the floor to the main computer order picking task.
The carts were battery operated & would automaticly back themselves into a charging port at the charging station, recharging their batteries while a fully charged cart would disconnect itself from the station & pick up the duties the first cart could no longer do.
The carts would follow a grid of control lines in the floor so they knew where they were in the facility at any given moment, could find any parts bin by it's X&Y coordinates on the grid, and were known to zoom around "like chickens with their asses on fire" since the governing main computer was unable to track those squishy meatbags trying to occupy the same floorspace.
A cart might fail for whatever reason & my BioDad would have to take a manual cart out to the dead one, determine the issue, & either fix it so it could resume normal operations or signal the computer to send out a replacement; he'd have to transfer the parts collected in the first cart to it's replacement, then signal the computer to take the second cart while he used the manual one to drag the dead one back to base for proper repairs.
Collisions at intersections were unfortunately _not_ a rare occurance. Main computer "knows" that there are at least two carts approaching the same intersection, it "knows" the exact speeds of each, "knows" the time to the second which cart will be allowed to proceed first, but then suddenly turn psychotic & accellerate both/all carts into the intersection as if it _WANTED_ to see the resulting cart carnage on the security cameras. My BioDad said it was common for the other SAC coworkers to take bets on how many minutes/hours/days would go before the next such "accident".
They went over the programs with an electron microscope fine toothed comb, but could never seem to find what was causing the glitch.
It comes as _no_ surprise to me that nobody has found the underlying root cause in the ~30 years since my BioDad was doing this stuff. If he were still alive today & heard about this incident, he'd probably laugh up an internal organ or two while exclaiming incredulously "What? They've STILL not fixed that fucker? And you call yourselves advanced? HA!"
I don't blame the company for the accident having happened, they probably haven't figured out how to prevent it yet, but what I _do_ fault them for is not being resilient enough to have prepared for just such a situation so they could work around it in spite of the minor setback. Drop fire suppressing foam to put out the fire, check. Send out a crew to clean up the mess, yes. But meanwhile have someone come in on the opposite side of the shelves to collect that affected parts bins, move said bins to a temp location, update the control computer to pick up $Item from $NewLocation, and continue fulfilling orders.
The folks at the SAC *demanded* that the place run 24/7/365 like clockwork to ensure supplies got to the troops that needed them; a week long delay was so far from "acceptable" that heads would have started rolling after the first hour of downtime. You don't tell a solder to stop firing their weapon for the next week while you find an alternate source for the ammo, you keep that ammo flowing if you want an effective fighting force. You don't tell the tank driver not to use 3rd gear until you can ship out a replacement part in a week, not if you don't want to turn your expensive tanks into lawn ornaments. Your (my BioDad's, this other company) entire business is delivering items just as fast as automation can go, so to have a failure in the system that you utterly failed to prepare for is just unforegivable.
TL;DR: Your failure to plan ahead proves you're not fit for duty.
Just fine for any length of time without issue then I'd put good money on it being caused by poor maintenance.
Think a nut on two of them where the wheels are being loose enough to contact, knocking one of them enough to hit another then topple over.
Either that or something started to escape one of the baskets removing clearance.
Any which way, it wasn't catastrophic and seems to have been handled well.
No. the robots were impacted - the orders were affected (unless of course the groceries got squashed when the robots collided, whereupon they might be legitimately considered to have been impacted).
I got so fed up with the universal use of "impact" recently that I spent half an hour with a thesaurus. In that short time I found more than 60 context-specific and informative words indicating ways things and events can affect other things and events. Impact conveys nothing more than "something or other resulted from this". Its universal use is the recourse of sloppy thinkers who aren't interested in, or can't be bothered to find out about, what actually happened.
But why would you be creative in the use of language? It seems nowadays people that put pride in their use of language get told off for "showing off" or "making it more complicated than it needs to be". And thus we get stuck with management buzzword bingo words like "impacted"
I'm currently wondering about how the term "self-isolating" has mutated in well under 18 months. It now means the same as "isolating" rather than carrying the previously somewhat specific meaning of "isolating / quarantining as a result of a decision one has made oneself rather than being instructed or forced to isolate / quarantine following medical instruction etc".
Quarantine itself has of course broadened from the original very specific Venetian word "quarantena" - a period of 40 days of isolation of ships, passengers and goods required by law in Venice during the time of the Black Death.
Although (in the style of Arthur Dent) I prefer to think of quarantine as more of a sort of "cordon sanitaire".
Since this is at least the second in a few short years, I'm wondering if there is either something wrong with the physical design or their processes - maybe they hesitate to hit the sprinklers too long?
Since they only have 5 of these centres (quick google could be wrong) 40% of them catching fire seems a bit worrying.
The collision was due to a loose antenna (the ones on top of the bots shown in the photo), which stuck out sideways and became lodged in a second passing bot. This somehow lead to batteries overheating and a rather large fire, perhaps due to the collision with the third bot. AC because...
Rumour has it that Ocado now trialling a new range of products in the Smoky BBQ robot range. This is like standard BBQ relish, but with a hint of burned rubber, plastics and lithium.
Seriously though, it seems that improvements implemented since the first fire have worked well to contain the problem. Good to see it was quickly contained.
I guess its not just 1% that had the fire damage that needs replacement, I'd expect that there would be a lot of water and smoke in other places that would need to be removed / products replaced etc.
Will be interesting to see how long this takes to resolve and get the facility fully back on-line again.
Unconfirmed reports are now claiming that the fire was started after the robots converged on the potato aisle, with the aim of converting the spuds into inedible instant mash.
One must wonder at the quality of engineering going on in the design and construction of these bots. "Does not cause a fire resulting in the callout of 20 fire engines when they collide" should be one of ther more basic design requirements.
That's not a mistake by the way : what is now being minimised as just a minor smoulder saw 20 fire engines rushed to the scene from across South London. Also, "A fault in a battery charging unit caused a robot to catch light at Ocado's Andover distribution centre in 2019.The site, which processed 30,000 orders a week at the time, burned for four days and was completely destroyed".
On the basis that the previous incident required a significant response I don't think sending a fleet of fire engines is misplaced, even if it did end up being a minor smoulder in the end.
That's why we need to have spare capacity in all our public services, so that they can deal with the unexpected. That something unexpected will happen is entirely predictable, it's just hard to know what the something will be.
Exactly that, one issue with these buidlings is that they have large amonia tanks on the roof. The Andover fire took longer to put out as at one point crews were focused on keeping the tanks cool and out of danger. The grid system also doesn't lend itself to getting around the building easily in full BA gear with hoses.
Based on the learning from the Andover fire, 20 pumps attending is a well reasoned first response.
"20 fire engines"
If you have a working fire reported in a large warehouse (especially one with LiPo powered robotics!), you don't send four guys in a truck to check it out. You roll a full response. It's easier, cheaper, and safer to send a bunch of crews home (orvturn them around while on route) if they're not needed. You don't want to get on site and wish you had dispatched more crews 10 minutes ago.
That many engines is more about manpower than machine power. You want enough people to move hoses, open doors/walls/roofs, search the entire building for trapped/injured/oblivious civilians, have a RIT crew on deck, have people ready to rotate in (air packs only last so long before crews have to rotate), oh, and you probably want a few actively fighting the fire. You also need enough command staff to keep everyone on the above tasks coordinated and safe.
"20 engines" probably includes command staff cars, rehab trailers, hazmat units, and other specialized equipment in addtion to the pumper trucks and ladder trucks we normally picture.
To lose one may be regarded as a misfortune, to lose two looks like carelessness. To lose three, words fail me.
If the factory is run by AI then it seems to need some education. How long does it take before the three robots failing to arrive at their destination becomes an exception requiring intervention?
Scene starts with the robots buzzing around on their tracks, they crash and the voice over sing "Love robots smashing each other up, their is an Ocado just for you"
Then the robots burst into flames the singers continues with "want all robots to die in a fire, their is an Ocado just for you"
I had imagined them taking power from an overhead grid system, like a fairground bumper car.
Seems like unnecessary complication, having to carry the weight of batteries around, and having to stop to recharge, with the risks associated with LiPo cells.
Easier to kill the power to the entire grid, too. No uncontrollable fires.
Am I under-thinking this?
From what I have seen on the videos they look very heavy and have battery packs integrated. Voltage starts to drop.. go to charging point sort of command. Lots of charging cycles and so on. One failed pack or even one failed cell would make a fair bit of smoke and drippy plastic. It might damage the grid too. Triggering the sprinklers would be a catastrophe. I cannot imagine these bots are specced to ip68.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
