A friend of mine worked for BT and told me of an incident from about 15 years ago. As all the digital telephone exchanges are unmanned, any alarms from exchanges when things go wrong, appear on screens at a central control centre. The control engineer at the remote centre then contacts a field engineer in that area, who will then drive to the exchange and fix the problem.
At the end of the day from about 5 to 5:30 pm, a small number of night staff take over in the control centre. If any alarms appear from then on, the night staff ring the appropriate engineer at home. They do get a disturbance allowance payment if control ring at say 3 in the morning !
Sometimes, engineers will be working overtime past 5:30 pm and so will be bringing up false alarms. These are noted by the control day staff as 'engineer working' and noted by the night staff on changeover. When the engineer working overtime has finished, he will clear the alarms and inform the night staff that he is going home.
Well one particular evening when handing over to the night control staff, they were informed that X power engineer would be working overtime at the main Bristol telephone exchange in the city centre. This was a huge 8 story building with every type of telecomms equipment imaginable. Unfortunately, there was some sort of breakdown in communications between this power engineer and the central control centre. He cleared all the alarms when he finished, and went home.
About ten minutes later, some contractors, presumably also working overtime, severed a huge power cable in the road feeding the city centre, and the whole of the city centre was going to be without power for at least 24 hours. But not to worry, because the main telephone exchange had a massive generator the size of a pair of semi's, and a back up of the same size in case the first one went kaput. Just one snag, on this occasion the first generator didn't kick in, and neither did the second one ! Apparently so my friend told me, all telephone exchanges have back up batteries for just such an occurrence and although they're to the same spec as the ones used on the moon landings, they only last for an hour.
But back to the night control staff and alarms coming up all over the place. "Blimey" they thought, "X the power engineer must be working like mad to bring this lot up !" What they didn't realise was that these alarms were real ones ! Eventually when the alarms started to tell them that the back up batteries had kicked in and there were just 40 minutes left before the whole building went dead, they frantically phoned X on his mobile to ask him what he was up to. They got the shock of their lives when his wife answered and said he was in the bath.
By the time X got back to this main exchange, everything was definitely dead. Despite other power engineers rushing in from everywhere, no one could get either generator working again. Mobiles, computer links, 999 services, you name it, everything was dead as a Dodo. An emergency was declared and with managers now on site tearing their hair out, a huge mobile generator was ordered to make the journey asap from London on a massive Pickfords truck.
It eventually arrived and as it was backing into the yard, someone noticed something. It wasn't going to clear the large brick archway. According to my friend, the reaction of everyone there would be totally unrepeatable.
Tyres were deflated on the Pickfords truck but all to no avail. An urgent request went out to every local BT stores and also local electrical suppliers to see if they had any appropriate cable, two pieces, each about as thick as someone's leg and at least 100 metres long.
The mobile generator eventually roared into action and as it was by now 6 am the next day, this was no mean feat. It took another 3 hours to get the exchange up and running again but the original breakdown in communications was never satisfactorily resolved. However, my friend said that the generator problem was. It seemed that despite being regularly tested, maintained, and run, by a million to one chance a different part failed on each one at the same time that could never had been anticipated. To get those parts would have taken far longer than bringing in a mobile generator.
It was a classic case of 'Sod's Law' i.e. if something can go wrong, it will. This was a zillion to one chance because three things went wrong at the same time. The breakdown in communications; both generators failing at the same time; and the brick archway being too low.
The name of this incident has been changed to Bristol, but the city where it happened was even bigger apparently.