The random expiry time
A large bank used a messaging product to send information around its systems.
Some of these "what is my bank balance" messages were not important, expired after 30 seconds and were thrown away - the requester could always resend.
Some of these were "transfer $100 Million" messages - and these were not allowed to expire. They were logged to disk.
Unfortunately a teeny-weeny application change put a random expiry time into the data.
The first the bank knew there was a problem when there was a phone call "Where is my $100 Million?"
I worked for the company that provided the messaging system. I got a call 10:00 from a stranger who explained the problem and asked if I could go on site to help.
I said I was happy to, but the customer would have to pay for a plane ticket etc. (That usually puts people off). I strolled over to my boss to give him early warning - but he was away from the office. I got back to my desk and had an email " there is a ticket for you on the 1200 flight to ... if you can make this it would be great"
This is when you think "They are serious". I left a note on my desk, booked a taxi - rushed home, picked up my passport, and change of underwear etc. I got the (business class wow!) flight with minutes to spare.
At the far end I was met and taken to the customer site. I knew the confidential layout of the log records on disk, and between us we came up with some rules - if this value is .. and that value is - then print out this other data. Some people then worked through the nigh and re-entered the expired data.
By 0900 the next morning, it was all fixed.
They then told me the true scale of the problem. They had "recovered" billions of dollars which was good. The banking auditors were due to arrive that day at 1200 for the annual review. If they found there was money missing, the bank would have been closed down.
Biting the hand that feeds IT
