back to article Overheating datacenter stopped 2.5 million bank transactions

Outages at two banks that stopped 2.5 million payment transactions were sparked by a technical issue with the datacenter's cooling system, according to the Monetary Authority of Singapore (MAS) on Monday. DBS and Citibank, the banks involved, experienced outages in the mid-afternoon of October 14, 2023 that resulted in full or …

  1. Korev Silver badge
    Coat

    > Running infrastructure in the tropics has its challenges – but so do failed disaster recovery plans

    So the plans were Singapore then...

    1. b0llchit Silver badge
      Coat

      Shouldn't that be Sangapore or maybe even Sungapore?

      cheek, you let my tongue escape!

  2. Doctor Syntax Silver badge

    Reed called it "odd" that such a core system like the handling of authentication for an online bank would be managed by a third-party provider.

    If it can be outsourced it will be outsourced. Nothing odd about that these days.

    1. Cliffwilliams44 Silver badge

      Many banks online payment systems are outsourced.

      That was a primary reason I left PNC when the bought BBCA/Compass. The 3rd party app was horrendous!

  3. tmTM

    Backup?

    If your backup datacentre doesn't function properly, it's not a backup at all, it's nothing more than a waste of money.

    Maybe test these things once in a while?

    1. Anonymous Coward
      Anonymous Coward

      Re: Backup?

      Real failure is the most authentic kind of test a DC could ask for.

  4. Anonymous Coward
    Anonymous Coward

    This is what happens when you have unmanned DataCenters

    Been to many "unmanned Datacenters", when i say unmanned you just deal with a security guard who's job is just that, nothing more, that's the only monitoring that is going on. Servers could be cooking for all they know or crashing. I have walked past racks that have been bleeping for weeks or red fault lights blinking away till the led gives up.

    1. Doctor Tarr

      Re: This is what happens when you have unmanned DataCenters

      I get your point but the DCs should have correct monitoring for those possibilities. Also inert gas fire suppression isn't compatible with meatsacks.

      I once got a bollocking for being in the DC of a catalogue based high street retailer. They didn't know we were working in there. They also didnt think to question how their security had let us in tho.

      1. Alan Brown Silver badge

        Re: This is what happens when you have unmanned DataCenters

        You don't need inert gas suppression to completely reduce oxygen levels to lethality. Very little burns at 16% oxygen levels and some DCs run 17% by default (yes, you DO feel it and it affects mental processes)

      2. Anonymous Coward
        Anonymous Coward

        Re: This is what happens when you have unmanned DataCenters

        >Also inert gas fire suppression isn't compatible with meatsacks.

        Modern gas suppression systems are compatible with meatsacks (and are MUCH more compatible than combustion byproducts).

  5. iansn

    Global warming???

    I visited regularly, lived and worked in Singapore since 1979. Its hotter in the city because of all the concreteand heat from the traffic. But it's no warmer anywhere else. And it's on the equator FFS. Is that the best you can do?

    1. David Pearce

      Re: Global warming???

      Singapore has hit all time temperature records this year. Even the equator is getting hotter

  6. Ken Moorhouse Silver badge

    ...the valves...

    There's your problem.

    Oh, maybe I will admit to taking this out of context :)

  7. ecofeco Silver badge

    What was the famous quote?

    The more they overthink the plumbing, the easier it is to stop up the drain.

  8. bernmeister

    Backups that fail.

    Backups that fail are a common problem in the industry. Backups get maintained but often not tested for fear that they will fail. Chernobyl disaster was reputed to be the result of a failed backup test. I personally crashed a system by switching over to an incorrectly set up backup. Luckily I was able to switch back after a few minutes. Nobody noticed except for the maintenance man and I. My boss never even knew about it. Whew!!

    1. Deni

      Re: Backups that fail.

      Right. It is a good strategy to switch between active and backup regularly. Back and forth, active/active really. Its the only way to be sure your backup is ready to work when needed.

  9. untrained_eye_of_newt

    In the old days when I worked graveyard on tymnet betore the internet I was rotating packs every other night from various errors. A pair running and spare on the shelf. I loved that job, still remember all those people, and had I kept that job I would've dodged being sucked into relationships because I wouldn't have had any money and so not suitable to bear a lot of responsibility for other bloke's children to then be kicked to the curb later after being used up just like the other bloke.

  10. An_Old_Dog Silver badge
    Coat

    Mr. Tan Hit the Nail on the Head

    ... but technologists, business managers, and governmental employees are not the only ones to blame. People in general seem to have become, in some respects, "stupidized" regarding backup systems, manual alternatives, and the need for them. People no longer demand systems reliability and resilience.

    Two contrasting incidents:

    (1) I was inside a major retailer's store whose power had failed. The building had a very tall roof, and large windows to let in sunlight. It was late afternoon, but people could see well enough to shop, even with the lights out. The electronic cash registers didn't work -- there was no backup electrical power system for them. They had failed over to a manual system: Each checkout operator had been equipped with a battery-powered adding machine, with printed paper tape. The store had no electricity, yet it was still doing business!

    (2) I was visiting a technophile couple (they both work in technical jobs for a major CPU-and-other-chips manufacturer). They own a Tesla automobile. The wife was moaning to us about how a friend of hers had suffered a smartphone GPS failure while driving around the city, and had became quickly and completely lost, and how horrible and devastating this was for her friend. I asked, "Why didn't she just use the map from her glove box?" His indignant reply: "Nobody uses maps any more!"

    In the megalopolis where we live, there are clear, legible street signs on every corner (though a few have been stolen or mis-pointed). They work even for persons lacking maps.

    I couldn't think of a reply that did not include the words, "fucking", "arrogant", "incompetent", "absolute", and "ninny", so I said nothing.

    (Mine's the coat with a plasticized city map in its pocket.)

  11. Frank Bitterlich

    Impressive Response

    I have to say I'm impressed with the response from the regulating authority. Instead of slapping a meaningless financial penalty on the bank (which in the end is paid by the customers and low-level employees anyway), they basically ordered them to stop playing around until they have fixed the mess.

    Typically it's the other way around; they get a massive fine, and in response, close some branches and fire part of their workforce.

    1. t245t
      Terminator

      Re: Impressive Response

      They should be seeing jail time. But then again, culpability cannot be laid at no one individual.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like