Disaster recovery blunder broke New York Stock Exchange this week On Tuesday, the New York Stock Exchange (NYSE) failed to open properly, disrupting and invalidating auction trading for more than 250 securities. The NYSE in a statement on Wednesday said that "a technical issue" prevented the opening auction process by which stocks get priced through the reconciliation of orders since the …
I've worked in Fintech and every single penny has to go to the right place and reconcile with other systems. If there's a known fault, the data has to be dumped for manual correction. If there's a silent fault, normal logging has to be enough to correct it later. A bug can change your plans for the rest of the week.
Millions of dollars per hour was stressful enough for me. A cold beer to sooth the nerves of everyone working at NYSE (not to be consumed during work hours) ->
Critical systems do not fail because a person makes a mistake, but because insufficient controls fail to prevent the mistake.
-- Dr. Johannes Ullrich
in a recent SANS NewsBites e-mail, on the subject of the NOTAM outage
The problem, according to Bloomberg, is that an NYSE employee failed to shut down a disaster recovery system at the exchange's secondary Chicago data center. Because the system was left running overnight, the market software at the NYSE acted as if trading had already begun and prevented opening auction prices from being set correctly.
Doesn't sound very well designed DR system. Surely it would (should) know which system is primary and which is not and act accordingly.
This is merely the latest in a rather long list of major WTFs by this institution. It took the melt-up to get them to admit that timestamping orders with the time that they were processed, as opposed to when they were received was a mistake. And they STILL haven't addressed the issue of orders being valid for less than a second.
Two years earlier:-
Tech: So, in conclusion, factoring in design, planning, development, testing and installation this should cost $x. We then have added confidence that live and DR cannot degrade each other's performance. Does the panel have any questions?
Beancounter: surely it's cheaper just to write a line in a process document somewhere saying "do it like this" and file it in a locked filing cabinet in a disused lavatory...
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
