Tech support done bad sure makes it hard to do tech support good 2024 has commenced, but in today's edition of On Call – The Register’s reader-contributed tales of tech support strife – a reader we'll Regomize as "Stuart" shared a tale caused by a temporal anomaly. "About two years back I was working for an online trading company as IT support manager and was on call for the week," Stuart …
I don't think that was the issue here.
The issue was the previous tech only did half a job. Why didn't they finish it? Who knows. They may have decided they couldn't be arsed. They may have decided it was enough. They may have been called away for a family emergency and never returned to finish it.
To be fair, the story reads like the feed was technically the same, probably just a slightly different SLA.
A migration shouldn't have involved changing more than the IP address (preferably while both are active so you have a fallback), and all was good.
The downtime wasn't caused by saving costs, it was caused by poor project management and/or documentation.
I've also seen some disasters of doing things on the expensive. Mostly, it's people who don't want to spend the time to build something right, so they figure that the expensive option will mean they can skip that part. Alternatively, people who actually do get things set up properly, but spend so much on the expensive things that they eventually run out of money for other things they also need, usually just at the level of the project budget rather than the entire company, but I'm sure other companies run into a similar problem.
This situation looks unrelated. I see nothing here that says the private feed they were using was required over the cheaper option, so someone who told the company that they should stick with the private option to avoid this mess would be needlessly spending money. What they actually needed to do was to properly implement the new option they switched to, but that's on their processes and the vendor's docs, not the decision to switch feeds.
I know Scott Adams has stuck his foot firmly down his own throat of late, but there were times when he had amusing, and insightful, observations on these kinds of things.
The short-lived Dilbert cartoon show had a Y2K episode where the main cast is talking about a fabled mainframe in the company called Black Betty. The PHB comes along and says something to the effect of how the company considered replacing it years ago, but instead decided to slave all the new systems to it with outdated coax cable and miles of spaghetti code. When asked why, he says how everyone involved figured they would have long since moved on to more profitable companies by the time any issues arose.
Stress?
You call that stress for a trading floor support market data feed issue lasting "several hours"?
When I did trading floor/dealer support for the London branch of an Italian merchant bank, the main "qualification" needed to do the job was the ability to get to the end of each day without:
a. Having a total nervous breakdown from the "customer attitude" and demands.
b. Having "lost it" with one or more of the traders and turned the area into a veritable blood soaked buchter's yard of dismemberred body parts and internal organs.
c. Having no access at all to firearms and other weapons including implements sharper than a spoon. The tempatation to come to work the next day and use them would be just too great.
Any technical issue which negatively affected dealers' ability to trade (and thus make money) you knew about in under 5 seconds from the very loud shouting/screaming/very foul language verbal abuse whch would be directed at IT.
Anything lasting several hours and they would have been swinging from the ceilings while chewing up their custom build "dealer desks" while threatening to lynch whoever in IT was on duty.
The cost of the market data feeds themselves were aboslutely eye watering. Not because they contained any "private" data.
Most of of the data was publicly available free. The free stuff was, however, typically 10-20 minutes old which for a trading floor was prehistoric. You even had different "levels" of data feed, the main differential between each level being how "up to date" it was. The closer to real time it became, the more digits you added on to the monthly cost of that feed.
Having dealt with dealers in the "Wild West" (Perth Western Australia) I can confirm that every minute the system is down or not available results in dealers banging on the roor with their very expensive custom keyboards. We were fortunate that we organised the security systems and had a lock on our door that only it staff and certain senior staff had access to.
Unfortunately that did not extend to the phones, and one day I resorted to the, "the longer you tell me how much money you are losing, the longer it takes me to get off the phone to fix it" line whilst my boss was in earshot. He told me to ignore the phone and fix the issue, there were enough others in the office who could answer the phone. A great boss and one I miss. He looked like the PHB from Dilbert but was the exact opposite.
As with in a gold rush you make money by selling mining equipment, you make money on the stock market by selling trading systems. I used to work for a data and trading equipment supplier for the HK Stock Exchange. They had so many data lines coming into the office that HK Telecom installed an exchange on the next floor for our exclusive use. ;)
I remember hearing a story about some guy who ran a trading house in the early days of the Internet. He talked about how some guy came into his office one day and wanted to sell him access to this new line he set up direct to the NYSE -- IIRC, the potential customer was in Chicago, so about 1500 miles away -- at some ridiculous price of like $10K/mo. Don't quote me on the numbers, but it's not really important to the story anyway. At first the trading house guy just yells at the guy to get the fuck out of his office, trying to sell him something so ridiculously overpriced, but before the salesman could even make it to the elevator the trading house guy realized he didn't have a choice if he wanted to remain competitive. A couple milliseconds could mean the difference between making or missing out on some serious money.
Reminds me of Michael Lewis' excellent book, "Flash Boys", which describes the absurd lengths (and expense) gone to in order to save milliseconds on market information transmission -- and what nefarious uses those milliseconds were then put to. Great read!
But then does all this "faster faster" approach make accidental stock market crashes more likely by everyone triggering "sell sell sell"? I think they have a circuit breaker that trips the stock market if the transaction volumes exceed a certain value???
How much suffering could have been avoided if the ticker server(s) had been configured from a DNS entry?
eg ticker.themarkets.com.
One doesn't have to been in this game for as long as moi to realize names invariably outlive addresses.
Even after acquisitions and other skulduggery a vestigal CNAME to the new band of thieves usually persists (for decades in many cases.) (Musk and other lunatics excepted.)
The problem in this scenario is that they were using the IP equivalent of private123.ticker.com, but should have moved to public.ticker.com when they cancelled their sub to the private ticker.
In this case, it's likely the private123 domain would at some point be removed, rather than CNAMEd to the public domain, so issue would still occur.
Using domain name is also not infallible. I've had plenty of cases at multiple employers where outbound access was restricted in firewalls using IP addresses (layer 3), but apps were connecting outbound using domain names (layer 7). If the vendor of the service you are connecting to changes IP range without prior notice, that still leads to connection failures. These days layer 7 whitelisting in firewalls can resolve that, as can setting up special prices to manage outbound connections.
Prior to having access to firewalls with layer 7 capability, our best solution was to use a tool we had built for outbound throttling, but using it as a proxy for connections that were going to cloud based services that could not guarantee the IP range they would be using. It was usually anti DDoS measures (and then cloud) that meant we had to use it. We found it also provided a way for us to also lock down some of our servers, so we had less holes opened up in the main firewall
>In the bank I work for IT security rules don't allow using FQDNs instead of IP adresses for that reason.
I still shudder at the world of pain when we tried to get a bank to accept the use of a dynamic range of IP addresses for O365 connectivity. Some are FQDNs, some are IP address ranges - which change on a periodic basis.,,
I used to work for one of the biggest insurance comparison sites in the country running the network.
I ended up doing a detailed explanation of how dns worked. As the highly paid programmers couldn’t understand that they needed to allow a drain timer before flipping the application from one set of servers to the second set.
They found it hard to understand the TTL on the dns cache records. It took about an hour before they usnderstood the reasoning and why setting the ttl on a customer facing website dns entry to 5 seconds was not going to work properly,…
This was before Anycast addresses were widely supported.
In the high speed trading world, every fraction of a second saved can mean big money. Even if it only took a couple milliseconds to look up the IP address via DNS, that could mean missing some momentary fluctuation in currency values or some other market, and cost millions. Just like how companies pay grocery stores more money to have their products put right at eye level on the shelf, companies pay through the nose to have their servers as physically close to the main systems for the stock exchange as possible. They measure everything down to how long the connection cable is, and how much delay that adds to the signal transmission. It's all well below the human threshold of perception, but the people who own the physical stock exchange building and can lease out each space in the server room, make some serious bank. They can practically write their own check for those spaces right next to the main server and people will pay it gladly.
I'm sure all of us have had a moment when an alert pops up and no one has any idea (a) where it's coming from and (b) what it does.
Little tip: if you ensure you do a grown-up test of your UPSs by actually shutting things down (because as long as the mains is connected the battery remaining is a guesstimate you really don't want to rely on) you can winkle out some dinosaur processes.
Also restarting your comms to flush out DHCP reservation issues.
This is the way it works… save some money, never look back*. The bean counter who made the proposal is long gone, the manager who approved the proposal has either moved on, or is untouchable due to his shimmy up the corporate pole. Enter IT guy who was hired after the own goal*, and has to clean up the mess, yep Standard Operating Procedure…
I spent some time in the down in the States working for a vendor of medical office automation hardware/software. If you had a support contract with our company, our fee was $95.00/hour. In order to have a support contract with the company, the customer had to pay for a dedicated modem line, and an external modem of at least 2400-baud capability. We required external modems because they had the necessary status lights (LEDs) for us to diagnose comms problems over the phone. The line itself was for us to dial into their Xenix or Unix system and work on the OS or on the app, as needed.
If you did not have a support contract with us, the fee was $295.00/hour, including travel time.
Enter the cheapskate customer who refused a support contract, due to the phone line+modem requirement. This doctor was adamant he was not going to pay for this useless phone line and modem. Their office was 110 miles away from ours. After my fifth callout to their office, his better half convinced him to pay for the phone line, modem, and support contract.
I was technical lead on the Ericsson dealer network project, Ericson had been learning pseudo-staff management from Sun Life of Canada. So we had fields for 1st emloyees birthday, second employees birthday and other such bullshit down to about 10 employees. One of the questions I asked was "What happens if there are more than 10 employees?" and was told "Oh shit". And of course these values were all held in a single data row on a SQL-Server 6.5 system... Normalisation was apparently what happened when you changed countries, like becoming English rather than Scottish.
So I had an incredibly short deadline to meet <2 months to have a working system. In a spirit of informed hatred a USA company was competing with us for the contract - we were supposed to share things but the mutual hatred meant that was improbable at best.
It was decided that the best testers of each system would be their competitors and whilst I was doing some tidying up I got a phone call from the States - telling me I had fucked up bigtime, and they were going to win as our system didn't work at all. They sent me some screen shots to prove it, and they were right. It didn't bloody well work on IE2, which is what they were testing our system on.
Now if they had only read the spec carefully they might have noticed that IE3 was required.
I read the briefing notes for testing and in them it was made clear that the debug results were not to be shared other than with Ericsson. So i didn't. I did ring one of their chaps whom I knew quite well and enquired whether or not the system was required to work on IE2 and was told "certainly not - we want the latest and greatest features of IE3". So I closed the call having sort of mentioned that it would be a good idea for them to look for the identifying features of the Browser used for testing...
Yep - read both manuals and specs and ensure you're doing what is requested.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
