Techie's test lab lands him in hot water with top tech news site Monday morning arrives once more for those of you holding the fort while colleagues are on holiday.So why not enjoy this extra special instalment of Who, Me?, El Reg’s weekly confessional column. Because this week, Vulture Towers plays a leading role in the tale that our brave reader "Victor" recounts. "Perhaps 13 years ago, …
So The Reg didn't have a redundant backup elsewhere? Just two servers in the same place?
One day you really need to write an article about the tech behind the site, because sometimes it just sounds quite worrying.
It's like having a bunch of sports journalists who comment on every match but have never had any involvement with the sports themselves.
You mean in 2005?
Er... yes...?
"Geographic redundancy" = "buy one each from two different companies, rather than two from the same".
I'm not talking hot-swap, fast-failover, high-availability, but I'm also talking "website doesn't fall over just because someone pressed a switch".
Go hit Google for "site:theregister.co.uk" and change the date range from to 2005-2006. They were just as big, just as sarky about downtime, doing articles about grid-computing, etc. etc.
We're talking the .xxx TLD era, not the dark ages.
Back then it was not as trivial as it is today - it was somewhat expensive to do properly with some pricey F5 GTM devices.
Two well setup servers behind a load balancer would give you 99.99% without breaking a sweat so it's understandable why they only had two servers back then.
At this particular hosting company we had staff operating 24x7 who would run over with a crash cart the second anything played up and we had a room full of spares.
So really unless you are doing something requiring a lot of reliability, why bother with geo-redundancy, it's more ballache than it's worth in many cases.
So really unless you are doing something requiring a lot of reliability, why bother with geo-redundancy, it's more ballache than it's worth in many cases.
This. Sometimes being 'good enough' is really 'good enough'
Also, best practices have to be married with some level of competence too. I remember a previous company I worked at had their primary B2B site running on a solaris box in the server room with an apache front end hosted by a 3rd party. No real redundancy (other than the text box that could be repurposed if needed) however it pretty much just 'worked' for most of the time as the servers were setup correctly and well maintained. (Sure, we were at risk of serious outage if a fire or flood occurred, but still).
Years passed and we got acquired by another company, they came and gave us the fancy presentation on their shiny new data center. The site was moved to this wondrous place, hosted across 6 servers with load balancing and all the 'best practice' boxes well and truly ticked.
Unfortunately it ticked boxes, but perhaps wasn't set up quite right, that site had more downtime in the next 3 months than it did over the previous 5 years. (Mainly due to randomly crashing servers, sticky sessions and load balancers configured to blithely ignore if the servers were actually up and responding or not...) Whilst the site was now more 'disaster proof' our customers were much less happy.
Sometimes working is good enough.
(Ironically, we did have full multi site geographically separate DR for our core systems, it's just this new fangled web thing had appeared in the meantime...)
IMO, that is no worse, probably better, than say the BBC's practice of "employing" a multitude, let's call it a magnitude, of ex-pro sports people who endlessly babble on about mundane matters or the blatantly obvious, or their relentless agenda. At exhorbitant rates too.
The less sport they show, the more they chat about it.
[The boss] said, ‘I’m not worried that you broke something, that's expected, the most important thing is how you deal with it’,
And that is how a boss should react. Shouting at people for making mistakes doesn't mean fewer mistakes are made. It just means more are covered up.
Admit the mistake and learn from it. I know it sounds like management bull****, but mistakes are fantastic learning experiences. Just don't learn by mistakes too often...
And that is how a boss should react. Shouting at people for making mistakes doesn't mean fewer mistakes are made. It just means more are covered up.
Not just bosses. One of the reasons I've always had the keys to the server room & the root password is that the admin not only has reasonable trust that I won't screw up, he knows that if I do I will call him, own up, and help to fix whatever I did.
I heard an interview with an old actor who did some TV in the early 60s. Back when all TV plays were broadcast live, as recording was fearfully expensive.
So the director is doing his final pep talk. Firslty the helpful, "Remember if you fluff you lines just keep going, there's no time to stop." Then the less helpful, "and anyone who dries up will never work for me again!"
Because I'm sure that helped with nerves... What an arse.
On a similar vain, I heard the head of a sports body said to an athlete as the athlete went out for their Olympic qualifying trails: "The future funding of our sport depends on you qualifying for the Olympics."
Yep, the athlete crashed & burned within seconds.
A textbook example of how not to give a motivational speech.
Literally just had to bring up a server that was refusing to come back up.
Long story short, new servers being built. To facilitate the new servers we power down a QNAP NAS temporarily, plug it back into the correct UPS and power it back up. At this point I'll explain that the job of doing this is actually that of the IT contractor who has come in to build the server.
So the QNAP is powered down, plugged back into the correct UPS and all is happy. Until about 10 minutes later when the file server starts becoming non-responsive. Decision made to reboot the file server, but it won't boot back up (it's a VM). Turns out it's complaining about a missing virtual drive.
We disable the virtual drive, boot up the server and check to see which drive appears to have failed. At which point I notice that the drive that's actually stored on the QNAP is the missing one. And the IT contractor suddenly remembers he forgot to turn it back on. QNAP turned back on, missing drive added back to the VM and all back up and running.
In case anyone thinks it looks like things are now back to normal, we then find out that the new server can't be built because a vendor who's name rhymes with Hell has installed the wrong HD controller card and the nice shiny hard drives can't be configured in RAID. So project delayed by another couple of days.
Fortunately, Victor learned his lesson, so El Reg's servers will never again face the risk of being inadvertently shut down at an inopportune mo
Broadcast message from victor@highlyimportantserver.theregister.co.uk
(/dev/pts/0) at 16:50 ...
The system is going down for halt NOW!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
