Asleep at the wheel: Why did it take 5 HOURS for Microsoft to acknowledge an Azure DevOps TITSUP*? In an impressively frank postmortem, Microsoft has admitted that at least part of its organisation was asleep at the wheel in a very real sense while its European DevOps tooling tottered. The travails of Azure during the current surge in usage are well-documented but, as well as showing the limits of cloudy tech, the pandemic- …
Those of us eschewing the concept of allowing someone else to run our vital computing needs, instead opting to keep it all in-house, have had no problems. That's none. Zero. Zilch. Everything works as designed and implemented.
What good is a computing infrastructure that grinds to a halt at the first sign of traffic? Especially traffic that your corporation is utterly helpless to help direct?
Have fun watching your clouds evaporate this fine Spring day :-)
In steady state, yes. Just out of interest though, if a new team has a requirement for additional build nodes for their their CI/CD pipeline, or a new project starts up with a major new build requirement, how long does that currently take you to fill, from the moment the ticket is filed to them being fully operational? How long is it if your virtual infrastructure is full, and you need to procure more servers? How long if your datacentre is near full, and you prioritise production environment capacity demands?
Perhaps your environment is different, with great capacity management, managers who are prepared to fund significant spare capacity ahead of known demand, and suppliers who don't let you down. But I've seen teams stuck for months with hugely inadequate CI/CD capacity in large enterprise environments, with massive impacts on productivity - far more than a day or so's Azure DevOps outage would have. The old-school workaround of sticking beefy tower PCs under half the teams' desks and running the Jenkins jobs through those tends to be (rightly) blocked by security nowadays, so it's a real issue.
Don't get me wrong though, the Microsoft outage was still shit.
" day or so's Azure DevOps outage"
But it's not a day or so, is it? Not a week goes by without ElReg reporting the various clouds going down, being broken into, slowing down, and what have you.
I guess in your scenario of fly-by-night industries & jumping about like a frog in a frying pan corporate "visions", where profits are often calculated on an hourly basis, clouds might make some sense (unless it all goes TITSUP, of course) ... but I don't have time for that kind of headache & ulcer inducing bullshit. Sane people calculate their profits quarterly or yearly.
And yes, I consult for people who have great capacity management, who are prepared to fund significant spare capacity ahead of known demand, and suppliers who don't let me down. So can you. Try it, you might like it. It even pays better.
You are both right, for a given value of right but I'll side with Jake here!
My own bijou cloud at the office has failed to rain for quite some time (years). Actually, even my home phone exchange has downtime measured in minutes (updates) per year and yes I could do better but I can't be arsed with a home PBX cluster: that's ridiculous.
It's not quite as simple as all that. With big cloud you should be able to flex fast in response to huge requests which my little cloud could never manage. My little cloud on the other hand has people who give a shit about named people and their workloads looking out for them. It also has better uptimes for now. Even now I'm running Windows updates on customer systems, some of whom should be doing their own but don't. I do it because I can and my thumbs (and Icinga) twitch. I live next door to a large park and can get out whenever I like 8)
It is how you deal with the mistakes that counts. If you review your processes, revamp what's necessary, and make your service better while compensating the affected parties as best you can - I can hold no grudge.
If, however, you continually spout platitudes while being too busy hauling your loot to the bank to acknowledge and rectify issues - nuke from orbit should be an option.
At least there is someone to talk to, even if they are asleep for most of your working day.
With Google, on the other hand, calling them results in 10 minutes of being pushed around an automated phone system, before a message saying check out the relevant part of the Google website and being spat out.
Writing an email results in an auto-reply, saying that they receive so many messages, that they are auto-deleted and never read, please see the relevant part of the Google website.
In my case, there was no part of the website I could find that dealt with being DOSed by a Google server in California...
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
