Google Cloud shows it can break things for lots of customers – not just one at a time In the week after its astounding deletion of Australian pension fund UniSuper's entire account, you might think Google Cloud would be on its very best behavior. Nope. At 15:22 last Thursday, US Pacific Time, Google Cloud ran "maintenance automation intended to shutdown an unused network control component in a single location …
I see nothing has changed in the past 10 years. Google is still making highly technical configuration changes on Friday before everyone goes home.
The funny thing is that some companies use the cloud so that they can bring up a canary instance, apply changes, and see if it's still alive enough to pass basic tests. It's a cheap way to borrow $$$$$$ of hardware for just a few minutes for testing.
> Google is still making highly technical configuration changes on Friday before everyone goes home.
Jeez, yeah, this used to drive me nuts when I was still a dev. Let's release critical changes when everyone in our location is scattering for the weekend (especially when they know a release is happening), the rest of the world has already gone home, and at least 50% of people will soon be drunk...
Big comms centre. In the corner is (still) a comms cabinet roped off containing a handful of 9600 v29 modems.
Big stern notice attached to the rope barrier advising that 'under no circumstances must the rack be touched' by facilities management
Yes, they could save a few pennies by switching off the cabinet to see if anyone noticed... but I don't think they would make it out of the building!
Some days Too Big To Fail is also Too Big To Succeed. I'm not saying you're wrong, but I've done a ton of stuff on Friday because not everything fits in the better days of the week. You may as well lambast them for having bugs in deployment code, which is both the obvious goal and unavoidable. You do your best to mitigate and then roll the dice.
.... you either better have a really good backup and go-to-carp plans or be prepared to suffer the consequences.
I've seen companies have business plans that require staff to work 50 hours/week and a vast amount of what they do is outsourced while trying to represent themselves as a traditional company. I discovered this at interview and through talking to other employees. Needless to say (I hope), I didn't take the application any further. I can work overtime when there's a rational need, but not 10 hours per day 5 days a week or 8/day and 6 days a week. There would be nothing left in the tank for those last minute marathons to get something done. What all of that tells me is that the company plan isn't viable. I don't have a problem with a company using a cloud service as part of their backup plan, but when that outside company becomes a critical component of operations, that's where it becomes a big problem. They have no control over any plans that outside company might have for changes or a discontinuation with little notice. An outage could also be a company killer.
The Clown was supposed to reduce costs -- it didn't. The Clown was supposed to reduce headcount -- it didn't. The Clown was supposed to reduce complexity -- it didn't. The Clown was supposed to get you off the IT treadmill -- it didn't.
Why are these customers paying to rent someone else's computer for, exactly?
Yeah, this is one of the many myriad reasons why putting your entire critical infrastructure onto a cloud that you don't have full control over is a fools move.
Front ends, non critical edge use, sure thing, great applications for public cloud.
Your main infrastructure? Might as well just hire all your staff straight from Uni and cross your fingers that they never have to deal with things experienced admins have already learned from.
Which is what it appears everyone is basically doing these days, crossing their fingers that these public cloud services have hired competence and integrity.
Because when you do this that is all you can do: hope.
And hope is not a viable computing strategy.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
