World+dog share in collective panic attack as Google slides off the face of the internet Google services such as YouTube and Gmail started the week with an almighty bang as the Chocolate Factory's cloud came crashing to the ground. Despite an insistence from the company's various health dashboards that all was fine and dandy, it most definitely was not. Those seeking distraction in video form were treated to …
I've been using Qwant as my default search engine for months now, and have found it to be really quite good. Previously tried to use DuckDuckGo as the default but found myself constantly repeating the search in Google - not so with Qwant!
Exposing friends and foe alike to naked robots with their innards on show. Not just gmail and YouTube etc. (my son found he could still access YouTube through a desktop browser, just not on his device, so could continue to chortle at those kittens and their antics); gsuite was borked too. Looked like a serious authentication infrastructure boo-boo. I was told that my gsuite email address wasn't recognized at all when attempting to login. Ooof.
It always makes me chuckle when you get error messages that say
"Try again in five minutes"
Why five? Where has that figure come from? They don't even know what the error is! So how can you reliably determine when someone can "try again"?
But it's somehow better than "try again later". When is later? 10 mins, tomorrow, next year?
Either way we'll all keep madly hitting Ctrl+F5 until we get the response we want.
Depending on the system, it's a reasonable approach because it acknowledges there is a problem and sets an expectation that the problem can be fixed soon — this is a much worse word to use — and that you should do something different for a couple of minutes rather than pressing refresh all the time.
A major outage like this can never be ruled out but should have some form of recovery plan. Individual plans for each system should also exist and are easier to test.
Makes me wonder how technical you actually are. What should they have said? "Our systems are currently overloaded. Please do your part by refreshing immediately."?
I understood that the "please retry in 5 minutes" was an attempt to limit the incoming DDOS by their legitimate users long before I was an SRE.
Well, they'll have a job logging in to StackExchange sites to find solutions, since those rely on JavaScripts hosted by googleapis (much to the annoyance of many).
(Makes you wonder how many sites will also be borked if Google's captcha service is also down: even if you don't use sign in with Google, they still try to ensnare you...)
Thank you, I now have a lovely mental image of nervously-sweating Google engineers frantically Bing/Qwant'ing for things like
- "how to reboot a server"
- "help I forgot my login password"
- "what does read error mean on floppy disk"
- "does anyone have a backup of google infrastructure"
...while managers scream at them :)
As reported by the BBC, a guy on Twitter said: "I’m sitting here in the dark in my toddler’s room because the light is controlled by @Google Home. Rethinking... a lot right now."
And THAT is why I don't have anything essential in my house controlled via the internet in any form or fashion. But you just can't tell people that - they have to experience the downsides for themselves before they get the clue!
Mine all have a local controller on the inside of my network as well as being RF controlled without IP so I don’t need internet or network. The light bulbs I have (those that are smart) rather sensibly also turn on if the supply is interrupted and restored so I reckon Twitterman is spouting bullshit.
IoT was crushed by corporation demands to control all the data, resulting in cloud services the users don't control. It's lovely to see the people that didn't realise this get a helpful education.
My favourite quote on the BBC though was "Google has been contacted for comment, but one spokesperson said they were unable to access their email."
Most mobile phone companies around here have repair crews equipped with....wait for it....phones linked to their competitor's networks. Because, obviously. Repair crews, obviously.
I wonder how long it takes Microsoft to switch to posting their status on GCP and Google to set their email backup plan to Office 362.
Until they're both down. Then, what?
“ IoT was crushed by corporation demands to control all the data, resulting in cloud services the users don't control. It's lovely to see the people that didn't realise this get a helpful education.”
Not sure about that.
In the late 80’s we had a variety of remote controlled plugs, different packs all incompatible with each other.
IoT means a common controller being Amazon, Google or Apple.
Guess who has the most privacy conscious IoT implementation.
If you want more private IoT you need the device to do most of the processing, and the cloud do what is commanded by you and your IoT.
Case in point is cctv on device person recognition, not cloud recognition.
Buy cheap pay twice.
In the late 80’s we had a variety of remote controlled plugs, different packs all incompatible with each other.
We had those in the 1580s and compatibility was ensured by a bit of beating & heating. HR keeps those in a locked cupboard. Although clothing may differ in the 21st Century, process is much the same. Once escorted away by security, you don't see them return. Just an occasional reminder from HR to return their pressure washer immediately.
IoT means a common controller being Amazon, Google or Apple.
I suspect some may be reconsidering that world view when all their Io Things became inaccessible via their Apple or Google phone, or Amazon app. Just as well that in the good'ol days, T meant 'thumbs', which most people still have. Despite best efforts of IoT vendors to disable those via shrink-plastic packaging.
…is figuring out whether 'failsafe' for a door lock means it's locked or unlocked if the Internet or cloud or power fails. Either way can be pretty bad. Which is why, while I'm not totally opposed to such 'smart' IoT things*, door locks will be about the last thing I'd 'upgrade' that way.
* Brought to you by the Redundant Department of Redundancy Department. Ask us about our PIN number for the ATM machine!
A school local to me found out the hard way that their electronic door locks (the only type they had on them at first) were "fail safe". When they did have a power outage, they discovered (to their surprise), that all and sundry could just walk in and out of the school with no problem. They rapidly "downgraded" to proper old fashioned locks after that.
Normally smart lights have an override. When you switch the power on to them first time they come on and they function just like any other. This is what happens with my TP-Link bulbs and I've never had any issues if there's been Internet connectivity problems. In fact the app looks for them locally. Just because I can control my lights via Google home or Alexa, it doesn't mean I can't independently control them. Either he's got crap smart bulbs or he doesn't know how to use them (or possibly the switch for that matter).
Given the outage was at lunchtime, why not open the curtains? Also, why remote control the light in a toddler's room? Does the toddler have a smartphone? If it's for the adults, why not use the conventional light switch when you walk in the room? Also^2, why haven't you fitted a smart switch with a local button instead of a stupid 'smart' bulb that requires the normal light switch to be left on?
If you really really want to gaslight your toddler reliably try one of these: https://www.home-assistant.io/blue no internet required as long as you have the correct bulbs/switches.
Local OpenHAB install unaffected by Google outage. Engage smug mode.
That being said, depending on the setup, the light switches at least don't rely on wifi to perform basic functions, they'll still operate as regular light switches even if your whole network dies.
Not sure how you'd build that sort of redundancy into a light bulb, but that's the price of convenience I guess.
TwitterTwat just couldn’t be bothered to go and turn the light on manually, probably thought he’d get a ton of likes and retweet’s if he posted what he did.
Clearly a feed back loop in progress, rewarding stupid behaviour.
I saw a trio of leccy company vans outside the other week from ~6pm, bang on 8 the power went out for ~40 mins. My lightwave & smart plug lights went out too. No end of tweeting would have resolved that. I didn’t even have the option of getting off my bottom to manually turn lights back on!!!!!
s/sides/times/
Indeed. Not even referring to the Revolver fiasco, the Otto lock firmware update fiasco and the hundreds of other similar clod-related fiasci will get through to dent the "must have because shiniez" attitude.
> Otto lock firmware update fiasco
Pardon, what firmware? My Ottolock (Google doesn't seem to find any Otto locks) is entirely hardware. Aren't they all? It's only a fancy café stop lock, I don't expect it to resist much more than penknives or nail clippers!
Ah sorry, mixed up two IdioT lock mishaps. The firmware bricking was by Lockstate, while Otto was developing a finger-scanning door lock but locked its own doors before even delivering a product.
... this finally gave me the kick up the arse I needed to pull my mail back onto my own servers, instead of hoping and praying it didn't "accidentally vanish", or at the very least, losing access to it. Free grandfathered G-suite is great, but it does mean absolute trust in the system holding it.
Hopefully roll-your-own-antispam is a lot better than I remember it being with manual SpamAssassin and DNSBL setups.
Same here, but had to have a few tries at re-authentication before it finally stuck.
In the intervening period I was going to email El Reg to tip them off about the outage, and realised I couldn't easily. (It was too much hassle to find my login credentials to my little-used Outlook account).
The last memorable time I was faced with a similar dilemma was when I got a replacement soldering iron element ... that had to be soldered on.
Google Maps was misbehaving, around 11.40 GMT 'Sorry something went wrong, please try again' and Your phone is offline' upon hitting g the 'Directions' button. Que me staring puzzled at 4G status bar, restarting Maps, toggling mobile data, and turning off WiFi in case it had latched onto a weak signal.
Haven't seen any report of Google Maps failing, but the timing fits, and no reason to think it was immune from Google's outage
I presume, then, you are unfamiliar with the Local Group of Galaxies?
(As an astronomer, I run into this sort of thing frequently. An event ten million years ago is "recent". Something ten light-years away is "nearby".)
I presume, then, you are unfamiliar with the Local Group of Galaxies?
Now that you mention it, but no, it wasn't on my mind when I wrote that.
We'll hear more of them once they're connected to the Galactic Area Network, and Google figures out what to punt to these new advertising targets.
Services that depended on Google Authentication were unusable. Youtube worked fine if you were not logged in. This affected Docs, GMail, Youtube, Photos, etc. Also Nest. Any open documents or data was okay, but new docs were not possible, and opening existing docs after the outage would fail or show old versions.
So like we lost all access to our 'enterprise grade application suite' which we are sooo super-excited to be working with. Truly, today, for once, I did not have my head in email. Thanks to Google.
I wonder if Google use Solarwinds, and were remediating *that* issue
anon because criticism of the decision to migrate to g-suite results in no longer being on the payroll the following week.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
