We have some sad news about Facebook. It has returned to the internet after six-hour mega outage Facebook has struggled back online today, though at the time of writing glitches are still very much a part of The Social Network™ experience. WhatsApp and Facebook became available to users at around 2210 UTC on October 4 after falling off the internet some six or so hours prior. Instagram and Facebook Messenger should be not …
During the early covid spread there was a global lockdown, for a short time the pollution haze cleared in many cities and the world breathed again.
I’d like to think the same thing happened for people’s souls for the brief time the world was rid of the sewer that is Faecebook and it’s associated slime channels.
"They are down? I hadn't noticed, the last time I tried to log in was November 2006."
Must have been interesting for all those sites that can't be bothered to set up their own logins and use "login with Faceboook" :-)
ISTR there was talk a some while ago about UK.gov using Facebook for authenticated logins. Dodged a bullet there!
https://blog.cloudflare.com/october-2021-facebook-outage/
Some people behind private DNS servers may have not been as impacted during parts of the outage as their resolvers cached the old info which was technically still valid.
My private DNS does the opposite, ensuring Zuckerburg's folley is unreachable year round, but not everyone has that luxury.
This post has been deleted by its author
And caused a share price drop that reportedly cost the boss $6 billion.
I thought I'd seen plenty of IT false economies in my time, but obviously they pale into insignificance compared to this. (as in, they should have tested it properly - not suggesting that scripting is bad!)
The share price drop was probably more linked to the whistleblower revealing just how much they don't care about their users whilst pretending they do.
Note: users, not customers. Their customers are the ones who are paying for the "sponsored posts," who more often than not are the ones that FB aren't doing enough about when it comes to preventing misinformation and hate speech being propagated on their platform.
"And caused a share price drop that reportedly cost the boss $6 billion."
Yes, and he "only" has 55% of the voting stock. So the other 45% of voting shareholders all lost another $5.5 billion or so between them. Not to mention any non-voting stock holders.
Sure, saving an admin eight hours of work per week seems trivial, but if you're automating a task like BGP updates, that should mean that you're removing eight hours worth of opportunity to foul up the network.
Of course, you do need to be sure that your automation process is more robust than the manual process for that to be true.
Looking forward to seeing the related "Who, Me?" entry.
The "blame" doesn't belong to whoever made the typo or mistake. The blame belongs to whoever designed the infrastructure for one of the world's biggest internet companies to not be partitioned and resilient.
The issue isn't what went wrong? it is how come it took down more than a limited geographic area and managed to damage both external and internal systems access?. Haven't they heard of management networks?
"Haven't they heard of management networks?"
Yes, but to them those are the special networks with the filtering, tracking, and analytics turned off.
Everyone knows you need them, so the C-level and the reset of the high ranks can do their off-shore banking, bet on the outcome of random events, influence "random" events, browse entertainment, securely communicate with others, etc.
On the serious side, it does seem odd they didn't have at least one cellular/cable/POTS Dial-in/etc connection.
"The issue isn't what went wrong? it is how come it took down more than a limited geographic area and managed to damage both external and internal systems access?. Haven't they heard of management networks?"
Well, according to Facebook, the proverbial "some people" were affected :-)
Why is that bad ?
It's a route-updating protocol. If it could only update routes some of the time it wouldn't be fit for purpose.
Do you want the utter crap that browsers exhibit, where complaints of faults are met by such admissions of failure as 'have you flushed your cache ?' ?
Protocols should have well-defined outcomes, not unknowable black holes of semi-reasoning.
This has become something of an issue for people that don't have social media accounts. It's even worse when council or Government services assume everybody uses those platforms.
"Contact us on Twitter" or Discord or Facebook or.. how about your website? You could even - and I know this is cutting edge technology here - provide a phone number.
Otherwise I'll write and spend the price of a Mars bar on a stamp and you get to deal with the costs of managing print mail.
Possibly overloaded. The news articles on this outage mention that other services are being way overloaded. I guess people need their meme fixes and photos of granny's lunch.
There was on Yahoo! News... https://www.yahoo.com/news/facebook-whatsapp-instagram-outage-down-reactions-twitter-220806155.html
Not just because they need their fix of memes. A huge number of small businesses depend on Facebook - if you look up their URL it is a link to a Facebook page. If they take orders they run through Facebook. Support? Through Facebook or Twitter. Internal employee communication? Private Facebook group.
Probably a lot of them were left scrambling for a way to talk to each other when they didn't have everyone's email addresses, trying to find out what was going on because they couldn't take any customer orders, etc.
Granted something that could just as easily if not more easily happen if their had their own web site, e-commerce site, support email, etc. but hopefully they will become more cautious of trusting Facebook. At least if you sign up with Amazon or Microsoft's cloud for your services you are their customer. Your small business is not Facebook's customer.
Any business, club, charity, etc. that has nothing but a Facebook presence gets blackholed by me. Some members of the village hall committee I was chair of until recently wanted to go Facebook only - I was delighted that several other members very vocally slapped that down before I could!
Any business, club, charity, etc. that has nothing but a Facebook presence gets blackholed by me. Some members of the village hall committee I was chair of until recently wanted to go Facebook only - I was delighted that several other members very vocally slapped that down before I could!
A bit harsh. I quite agree with the sentiment but it's entirely understandable why people go down that route. We're a techie crowd here. Most people are not.
It's entirely understandable why volunteers running a community group or club who are already busy with running the group, keeping the accounts, keeping up to date with the latest insurance/liability information; POR; child protection; equality policy; etc shy away from learning a new thing (which invariably costs money) when they already have a Farcebook account and can sling up a page or group in a couple of minutes (not that they often understand the difference and then wonder why they struggle to attract new members to their private group or why their conversations on a public page are... public).
I've similarly had this conversation. People suggested focussing on social. I pointed out that being target sports (specifically Olympic rifle shooting) we were only one (probably American) atrocity away from someone in California deciding they didn't like guns and nuking all the shooting and target-sport related groups and pages. In that light they all saw the value in having our own site (which is accompanied by a FB page).
It's not really sufficient to belittle and demean people for using Facebook as a social outreach/marketing tool when they're trying to run (say) a community cafe and their core competency is not in web. It's probably for the likes of us to come up with better solutions for them to easily host their own space.
"trying to find out what was going on because they couldn't take any customer orders, etc."
Almost makes me want to learn to play the violin.
I was sent a mailshot by some magazine company. Some of the offerings sounded good, but their contact was on Facebook and their method of contact was Messenger.
Mailshot -> recyclage.
My opinion of a company using Facebook for their services is lower than my opinion of a company with a Wanadoo/Yahoo/GMail email address.
"My opinion of a company using Facebook for their services is lower than my opinion of a company with a Wanadoo/Yahoo/GMail email address."
I saw a van the other day and their email address on the back was $name@aol.com
Apparently that's still a thing. Or they need to hire a signwriter.
I get the people saying they wouldn't patronize a business like this, but realize that you probably represent less than 1% of the public in that stance.
Most people aren't bothered at all by a business using Facebook for its online presence, though with what the whistleblower has revealed about Facebook (and especially if her coming forward encourages other Facebook employees or former employees to come forward with more damaging material) that may change.
Yes, remember when .biz and other TLDs came online? Everyone techie all said it'd never catch on and no one would use it because only spammers would bother. Then the non-techie world realised they could get myname.biz or whatever and flocked to it and other TLDs. Now, creating new TLDs is big money, and we're all saying the same thing while the non-techies are all over it.
Back I say, go back tide!
Yes I was one of those who thought the creation was new TLDs was pointless. Wish I had bought up some of the obvious candidates in TLDs like .biz, .tv, etc. Wouldn't be sex.com money but probably enough to retire comfortably on if I'd registered a few dozen good candidates.
Yep - any web based access for Outlook ("Outlook Live" or other flavours) is currently down, or at least degraded, here in the Antipodes. Apparently due to a recent configuration change, and they're throwing extra resources at it before everyone in Europe wakes up. It's been like this for about 6 hours. There's some collateral damage to on-prem Exchange too (I'm assuming for those that route via o365).
To be fair, this is not really the fault of the automated change system not being tested properly - though that is probably one contributing factor.
It's really a failure of the change remediation not being tested properly.
If you want to move quickly, and accept that failure is a possibility - a luxury not afforded those running nuclear power stations - then you really do need to make sure you have a very effective roll-back solution, that is bulletproof.
That it took them six hours, and a site-visit, to roll back the faulty configuration change, establishes that it was not properly designed and tested.
The moral of the story is that, if you're modifying BGP automatically, you need, first, to design the safety-net, by writing, and testing, code that will reset it all to its last known working state -- reliably, every time.
To fail-fast, you must be able to reset-fast.
While I agree with you that the WHATIF process needed to be run, since they run everything through Facebook, there was no out of band recovery process for when things go wrong. I'm still speculating that they somehow added an illegal character into a change and the system doing in the BGP modification received some form of empty string that was empty that was then pushed.
Some things just can't be tested because it is impossible to simulate the production environment, so you just have to continually improve your failsafe mechanisms, to mitigate that.
If that was their failure, they most certainly could have tested it, but it would have required a couple of virtual cisco/juniper routers, in addition their own platform hardware, to test it properly. If you can make a bad change and cause that much damage, spending $100k on test hardware is a bargain.
'Malicious' doesn't cover things that result in a benefit for mankind :)
I tend to use a 'restart in x mins' command when doing things that will pull the rug out from under you if they go wrong, If that's not normal practise with remote routing changes at FB then this outage was inevitable.
"How do you know that this wasn't a malicious attack?"
Hume's Problem of Induction. You cant know anything for sure, so you go with the balance of probabilities.
Access to BGP tables is something very few people need to have and would likely be extremely hard to hack from outside. As supported by the fact it had to be fixed from inside.
So it was most likely a fat finger or equivalent.
Whether that finger was paid to be fat? Or was consciously fat?
We can argue for the nmext ten years and not know the answer...
“ The moral of the story is that, if you're modifying BGP automatically, you need, first, to design the safety-net, by writing, and testing, code that will reset it all to its last known working state -- reliably, every time.
To fail-fast, you must be able to reset-fast.”
Cisco
Reload in 10
Juniper
Commit confirmed
Critical infrastructure should always have out of band management!!
In a previous life, our HA product would always crap itself if the 2 MySQL db’s got out of sync for what ever reason. Replaying the log files always required some sacrifice to some deities. I always preferred to bin the errant db and resync from primary, not sure why that wasn’t the first solution in the published support manual.
BGP or border getaway protocol is a very old method of advertising routes to large networks.
Networks are arranged in units called autonomous systems, each AS as they are known will peer with other AS and exchange information
Within an AS they can be any number of routes and the routing tables are constantly updating
A bgp route looks like a list of numbers each number being it’s own as.
The problem comes if you are advertising a route ie 1.1.1.1 and you stop advertising it then after a few minutes that route will be marked as down by your peers, they will then send a routing update saying this route isn’t available to their peers and it will spread rapidly.
This is what the Facebook system did as it stopped advertising then the peers picked up the messages and relayed them around. It didn’t matter that the devices were still there as the instructions on how to get to them had been removed.
Reinstating that takes time to get to the routers and log in and restart the peering process. If someone had done a manual change it is quick to revert back with automation you need to know what has been changed and how to fix it.
BGP also lacks proper authentication processes as when it was designed most people administering it knew each other (the internet was a lot smaller then) so it was easy to call Bob and ask what he had just done as something isn’t right now
True, but it doesn’t help when you stop advertising the route all it does it ensure that the correct peer is being sent the update.
Even with the passwords set I can still go into the routing process and issue a “no network x.x.x.x” at which point it will drop the advertisement in the next update.
border getaway protocol [SIC] - but I knew what you meant. ;)
Thank you. ------>
These three words were missing from pretty much every news story I read about this today.
(And yes, I had actually gone and looked it up on my own eventually when I made it to a real computer, but come on, El Reg, this isn't rocket science.)
This post has been deleted by its author
... How about a "I could of told you so".
That's why you have all your tools and documentation on a couple of laptops, that don't require online access.
Signed every techie over 40 years old.
My experience. Internal DNS went nuts and pretty took out the entire business. Luckily us telecoms guys had the ability jump on our kit, get going and let the server kids fix their fuck up.
Any half decent door access system should be able to function without the network being in operation.
The systems I have come across all have local copies of the cards permitted database stored on the controller so if this does happen you aren’t locked out.
Mind you I had to tell someone once that the controlling pc for the door entry system needed to be kept in a key locked room not a swipe card controlled one.
But it appears that the DR plans never took into account losing the entire network which is an oversight really. Hindsight is wonderful, hindsight and a time machine would be perfect
Everyone knows that picture of someone cutting a branch off a tree while sitting on it. On the wrong side of the cut, of course.
Looking at the BGP visualisation posted in the other article it looks very much like they ran an automated branch-cutting chainsaw over the entire tree leaving just a bare, smooth trunk, and requiring some network jockeys to first find climbing irons to get at the core gear halfway up.
It probably wouldn't help if their support is outsourced and offsite.
You probably need someone onsite physically, and knows what they are doing.
And also to not shit themselves because the ENTIRE management team is asking for updates on progress every microsecond!
Many moons ago, in this parish, the original "could of" offender came back at me with some haughty supercilious nonsense about it being "a colloquialism".
A later reply summed it up perfectly: "I do so love it when the comments thread on a Reg article turns into a grammatical pissing contest"
:D
It did no such thing. It just said "configuration changes".
To paraphrase:-
"These aren't accidents, they're throwing themselves into the outage! Throwing themselves into the outage gladly to escape all this hideousness."
Seeing as the entire point of the internet is to protect it from single points of failure, up to and including destruction of entire cities, the fact that FB runs not only all their services but also all their internal management tools through a system vulnerable in this way is pretty special.
Only 5% loss in share value? They were lucky.
"Sorry for the disruption today – I know how much you rely on our services to stay connected with the people you care about."
No. I don't and I never will. All of the people I care about have their numbers stored in my phone and we communicate by different means to Farcebork.
You can still f**k off though.
I read a worrying statistic a while back claiming 50% of people use their Facebook feed as their primary source of news. If this is the case how, if at all, did those people find out what was happening? They couldn’t even ask on Messenger or WhatsApp and you know very well that’s what most of them would be using as their main message service.
When I read that those services were offline it reminded me of a quote from Never Mind The Buzzcocks where Mark once said, “we’ve lost many musicians this year…but sadly that list is nowhere near long enough”.
"If only those had included the doors to the admin center where the system now had to be accessed directly because the networking was borked."
I've not seen it mentioned here but have elsewhere: apparently the access card readers didn't work for the security cage in the (shared?) Santa Clara data centre containing the relevant servers and they had to use an angle grinder to open the cage to gain access.
I've not seen it mentioned here but have elsewhere: apparently the access card readers didn't work for the security cage in the (shared?) Santa Clara data centre containing the relevant servers and they had to use an angle grinder to open the cage to gain access.
On Facebook, perhaps?
ElReg, re Facebook :
" .. engineers needed to get physical access to the routers to fix and restart them, and a crack team was sent into Facebook's Santa Clara, California, data center to do that"
IMDB, re 'Resident Evil' :
"The complex computer shuts down the base to prevent infection. The parent corporation sends in an elite military unit, where they meet Alice, who is suffering from amnesia due to exposure to nerve gas."
Hmmmm.
I presume you've heard of the Царь-бо́мба, but (un)fortunately only one has been built. It would definitely take out all of Santa Clara, Mountain View as well as Cupertino if you'd aim at some point in the middle between those places and detonate it at about 5km up. Even a 20 megaton bomb would do, But that's going for full physical destruction; even smaller bombs would do sufficient damage as well as making the area somewhat unhealthy. In addition there will have been a bit of EMP playing havoc with things electronic. Like computers and their storage.
It also stressed there is "no evidence that user data was compromised," well, anymore than it usually is on Facebook.
If you don't want it to be public, don't post it on FB, even privately. They change their user security polices and settings more often than most people change their socks.
said my wife. Tablet, no Facebooky.
Quick check revealed BBC.CO.Uk was fine *as was everything else but Facebook*
Me: "Internet fine, all Facebook AWOL"
(did check my pihole hadnt gone AWOL though - has a habit of this every few months...)
Wife couldn't FB any friends or message them. I refuse FB on any of my PCs or phone so all my comms were (email and SMS) were OK.
Wish FB had permanently imploded.
Isn't it interesting how this happened just as a whistleblower came forward on Facebook policies.
Anyone still remember that? Anyone? Beuller? Anyone?
I do wonder how much has been "accidentally deleted" during the downtime.
Is anyone else waiting for this event to appear on El Reg's "Who Me?" thread?
The outage comes at a terrible time for Facebook...
1) There's a good time to drop your entire business on the floor with an almighty crash? Who knew?
2) Presumably nobody on the inside can leak stuff while the network is titsup.com, so if there is a good time, this might be it..
Facebook knows in the 4hrs or so after a major whistleblower story, 10s of millions of people will drop Facebook like a hot potato and never look back.
Then the story becomes less urgent and the risk recedes. So they calculated that a full-on service outage would lose less users than leaving it running.
A deliberate calculated act. It'd be interesting if they re-run this story if facebook etc will have further "outages"
personally I'd run the story twice a day every day for a week to see how angry Zuckerbeg gets.
"Few young people: half-dead to give a start. Dead through spite, he will cause the others to shine, And in an exalted place some great evils to occur: Sad concepts will come to harm each one, Temporal dignified, the Mass to succeed. Fathers and mothers dead of infinite sorrows, Women in mourning, the pestilent she-monster: The Great One to be no more, all the world to end."
Or maybe he meant google or Windows 11 ... might even be the start of a zombie apocalypse ...
You are oh so very close to the truth...
On a Windows 11 laptop, one uses Google to get to Facebook (because it's somehow easier to type Google and then search for Facebook than to remember a 12 letter URL).
A few months of Facebook activity, and one is in danger of qualifying as one of the aforementioned zombies.
The apocalypse happens when more people depend upon the services of a single corporate, than don't (I find it worrying how many people equated Facebook's failure as "the internet isn't working").
Given that it's been years and numerous stories about profiling people, pushing just their right buttons, screwing with democracy, and giving a platform to the terminally stupid to amass followers like sheep caught in a web of confirmation bias, I wonder if the apocalypse is already on its way?
Is good though. Remember, it was never owned by Facebook until they bought it. I have been using it since it was a paid yearly thing. Saved me £100s in txt messages.
Facebook though, although I hate it others find it useful. I see the news moaning how big a reach they have with services that require an account being unable to use, but that is the fault of those sites like plenty of news sites that only allow you to login with Facebook instead of having multiple options for logins.
"Facebook, Instagram, WhatsApp and Messenger are coming back online now. Sorry for the disruption today – I know how much you rely on our services to stay connected with the people you care about."
Facebook, Instagram, WhatsApp and Messenger are coming back online now. Sorry for the disruption today – I know how much you missed the targeted ads you care about.
FTFY
Founder Mark Zuckerberg chimed in: "Facebook, Instagram, WhatsApp and Messenger are coming back online now. Sorry for the disruption today – I know how much you rely on our services to stay connected with the people you care about."
Yes, he really does know, exactly how much each and every one of you relies on FaceBook's services to stay connected. And I bet it isn't in one humungous Excel spreadsheet either.
Just like every socially rejected kid out there, I started a blog. It was going fine, I created a page for promoting my post.
Then those morons marked my links as spam. When I requested that spam tag be removed, they disabled my account.
Yeah, I know it better now. It was a foolish idea to use facebook in the first place, but alas, you have to get burned once in a while to learn the lessons.
is everyone assuming it was "accidental", and not a planned, but unannounced, major outage for FB to install a new backend tracking system? A system that's so pervasive and far-reaching it makes the old one look like a champion of privacy, but it required downtime to install and migrate the data to the new system.
Sorry, my tinfoil hat is wrinkled. Must make a new one.
This post has been deleted by its author
"We're starting to get a TON of negative publicity. How about we "accidentally" go down for five or six hours and let them see how much they REALLY NEED us, and CAN'T DO WITHOUT US?"
"GREAT idea, boss!"
"Maurice, you've got four hours to "engineer" this "accident"..."
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
