back to article We have some sad news about Facebook. It has returned to the internet after six-hour mega outage

Facebook has struggled back online today, though at the time of writing glitches are still very much a part of The Social Network™ experience. WhatsApp and Facebook became available to users at around 2210 UTC on October 4 after falling off the internet some six or so hours prior. Instagram and Facebook Messenger should be not …

  1. lowwall

    So they switched it off and then switched it back on again. Did Daddy Pig lead the crack team of engineers?

    1. HildyJ Silver badge
      Holmes

      OMZ?

      It was a simple fix and they knew how to fix it from the beginning by, essentially, switching it off and on again.

      Is it a coincidence that it happened just after the 60 Minutes interview with the whistle-blower. All those revelations got lost in the news cycle.

      1. bombastic bob Silver badge
        Coat

        Re: OMZ?

        OMZ = "Oh My Zuck"??

        1. HildyJ Silver badge
          Angel

          Re: OMZ?

          Indeed. I release it to the public domain for commenters to use as they see fit.

  2. aregross
    Mushroom

    OMG!

    Facebook, Instagram, WhatsApp and Messenger are Down!

    You said that just like I cared!

    1. ghp

      Re: OMG!

      Heard they're back. Proves there's no G.

      1. Adrian 4 Silver badge

        Re: OMG!

        Why do we have to LET them back ?

        The reason the internet used to run on trust is that only trustworthy people were on it.

      2. adam 40 Silver badge

        Re: OM!

        OM...... OMMMMM......

        yes, if you meditate on it, there IS no G!

        1. Dave559 Silver badge

          Re: OM!

          I saw what you did there, adam 40. That's a very enlightened pun.

    2. werdsmith Silver badge

      Re: OMG!

      During the early covid spread there was a global lockdown, for a short time the pollution haze cleared in many cities and the world breathed again.

      I’d like to think the same thing happened for people’s souls for the brief time the world was rid of the sewer that is Faecebook and it’s associated slime channels.

    3. Anonymous Coward
      Anonymous Coward

      Re: OMG!

      I hate to be the one that says this but commenting you don't care means you care.

      1. werdsmith Silver badge

        Re: OMG!

        I hate to be the one that says this but commenting you don't care means you care.

        That’s a very frequently appearing web discourse banality and it makes no sense.

        People who don’t need chavbook are not affected if it is not available. That’s all it means.

        1. Anonymous Coward
          Anonymous Coward

          Re: OMG!

          Can I introduce you to a new word? Sarcasm. Have my upvote.

          1. werdsmith Silver badge

            Re: OMG!

            Yes, that's the usual get-out claim. I'm already familiar.

    4. TeeCee Gold badge

      Re: OMG!

      <Battery Sergeant-Major Williams>

      Oh dear, how sad, never mind.

      </BSM Williams>

    5. Version 1.0 Silver badge
      Angel

      Re: OMG!

      They are down? I hadn't noticed, the last time I tried to log in was November 2006.

      1. anonymousI

        Re: OMG!

        Serve you right for the self-indulgence of using Sewerbook so often...

      2. John Brown (no body) Silver badge

        Re: OMG!

        "They are down? I hadn't noticed, the last time I tried to log in was November 2006."

        Must have been interesting for all those sites that can't be bothered to set up their own logins and use "login with Faceboook" :-)

        ISTR there was talk a some while ago about UK.gov using Facebook for authenticated logins. Dodged a bullet there!

  3. Anonymous Coward
    Anonymous Coward

    Clouldflare has a decent write-up on the BGP incident

    https://blog.cloudflare.com/october-2021-facebook-outage/

    Some people behind private DNS servers may have not been as impacted during parts of the outage as their resolvers cached the old info which was technically still valid.

    My private DNS does the opposite, ensuring Zuckerburg's folley is unreachable year round, but not everyone has that luxury.

    1. Mr Tumnus

      Re: Clouldflare has a decent write-up on the BGP incident

      This isn't correct. Their BGP routers stopped advertising their address space out. You can cache DNS records pointing to that address space if you like, you still can't get to it, if the internet doesn't know where it is.

      1. bombastic bob Silver badge
        Meh

        Re: Clouldflare has a decent write-up on the BGP incident

        if only the DNS servers are affected by the routing SNAFU, then cached DNS records will still work

      2. This post has been deleted by its author

  4. Ceyarrecks

    Funny thing about zombies,.. they always seem to get back up pining for,... BRAAAIIINNNNSS.

  5. Howard Sway Silver badge

    it saved a single network administrator over eight hours of work each week

    And caused a share price drop that reportedly cost the boss $6 billion.

    I thought I'd seen plenty of IT false economies in my time, but obviously they pale into insignificance compared to this. (as in, they should have tested it properly - not suggesting that scripting is bad!)

    1. Loyal Commenter Silver badge

      Re: it saved a single network administrator over eight hours of work each week

      The share price drop was probably more linked to the whistleblower revealing just how much they don't care about their users whilst pretending they do.

      Note: users, not customers. Their customers are the ones who are paying for the "sponsored posts," who more often than not are the ones that FB aren't doing enough about when it comes to preventing misinformation and hate speech being propagated on their platform.

      1. DougMac

        Re: it saved a single network administrator over eight hours of work each week

        If you don't pay for the product, you _are_ the product.

    2. Someone Else Silver badge

      Re: it saved a single network administrator over eight hours of work each week

      And caused a share price drop that reportedly cost the boss $6 billion.

      You say that as if it was a bad thing.

    3. bazza Silver badge

      Re: it saved a single network administrator over eight hours of work each week

      What do you mean by false economy? I'd heard that their admins get paid way more than the company lost yesterday, so they're still upon the deal.

    4. John Brown (no body) Silver badge

      Re: it saved a single network administrator over eight hours of work each week

      "And caused a share price drop that reportedly cost the boss $6 billion."

      Yes, and he "only" has 55% of the voting stock. So the other 45% of voting shareholders all lost another $5.5 billion or so between them. Not to mention any non-voting stock holders.

  6. Anonymous Coward
    Anonymous Coward

    Automation

    Sure, saving an admin eight hours of work per week seems trivial, but if you're automating a task like BGP updates, that should mean that you're removing eight hours worth of opportunity to foul up the network.

    Of course, you do need to be sure that your automation process is more robust than the manual process for that to be true.

    Looking forward to seeing the related "Who, Me?" entry.

    1. Mark 85 Silver badge

      Re: Automation

      Looking forward to seeing the related "Who, Me?" entry.

      You beat me to this. I suspect that eventually, some poor intern or PFY will get the blame though. The person who did mess it up won't say word until they're long retired. Pity... the "Who Me?" would be classic.

      1. Graham Cobb Silver badge

        Re: Automation

        The "blame" doesn't belong to whoever made the typo or mistake. The blame belongs to whoever designed the infrastructure for one of the world's biggest internet companies to not be partitioned and resilient.

        The issue isn't what went wrong? it is how come it took down more than a limited geographic area and managed to damage both external and internal systems access?. Haven't they heard of management networks?

        1. Zarno Silver badge
          Joke

          Re: Automation

          "Haven't they heard of management networks?"

          Yes, but to them those are the special networks with the filtering, tracking, and analytics turned off.

          Everyone knows you need them, so the C-level and the reset of the high ranks can do their off-shore banking, bet on the outcome of random events, influence "random" events, browse entertainment, securely communicate with others, etc.

          On the serious side, it does seem odd they didn't have at least one cellular/cable/POTS Dial-in/etc connection.

        2. John Brown (no body) Silver badge

          Re: Automation

          "The issue isn't what went wrong? it is how come it took down more than a limited geographic area and managed to damage both external and internal systems access?. Haven't they heard of management networks?"

          Well, according to Facebook, the proverbial "some people" were affected :-)

  7. a_yank_lurker Silver badge

    Unfortunately

    Unfortunately it appears Failbook, et. al. are back up.

  8. msobkow Silver badge

    BGP has apparently rechristend the "Borked Gandalf Protocol", for the way it magically takes entire organizations offline due to some screwups on their part 9 times out of 10 (like forgetting to renew certificates, registrations, or configuration pushes. *LOL* )

    1. Adrian 4 Silver badge

      Why is that bad ?

      It's a route-updating protocol. If it could only update routes some of the time it wouldn't be fit for purpose.

      Do you want the utter crap that browsers exhibit, where complaints of faults are met by such admissions of failure as 'have you flushed your cache ?' ?

      Protocols should have well-defined outcomes, not unknowable black holes of semi-reasoning.

      1. Intractable Potsherd Silver badge

        This was certainly a well-defined outcome.

    2. Anonymous Coward
      Anonymous Coward

      BGP did a good job this time. Pity that it lasted just some hours.

  9. Anonymous Coward
    Facepalm

    LOL

    "But today’s outages may have been extremely serious for those who rely on its services for day-to-day communications"

    Don't rely on it. Simples.

    1. Anonymous Coward
      Anonymous Coward

      Re: LOL

      Indeed; with WhatsApp down, I actually had to resort to ... emailing somebody! The shame :(

      1. Martin Silver badge

        Re: LOL

        I use WhatsApp and Signal (Signal for preference). If one is down, I use the other one.

        And it's interesting that I didn't even notice WhatsApp was down until its return made the news.

        1. Anonymous Coward
          Anonymous Coward

          Re: LOL

          I only ended up on a WhatsApp group after missing out on too many Friday nights at the pub. It says it all that my non-techie wife told me WhatsApp had been down for a few hours, otherwise I'd probably not have noticed, even on a Friday.

          1. John Brown (no body) Silver badge

            Re: LOL

            Same here. I didn't get home till about 9:30pm and had no idea the world was in melt-down! Traffic on the roads seemed a little busier than usual. Probably Facebook users out war-driving for a WiFi signal :-)

    2. Adrian 4 Silver badge

      Re: LOL

      "The Register staff in the United States and Australia have experienced different levels of service since the resumption."

      That's sad. I somehow imagined el reg to have more taste.

      1. heyrick Silver badge

        Re: LOL

        It seems Facebook and Twitter are where one must go in order to interact with companies these days. It's almost as if customer service is an entirely alien concept.

        1. R Soul

          Re: LOL

          You seem to think the morons who use Facebook and Twitters are customers.You are badly mistaken.

        2. Cederic Silver badge

          Re: LOL

          This has become something of an issue for people that don't have social media accounts. It's even worse when council or Government services assume everybody uses those platforms.

          "Contact us on Twitter" or Discord or Facebook or.. how about your website? You could even - and I know this is cutting edge technology here - provide a phone number.

          Otherwise I'll write and spend the price of a Mars bar on a stamp and you get to deal with the costs of managing print mail.

          1. Adelio Silver badge

            Re: LOL

            Have you notices that any phone numbers are getting harder and harder to find. Almost as if they did not want you to contact them!

            1. Shez

              Re: LOL

              Providing humans for people to speak to on the phone cost far too much money

        3. Anonymous Coward
          Anonymous Coward

          Re: LOL

          Or, the garbage disposal that is salesforce.com

  10. Anonymous Coward
    Anonymous Coward

    Facebook is like dog poop

    It exists, but it’s best avoided.

    IMHO

    1. DJV Silver badge

      Re: Facebook is like dog poop

      Take a look at DownDetector to see how the FB SNAFU had repercussions all over the place and not just FB-associated sites.

  11. Uncle Ron

    Something Else

    It appears now, at around 0200 GMT, that Outlook Live is down. Huh?

    1. Mark 85 Silver badge

      Re: Something Else

      Possibly overloaded. The news articles on this outage mention that other services are being way overloaded. I guess people need their meme fixes and photos of granny's lunch.

      There was on Yahoo! News... https://www.yahoo.com/news/facebook-whatsapp-instagram-outage-down-reactions-twitter-220806155.html

      1. DS999 Silver badge

        Re: Something Else

        Not just because they need their fix of memes. A huge number of small businesses depend on Facebook - if you look up their URL it is a link to a Facebook page. If they take orders they run through Facebook. Support? Through Facebook or Twitter. Internal employee communication? Private Facebook group.

        Probably a lot of them were left scrambling for a way to talk to each other when they didn't have everyone's email addresses, trying to find out what was going on because they couldn't take any customer orders, etc.

        Granted something that could just as easily if not more easily happen if their had their own web site, e-commerce site, support email, etc. but hopefully they will become more cautious of trusting Facebook. At least if you sign up with Amazon or Microsoft's cloud for your services you are their customer. Your small business is not Facebook's customer.

        1. Synonymous Howard

          Re: Something Else

          Any company that >only< has a facebook/insta/whatsapp presence does not get my money as my DNS blackholes all requests for those varied domains.

        2. Intractable Potsherd Silver badge

          Re: Something Else

          Any business, club, charity, etc. that has nothing but a Facebook presence gets blackholed by me. Some members of the village hall committee I was chair of until recently wanted to go Facebook only - I was delighted that several other members very vocally slapped that down before I could!

          1. rg287 Silver badge

            Re: Something Else

            Any business, club, charity, etc. that has nothing but a Facebook presence gets blackholed by me. Some members of the village hall committee I was chair of until recently wanted to go Facebook only - I was delighted that several other members very vocally slapped that down before I could!

            A bit harsh. I quite agree with the sentiment but it's entirely understandable why people go down that route. We're a techie crowd here. Most people are not.

            It's entirely understandable why volunteers running a community group or club who are already busy with running the group, keeping the accounts, keeping up to date with the latest insurance/liability information; POR; child protection; equality policy; etc shy away from learning a new thing (which invariably costs money) when they already have a Farcebook account and can sling up a page or group in a couple of minutes (not that they often understand the difference and then wonder why they struggle to attract new members to their private group or why their conversations on a public page are... public).

            I've similarly had this conversation. People suggested focussing on social. I pointed out that being target sports (specifically Olympic rifle shooting) we were only one (probably American) atrocity away from someone in California deciding they didn't like guns and nuking all the shooting and target-sport related groups and pages. In that light they all saw the value in having our own site (which is accompanied by a FB page).

            It's not really sufficient to belittle and demean people for using Facebook as a social outreach/marketing tool when they're trying to run (say) a community cafe and their core competency is not in web. It's probably for the likes of us to come up with better solutions for them to easily host their own space.

            1. John Brown (no body) Silver badge

              Re: Something Else

              "It's probably for the likes of us to come up with better solutions for them to easily host their own space."

              Yeah, whatever did happen to MS Frontpage? :-)

        3. heyrick Silver badge

          Re: Something Else

          "trying to find out what was going on because they couldn't take any customer orders, etc."

          Almost makes me want to learn to play the violin.

          I was sent a mailshot by some magazine company. Some of the offerings sounded good, but their contact was on Facebook and their method of contact was Messenger.

          Mailshot -> recyclage.

          My opinion of a company using Facebook for their services is lower than my opinion of a company with a Wanadoo/Yahoo/GMail email address.

          1. John Brown (no body) Silver badge

            Re: Something Else

            "My opinion of a company using Facebook for their services is lower than my opinion of a company with a Wanadoo/Yahoo/GMail email address."

            I saw a van the other day and their email address on the back was $name@aol.com

            Apparently that's still a thing. Or they need to hire a signwriter.

        4. DS999 Silver badge
          Stop

          Re: Something Else

          I get the people saying they wouldn't patronize a business like this, but realize that you probably represent less than 1% of the public in that stance.

          Most people aren't bothered at all by a business using Facebook for its online presence, though with what the whistleblower has revealed about Facebook (and especially if her coming forward encourages other Facebook employees or former employees to come forward with more damaging material) that may change.

          1. John Brown (no body) Silver badge

            Re: Something Else

            Yes, remember when .biz and other TLDs came online? Everyone techie all said it'd never catch on and no one would use it because only spammers would bother. Then the non-techie world realised they could get myname.biz or whatever and flocked to it and other TLDs. Now, creating new TLDs is big money, and we're all saying the same thing while the non-techies are all over it.

            Back I say, go back tide!

            1. DS999 Silver badge

              Re: Something Else

              Yes I was one of those who thought the creation was new TLDs was pointless. Wish I had bought up some of the obvious candidates in TLDs like .biz, .tv, etc. Wouldn't be sex.com money but probably enough to retire comfortably on if I'd registered a few dozen good candidates.

    2. stratcat

      Re: Something Else

      Yep - any web based access for Outlook ("Outlook Live" or other flavours) is currently down, or at least degraded, here in the Antipodes. Apparently due to a recent configuration change, and they're throwing extra resources at it before everyone in Europe wakes up. It's been like this for about 6 hours. There's some collateral damage to on-prem Exchange too (I'm assuming for those that route via o365).

  12. Fustbariclation
    Coat

    Not a failure of testing - a failure of change enablement

    To be fair, this is not really the fault of the automated change system not being tested properly - though that is probably one contributing factor.

    It's really a failure of the change remediation not being tested properly.

    If you want to move quickly, and accept that failure is a possibility - a luxury not afforded those running nuclear power stations - then you really do need to make sure you have a very effective roll-back solution, that is bulletproof.

    That it took them six hours, and a site-visit, to roll back the faulty configuration change, establishes that it was not properly designed and tested.

    The moral of the story is that, if you're modifying BGP automatically, you need, first, to design the safety-net, by writing, and testing, code that will reset it all to its last known working state -- reliably, every time.

    To fail-fast, you must be able to reset-fast.

    1. Rob F

      Re: Not a failure of testing - a failure of change enablement

      While I agree with you that the WHATIF process needed to be run, since they run everything through Facebook, there was no out of band recovery process for when things go wrong. I'm still speculating that they somehow added an illegal character into a change and the system doing in the BGP modification received some form of empty string that was empty that was then pushed.

      Some things just can't be tested because it is impossible to simulate the production environment, so you just have to continually improve your failsafe mechanisms, to mitigate that.

      1. Anonymous Coward
        Anonymous Coward

        Re: Not a failure of testing - a failure of change enablement

        If that was their failure, they most certainly could have tested it, but it would have required a couple of virtual cisco/juniper routers, in addition their own platform hardware, to test it properly. If you can make a bad change and cause that much damage, spending $100k on test hardware is a bargain.

        1. ICam

          Re: Not a failure of testing - a failure of change enablement

          By my reckoning, based on FB's 2020 USD$85.9b total, they were losing revenue at a rate of USD$163.4k per second or ~USD$54m for the five and a half hour total duration. Ouch.

    2. Symon
      Paris Hilton

      Re: Not a failure of testing - a failure of change enablement

      How do you know that this wasn't a malicious attack?

      1. Wellyboot Silver badge

        Re: Not a failure of testing - a failure of change enablement

        'Malicious' doesn't cover things that result in a benefit for mankind :)

        I tend to use a 'restart in x mins' command when doing things that will pull the rug out from under you if they go wrong, If that's not normal practise with remote routing changes at FB then this outage was inevitable.

      2. itzman
        Linux

        Re: Not a failure of testing - a failure of change enablement

        "How do you know that this wasn't a malicious attack?"

        Hume's Problem of Induction. You cant know anything for sure, so you go with the balance of probabilities.

        Access to BGP tables is something very few people need to have and would likely be extremely hard to hack from outside. As supported by the fact it had to be fixed from inside.

        So it was most likely a fat finger or equivalent.

        Whether that finger was paid to be fat? Or was consciously fat?

        We can argue for the nmext ten years and not know the answer...

        1. Symon

          Re: Not a failure of testing - a failure of change enablement

          Who said anything about a 'hack from outside'. As any fule kno, most of them come from inside.

      3. Anonymous Coward
        Anonymous Coward

        Re: Not a failure of testing - a failure of change enablement

        Anything that takes Facebook offline cannot possibly be malicious.

    3. tip pc Silver badge

      Re: Not a failure of testing - a failure of change enablement

      “ The moral of the story is that, if you're modifying BGP automatically, you need, first, to design the safety-net, by writing, and testing, code that will reset it all to its last known working state -- reliably, every time.

      To fail-fast, you must be able to reset-fast.”

      Cisco

      Reload in 10

      Juniper

      Commit confirmed

      Critical infrastructure should always have out of band management!!

      In a previous life, our HA product would always crap itself if the 2 MySQL db’s got out of sync for what ever reason. Replaying the log files always required some sacrifice to some deities. I always preferred to bin the errant db and resync from primary, not sure why that wasn’t the first solution in the published support manual.

      1. Stoneshop Silver badge
        Holmes

        Critical infrastructure should always have out of band management!!

        And a way to actually access that management network NO MATTER WHAT.

  13. Androgynous Cow Herd

    oops...

    someone moved fast and broke something!

    Still think that's a great motto, Zuck?

  14. AbeSapian
    Pirate

    I love the Register

    I love reading The Register because I always learn something - even if it's something I'll never use.

    On a personal note: I was not even aware of the Facebook outage until I read this article. What does that tell you?

    1. Piro Silver badge

      Re: I love the Register

      That you've sensibly avoided their services.

      I found out several hours later because of a post on a forum. I've never had a Facebook account.

    2. Steve K Silver badge

      Re: I love the Register

      Errrm, that you live in a different Time Zone?

    3. DF118

      Re: I love the Register

      > What does that tell you?

      That you enjoy a humblebrag?

    4. Def Silver badge
      WTF?

      Re: I love the Register

      I love reading The Register because I always learn something - even if it's something I'll never use.

      Although in this case that wasn't true. I am still non the wiser as to what the fuck BGP is.

      1. Mishak

        I am still non the wiser as to what the fuck BGP is.

        Let me try to fix that for you:

        BGP - A protocol designed to make it very easy for a simple typo to rapidly bring large (generally unimportant) portions of the internet to a halt.

        1. Anonymous Coward
          Anonymous Coward

          Re: I am still non the wiser as to what the fuck BGP is.

          That really should be the wikipedia page, definition for BGP.

          1. Bowlers

            Re: I am still non the wiser as to what the fuck BGP is.

            Perhaps amanfrommars1 should be asked to explain it?

      2. itzman
        Coat

        Re: I love the Register

        "I am still none the wiser as to what the fuck BGP is"

        Think of it as stanav for IP packets.

        "At the router, take the third exit"

      3. Giles C Silver badge

        Re: I love the Register

        BGP or border getaway protocol is a very old method of advertising routes to large networks.

        Networks are arranged in units called autonomous systems, each AS as they are known will peer with other AS and exchange information

        Within an AS they can be any number of routes and the routing tables are constantly updating

        A bgp route looks like a list of numbers each number being it’s own as.

        The problem comes if you are advertising a route ie 1.1.1.1 and you stop advertising it then after a few minutes that route will be marked as down by your peers, they will then send a routing update saying this route isn’t available to their peers and it will spread rapidly.

        This is what the Facebook system did as it stopped advertising then the peers picked up the messages and relayed them around. It didn’t matter that the devices were still there as the instructions on how to get to them had been removed.

        Reinstating that takes time to get to the routers and log in and restart the peering process. If someone had done a manual change it is quick to revert back with automation you need to know what has been changed and how to fix it.

        BGP also lacks proper authentication processes as when it was designed most people administering it knew each other (the internet was a lot smaller then) so it was easy to call Bob and ask what he had just done as something isn’t right now

        1. tip pc Silver badge

          Re: I love the Register

          you can set passwords for BGP peering neighbours.

          1. Giles C Silver badge

            Re: I love the Register

            True, but it doesn’t help when you stop advertising the route all it does it ensure that the correct peer is being sent the update.

            Even with the passwords set I can still go into the routing process and issue a “no network x.x.x.x” at which point it will drop the advertisement in the next update.

        2. Def Silver badge
          Pint

          Re: I love the Register

          border getaway protocol [SIC] - but I knew what you meant. ;)

          Thank you. ------>

          These three words were missing from pretty much every news story I read about this today.

          (And yes, I had actually gone and looked it up on my own eventually when I made it to a real computer, but come on, El Reg, this isn't rocket science.)

          1. Giles C Silver badge

            Re: I love the Register

            Dammed autocorrect

            1. W.S.Gosset Silver badge
              Happy

              Re: I love the Register

              Well, BGP got away from 'em on this occasion, so the autocorrect may have acquired both sentience and a sense of humour.

    5. This post has been deleted by its author

  15. Stoneshop Silver badge
    Facepalm

    That message should have read

    "To the huge community of people and businesses around the world who depend on us: STOP DOING THAT YOU STUPID OAFS[0]"

    [0] I believe the unesteemed Marc Z. used a similar, though stronger, expression once referring to his customersaddicts

  16. IGotOut Silver badge

    All their tools were down as well...

    ... How about a "I could of told you so".

    That's why you have all your tools and documentation on a couple of laptops, that don't require online access.

    Signed every techie over 40 years old.

    My experience. Internal DNS went nuts and pretty took out the entire business. Luckily us telecoms guys had the ability jump on our kit, get going and let the server kids fix their fuck up.

    1. Already?

      Re: All their tools were down as well...

      I find it hard to take seriously anyone who uses the phrase “I could of…”

      There is no verb To of. Of has no business existing in that sentence.

      1. Loyal Commenter Silver badge
        Headmaster

        Re: All their tools were down as well...

        ...also failure to understand when you use "an" rather than "a".

        Reading that comment made my brain emit several syntax errors. Consider this post the resulting log file.

    2. I am David Jones
      FAIL

      Re: All their tools were down as well...

      Even that won‘t help you if you’re locked out of the building. As Facebook techs were.

      Too many eggs in one basket?

      1. Adrian 4 Silver badge

        Re: All their tools were down as well...

        Too much confidence in their own ability ?

      2. Giles C Silver badge

        Re: All their tools were down as well...

        Any half decent door access system should be able to function without the network being in operation.

        The systems I have come across all have local copies of the cards permitted database stored on the controller so if this does happen you aren’t locked out.

        Mind you I had to tell someone once that the controlling pc for the door entry system needed to be kept in a key locked room not a swipe card controlled one.

        But it appears that the DR plans never took into account losing the entire network which is an oversight really. Hindsight is wonderful, hindsight and a time machine would be perfect

        1. My-Handle Silver badge

          Re: All their tools were down as well...

          I built a time machine once, but accidentally caused a paradox and removed it and a hefty chunk of the timeline from the continuum.

          In hindsight, building a time machine wasn't the best of ideas.

      3. thondwe

        Re: All their tools were down as well...

        This was beautifully summarized elsewhere - they locked the keys in the car. Actually seems they locked the keys and the phone in the car - so they struggled to get hold of the person with the spare key...

        1. Stoneshop Silver badge
          Happy

          Re: All their tools were down as well...

          Everyone knows that picture of someone cutting a branch off a tree while sitting on it. On the wrong side of the cut, of course.

          Looking at the BGP visualisation posted in the other article it looks very much like they ran an automated branch-cutting chainsaw over the entire tree leaving just a bare, smooth trunk, and requiring some network jockeys to first find climbing irons to get at the core gear halfway up.

          1. adam 40 Silver badge

            Re: All their tools were down as well...

            It probably wouldn't help if their support is outsourced and offsite.

            You probably need someone onsite physically, and knows what they are doing.

            And also to not shit themselves because the ENTIRE management team is asking for updates on progress every microsecond!

    3. The Rope

      Re: All their tools were down as well...

      Could have or could've ffs

    4. Jan 0 Silver badge

      Re: All their tools were down as well...

      Doesn’t every techie over 40, with English as their main language, understand the difference between “of” and “‘ve”?

      If you’re not techie enough to understand the language you’re writing, how much other technical stuff do you misunderstand?

    5. heyrick Silver badge
      Happy

      Re: All their tools were down as well...

      "I could of have told you so"

      FTFY.

    6. gotes

      Re: All their tools were down as well...

      I know several others have made the same comment, but it annoys me enough to add.. Could HAVE, FFS.

      1. DF118

        Re: All their tools were down as well...

        Many moons ago, in this parish, the original "could of" offender came back at me with some haughty supercilious nonsense about it being "a colloquialism".

        A later reply summed it up perfectly: "I do so love it when the comments thread on a Reg article turns into a grammatical pissing contest"

        :D

  17. tip pc Silver badge

    Facebook, Instagram, WhatsApp and Messenger are Down!

    And I never knew!

  18. Detective Emil
    Pint

    _My_ glass is half empty

    I do hope that this does not spur FB to bulldoze through a "fix" to DNS that just happens to serve its interests at the expense of everybody else's (cf. Google's AMP).

    1. Mr Tumnus

      Re: _My_ glass is half empty

      Why would them pushing a misconfigured BGP config change prompt them to rewrite DNS protocols?

  19. Symon
    Devil

    "It also confirmed an accidental configuration change ultimately caused the loss in connectivity"

    It did no such thing. It just said "configuration changes".

    To paraphrase:-

    "These aren't accidents, they're throwing themselves into the outage! Throwing themselves into the outage gladly to escape all this hideousness."

  20. Felonmarmer

    Seeing as the entire point of the internet is to protect it from single points of failure, up to and including destruction of entire cities, the fact that FB runs not only all their services but also all their internal management tools through a system vulnerable in this way is pretty special.

    Only 5% loss in share value? They were lucky.

  21. Potemkine! Silver badge

    Gloria in Excelsis Deo!

    I wonder how many lives were saved thanks to this 6 hours of suppression of fake news, online harassment and hate speeches.

    1. Stoneshop Silver badge
      Devil

      Re: Gloria in Excelsis Deo!

      As well as a reduction in drought conditions worldwide die to the tears being shed by influenzers unable to influenza.

  22. Anonymous Coward
    Anonymous Coward

    crazy

    " I know how much you rely on our services"

    Nope, I don't ! Whatsapp going all facebook cured it for me.

    "How could a company of Facebook’s scale get BGP wrong?"

    Too much automation and politicians, I think.

    1. Anonymous Coward
      Anonymous Coward

      Re: crazy

      "How could a company of Facebook’s scale get BGP wrong?"

      All you do is relegate your engineers to the lowest paid least respected oiks in the system.

      Paid greasemonkeys to 'make it all work'

      Meanwhile you make Important Human And Political Decisions.

      Cf ClimateChange™ The Movie

  23. Aristotles slow and dimwitted horse Silver badge

    Just f**k off you insidious prick...

    "Sorry for the disruption today – I know how much you rely on our services to stay connected with the people you care about."

    No. I don't and I never will. All of the people I care about have their numbers stored in my phone and we communicate by different means to Farcebork.

    You can still f**k off though.

  24. andy 103

    Where did 50% of people find out?

    I read a worrying statistic a while back claiming 50% of people use their Facebook feed as their primary source of news. If this is the case how, if at all, did those people find out what was happening? They couldn’t even ask on Messenger or WhatsApp and you know very well that’s what most of them would be using as their main message service.

    When I read that those services were offline it reminded me of a quote from Never Mind The Buzzcocks where Mark once said, “we’ve lost many musicians this year…but sadly that list is nowhere near long enough”.

    1. katrinab Silver badge
      Unhappy

      Re: Where did 50% of people find out?

      They didn't. They contacted their cellphone providers to complain that the internet was down.

    2. Warm Braw Silver badge

      Re: Where did 50% of people find out?

      I read a worrying statistic a while back

      You really need to share the source with the 3.4 billion people deprived of the most toxic well of worrying alleged statistics for six hours - it might help keep them occupied during the next outage.

    3. Spanners Silver badge
      Boffin

      Re: Where did 50% of people find out?

      I got it from the BBC. Far (and getting further) from perfect but hugely better than getting your news from something controlled by a US corporation!

  25. steelpillow Silver badge
    Childcatcher

    caused door keycards to stop working

    If only those had included the doors to the admin center where the system now had to be accessed directly because the networking was borked.

    One day, this will be the first sign that Teh Singularity hath awoken.

    1. katrinab Silver badge
      Coat

      Re: caused door keycards to stop working

      I believe it did. But nothing that can't be fixed by The Lockpicking Lawyer, or a sledge hammer.

    2. Anonymous Coward
      Anonymous Coward

      Re: caused door keycards to stop working

      "If only those had included the doors to the admin center where the system now had to be accessed directly because the networking was borked."

      I've not seen it mentioned here but have elsewhere: apparently the access card readers didn't work for the security cage in the (shared?) Santa Clara data centre containing the relevant servers and they had to use an angle grinder to open the cage to gain access.

      1. Doctor Evil
        Coat

        Re: caused door keycards to stop working

        I've not seen it mentioned here but have elsewhere: apparently the access card readers didn't work for the security cage in the (shared?) Santa Clara data centre containing the relevant servers and they had to use an angle grinder to open the cage to gain access.

        On Facebook, perhaps?

        1. ICam

          Re: caused door keycards to stop working

          I don't have a URL for the article, but it was mentioned in an NY Times article that an angle grinder had been used, but that was later corrected to say a misstatement had been made.

      2. Ken Moorhouse Silver badge
        IT Angle

        Re: an angle grinder

        See icon--->

  26. mikecoppicegreen

    I'll just leave this here.......

    http://www.dorktower.com/

  27. Shak

    Obvious timing?

    Facebook gets taken out via internal means of a level of plausible deniability the day a whistleblower gets air time.

    Sounds like a shakedown to me.

  28. noisy_typist

    Having finally created shadow profiles for the last few members of an undiscovered Amazonian tribe, Zuck declared "my work here is done" and switched it off.

    1. Anon
      Angel

      Something like Arthur C. Clarke's Nine Billion Names of God?

  29. revenant

    Seems familiar ...

    ElReg, re Facebook :

    " .. engineers needed to get physical access to the routers to fix and restart them, and a crack team was sent into Facebook's Santa Clara, California, data center to do that"

    IMDB, re 'Resident Evil' :

    "The complex computer shuts down the base to prevent infection. The parent corporation sends in an elite military unit, where they meet Alice, who is suffering from amnesia due to exposure to nerve gas."

    Hmmmm.

    1. WolfFan Silver badge

      Re: Seems familiar ...

      My only question: how close is Santa Clara to Mountain View? Could (asking for a friend) one nuke get both?

      1. bombastic bob Silver badge
        Trollface

        Re: Seems familiar ...

        Santa Clara and Mountain View...

        as I recall they're next to one another (checked, Sunnyvale between them)

        Aim for Palo Alto and Santa Clara might be spared...

      2. Stoneshop Silver badge
        Mushroom

        Could (asking for a friend) one nuke get both?

        I presume you've heard of the Царь-бо́мба, but (un)fortunately only one has been built. It would definitely take out all of Santa Clara, Mountain View as well as Cupertino if you'd aim at some point in the middle between those places and detonate it at about 5km up. Even a 20 megaton bomb would do, But that's going for full physical destruction; even smaller bombs would do sufficient damage as well as making the area somewhat unhealthy. In addition there will have been a bit of EMP playing havoc with things electronic. Like computers and their storage.

    2. Ken Moorhouse Silver badge

      Re: Alice, who is suffering from amnesia due to exposure to nerve gas.

      Couldn't they ask Bob instead?

  30. Anonymous Coward
    Anonymous Coward

    Locked out of the Facebook building...

    "If your fingerprint is not on the list, you're not getting in".

    Get in touch with Security...

    Security: "We'd add you to the list, but we can't access the systems to do that, right now".

    1. Stoneshop Silver badge
      Mushroom

      "If your fingerprint is not on the list, you're not getting in".

      "There's no problem that can't be solved by the judicious use of high explosives."

  31. Anonymous Kiwi

    <blockquote>

    Facebook tweeted...

    </blockquote>

    <strikethrough>You'd think they'd post it their Facebook page wouldn't you?</strikethrough>

    Or insta

  32. anthonyhegedus

    If it's not DNS, it's "Low on Cyan toner"

    1. heyrick Silver badge

      PC LOAD LETTER

  33. Loyal Commenter Silver badge

    It also stressed there is "no evidence that user data was compromised," well, anymore than it usually is on Facebook.

    If you don't want it to be public, don't post it on FB, even privately. They change their user security polices and settings more often than most people change their socks.

    1. Doctor Syntax Silver badge

      That bit was true. For six hours it wasn't.

  34. Jean Le PHARMACIEN

    "my internet is down"

    said my wife. Tablet, no Facebooky.

    Quick check revealed BBC.CO.Uk was fine *as was everything else but Facebook*

    Me: "Internet fine, all Facebook AWOL"

    (did check my pihole hadnt gone AWOL though - has a habit of this every few months...)

    Wife couldn't FB any friends or message them. I refuse FB on any of my PCs or phone so all my comms were (email and SMS) were OK.

    Wish FB had permanently imploded.

    1. Yet Another Anonymous coward Silver badge

      Re: "my internet is down"

      We had a short lived panic that corporate IT had discovered Whatsapp and were blocking it.

  35. WonkoTheSane
    Holmes

    Suspicious

    Isn't it interesting how this happened just as a whistleblower came forward on Facebook policies.

    Anyone still remember that? Anyone? Beuller? Anyone?

    I do wonder how much has been "accidentally deleted" during the downtime.

    Is anyone else waiting for this event to appear on El Reg's "Who Me?" thread?

    1. nautica Bronze badge
      Alien

      Re: Suspicious

      I would give you twenty up-votes if I could.

  36. KBeee Silver badge
    Happy

    Facebook was down?

    Oh dear. How sad. Nevermind.

    1. Dave559 Silver badge

      Re: Facebook was down?

      "Oh dear. How sad. Nevermind."

      Here we are now, entertain us? :-P

  37. nijam Silver badge

    > How could a company of Facebook’s scale get BGP wrong?

    Therey've been practising with all the other things they do, I expect.

  38. Coastal cutie
    Mushroom

    To err is human - to really screw things up, you need human designed automation. Icon as that's what a lot of peoples brains seem to have done yesterday

  39. TeeCee Gold badge
    Meh

    Yes, but...

    The outage comes at a terrible time for Facebook...

    1) There's a good time to drop your entire business on the floor with an almighty crash? Who knew?

    2) Presumably nobody on the inside can leak stuff while the network is titsup.com, so if there is a good time, this might be it..

  40. Marty McFly Silver badge
    Megaphone

    I did not notice!

    Proudly stating this is the first I heard about FB going off-line. And glad my life does not revolve around its existence.

    1. Fruit and Nutcase Silver badge
      Joke

      Re: I did not notice!

      @Marty McFly

      - you could make a deal with the Devil* and fire up the DeLorien, go back in time and prevent the BGP configuration from happening!

      *no prizes for guessing who is the incarnation that entity on Earth in relation to this post

  41. xyz123

    Facebook knows in the 4hrs or so after a major whistleblower story, 10s of millions of people will drop Facebook like a hot potato and never look back.

    Then the story becomes less urgent and the risk recedes. So they calculated that a full-on service outage would lose less users than leaving it running.

    A deliberate calculated act. It'd be interesting if they re-run this story if facebook etc will have further "outages"

    personally I'd run the story twice a day every day for a week to see how angry Zuckerbeg gets.

  42. Bartholomew Bronze badge

    nostradamus totally totally predicited this would happen

    "Few young people: half-dead to give a start. Dead through spite, he will cause the others to shine, And in an exalted place some great evils to occur: Sad concepts will come to harm each one, Temporal dignified, the Mass to succeed. Fathers and mothers dead of infinite sorrows, Women in mourning, the pestilent she-monster: The Great One to be no more, all the world to end."

    Or maybe he meant google or Windows 11 ... might even be the start of a zombie apocalypse ...

    1. heyrick Silver badge

      Re: nostradamus totally totally predicited this would happen

      You are oh so very close to the truth...

      On a Windows 11 laptop, one uses Google to get to Facebook (because it's somehow easier to type Google and then search for Facebook than to remember a 12 letter URL).

      A few months of Facebook activity, and one is in danger of qualifying as one of the aforementioned zombies.

      The apocalypse happens when more people depend upon the services of a single corporate, than don't (I find it worrying how many people equated Facebook's failure as "the internet isn't working").

      Given that it's been years and numerous stories about profiling people, pushing just their right buttons, screwing with democracy, and giving a platform to the terminally stupid to amass followers like sheep caught in a web of confirmation bias, I wonder if the apocalypse is already on its way?

      1. Anonymous Coward
        Anonymous Coward

        Re: nostradamus totally totally predicited this would happen

        On a Windows 11 laptop, one uses Google to get to Facebook (because it's somehow easier to type Google and then search for Facebook than to remember a 12 letter URL).

        Does nobody use Favourites or Bookmarks anymore?

  43. steviebuk Silver badge

    WhatsApp

    Is good though. Remember, it was never owned by Facebook until they bought it. I have been using it since it was a paid yearly thing. Saved me £100s in txt messages.

    Facebook though, although I hate it others find it useful. I see the news moaning how big a reach they have with services that require an account being unable to use, but that is the fault of those sites like plenty of news sites that only allow you to login with Facebook instead of having multiple options for logins.

  44. Nightkiller

    "Facebook, Instagram, WhatsApp and Messenger are coming back online now. Sorry for the disruption today – I know how much you rely on our services to stay connected with the people you care about."

    Facebook, Instagram, WhatsApp and Messenger are coming back online now. Sorry for the disruption today – I know how much you missed the targeted ads you care about.

    FTFY

  45. Eclectic Man Silver badge
    Big Brother

    Zuck knows

    Founder Mark Zuckerberg chimed in: "Facebook, Instagram, WhatsApp and Messenger are coming back online now. Sorry for the disruption today – I know how much you rely on our services to stay connected with the people you care about."

    Yes, he really does know, exactly how much each and every one of you relies on FaceBook's services to stay connected. And I bet it isn't in one humungous Excel spreadsheet either.

  46. Anonymous Coward
    Anonymous Coward

    Uh?

    What’s this Facebook thingy-me-jig??

    AC ... natch

  47. Dark Eagle

    I really don't have much sympathy.

    Just like every socially rejected kid out there, I started a blog. It was going fine, I created a page for promoting my post.

    Then those morons marked my links as spam. When I requested that spam tag be removed, they disabled my account.

    Yeah, I know it better now. It was a foolish idea to use facebook in the first place, but alas, you have to get burned once in a while to learn the lessons.

  48. Anonymous Coward
    Anonymous Coward

    Why

    is everyone assuming it was "accidental", and not a planned, but unannounced, major outage for FB to install a new backend tracking system? A system that's so pervasive and far-reaching it makes the old one look like a champion of privacy, but it required downtime to install and migrate the data to the new system.

    Sorry, my tinfoil hat is wrinkled. Must make a new one.

    1. Stoneshop Silver badge
      Holmes

      Hanlon's Razor

      Never attribute to malice that which can be adequately explained by stupidity (or incompetence)

  49. This post has been deleted by its author

  50. Adelio Silver badge

    Never noticed it as I do not use facebook. We were going to watsapp our son yesterday but we decided to move it to today (we did not know about the outage)

    If they all died a horrible deaths I would not notice or care!

  51. nautica Bronze badge
    Happy

    Facebook would NEVER do this.

    "We're starting to get a TON of negative publicity. How about we "accidentally" go down for five or six hours and let them see how much they REALLY NEED us, and CAN'T DO WITHOUT US?"

    "GREAT idea, boss!"

    "Maurice, you've got four hours to "engineer" this "accident"..."

    1. cyberdemon Silver badge
      Pint

      Re: Facebook would NEVER do this.

      "What do you mean they are celebrating and say they hope we never come back?? Get it back online NOW Maurice!"

      "Boss, I can't open the door to the server room.."

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021