back to article Major telco outage leaves millions of Australians disconnected

Australia's second-largest telco, Singapore-owned Optus, experienced an outage beginning Wednesday around 4am Sydney and Melbourne time that left millions nationwide without phone or internet access – either mobile or terrestrial. The outage also affected transport as Melbourne trains were temporarily inoperable and network …

  1. Sorry that handle is already taken. Silver badge

    Who, Me?

    I cannot wait for the Who, Me? episode about this one.

    1. Rikki Tikki Bronze badge

      Re: Who, Me?

      You beat me to it ...

      Though, at the time it certainly wasn't funny. I was out in the garden this morning when two police cars and 4 coppers arrived, because the social worker couldn't contact my elderly neighbour on the phone (and the 4 large police couldn't get a response from knocking). They got through in the end , and she was OK - but I'm sure similar stories have been happening all over Aus today.

      1. Sorry that handle is already taken. Silver badge

        Re: Who, Me?

        Even 000 wasn't working from Optus landlines according to some reports which is... yeah.

        What staggers me is the sheer amount of critical public infrastructure that apparently doesn't have a fallback mode and simply fails when it loses comms.

        1. Jellied Eel Silver badge

          Re: Who, Me?

          What staggers me is the sheer amount of critical public infrastructure that apparently doesn't have a fallback mode and simply fails when it loses comms.

          Remember, the Internet routes around problems. Except when the routers are sulking. I suspect the problem will end up being a BFR for everything design, ie over relying on routers for both IP and MPLS services. That can mean an over dependence on the router for routing, and 'transmission' services like keeping pseudo-Ethernet services running. So a bad day on the router, and everything collapses. Much safer (and often cheaper) to let routers route, and switches switch. Especially when switches can usually switch far faster than an IGP or EGP like BGP can notice.

  2. David Newall

    If you were visiting from overseas you would just switch to a different network, but if you had an Optus sim you were off the air. Optus. Yeh. :/

    1. Dog Eatdog

      Given how essential these services are now, perhaps the mobile networks should agree a protocol to switch on national roaming if any network goes down.

      Because they know it could be their network next (and in fact it happened to Vodafone in the past).

      1. Anonymous Coward Silver badge
        Alien

        So you have to scale your network to cope with at least twice as many subscribers as you have??

        Yeah, can't see that happening.

        1. Jason Hindle

          “ So you have to scale your network to cope with at least twice as many subscribers as you have??”

          Traffic shaping. Minimal viable service for emergency national roomers. Emphasis on reliable access to emergency services. Etc.

  3. Anonymous Coward
    Anonymous Coward

    No cash?

    Surely you jest.

    Cash is still king with me and mine.

    And Optus? No!

    1. hitmouse

      Re: No cash?

      Optus has perpetual dead-spots in areas within 5km of downtown Sydney CBD that get no attention.

      Reporting them to Optus (with signal-strength analytics) is about as effective as writing it on my forehead in crayon.

  4. Mike 137 Silver badge

    "left millions nationwide without phone or internet access – either mobile or terrestrial"

    Witness the danger of convergence. The 'digital revolution' has replaced several entirely independent comms systems, with its associated resilience, with everything depending on a single infrastructure, so everything dies when this breaks. That's not really progress from the human perspective -- indeed it presents a real danger (even to life and limb).

    1. Dog Eatdog

      Re: "left millions nationwide without phone or internet access – either mobile or terrestrial"

      It also warns customers to avoid bundling deals.

      My internet stayed up, because only my mobile is with Optus.

    2. Anonymous Coward
      Anonymous Coward

      Re: "left millions nationwide without phone or internet access – either mobile or terrestrial"

      I hope you aren't suggesting they should have put people over profit. The world stopped doing that a long time ago.

  5. lglethal Silver badge
    Go

    I Feel for the small shops...

    If I had read this 2 months ago, I probably would have made some snarky comment about, Millenials not carrying cash, and how I always keep cash on me in case you come to a place that doesnt accept cards or an emergency like this happens.

    BUT I actually just got back from a month in Aus, and wow, I have never been to a place where cash is used LESS than in Australia now! I always had cash on me, but a LOT of places wouldnt even take cash, even for small purchases. Everything was digital and over Eftpos. Probably because finding an ATM was also a struggle in a lot of places (or with massive fees piled on top!). Even my 80 year old mother, didnt carry any cash anymore.

    So when the network goes down like this, a lot of the small shops would be screwed! First, they cant process payments, and second, their customers dont have cash on them to make the payments in another way. This will have hurt a lot of small businesses. Will they get any sort of compensation? Not f%&kin likely...

    1. Sorry that handle is already taken. Silver badge

      Re: I Feel for the small shops...

      We were already enthusiastic adopters of contactless payments pre-pandemic, but that was basically the final nail in cash's coffin.

    2. Lee D Silver badge

      Re: I Feel for the small shops...

      Almost like if you're running a business-critical system like payment processing that you need an independent backup that's not reliant on a single telco.

      No matter the practicalities of that, small business is no different - if it's going to cost you, say, thousands of dollars to go a day without payment processing, then you should at least be spending a thousand dollars to try to mitigate that in most circumstances.

      They could buy Starlink (or other satellite broadband), they could use another cellular provider, they could link with other local shops and get another / better line, etc.

      I cannot imagine - as a self-employed person - running any kind of physical retail shop without at least a Square or iZettle reader under the counter for backup, and some other way of getting online.

      I kind of get it in the middle of nowhere out in the sticks, but that's not where the vast, vast majority of Australians live or Australian shops are.

      Honestly, if I had my entire income stream reliant on a little box that swipes cards, I'd have several backups of all the parts necessary if that were to fail.

      1. Sorry that handle is already taken. Silver badge

        Re: I Feel for the small shops...

        Square went down here a couple of months back. A restaurant had me pay for my lunch by direct deposit.

        It was only when I got home that I noticed I'd still paid them for the Square transaction fee...

        1. very angry man

          Re: I Feel for the small shops...

          Single point of failure NBN!

      2. very angry man

        Re: I Feel for the small shops...

        Just wait to!

    3. Gene Cash Silver badge

      Re: I Feel for the small shops...

      > a LOT of places wouldn't even take cash

      Well then as I see it, it's their own damn fault, and I shed not a tear for them. Taking more than one form of payment is just plain business sense.

      It's like all the websites that used to just take Paypal only. Nope, I don't care how unique and great your product is, I'm not touching Paypal with someone else's 3-metre pole.

      We've had a couple network outages here, and some of the restaurants even hauled out the old slide receipt embossing machines... the ones where you put the card over carbon paper, slide the wiper and it embosses the card number into the receipt.

      When it comes down to your money and your livelihood, you need to be resilient and not a stubborn asshole.

      1. Dagg Silver badge
        Mushroom

        Re: I Feel for the small shops...

        Fuck em, if they don't take cash I don't use them.

        One thing that REALLY pisses me off is the bastards charge a card fee between 1.5% and up to 6% again fuck em.

      2. Richard 12 Silver badge

        Re: I Feel for the small shops...

        Cash is getting expensive to accept because the banks are closing all their branches.

        Small businesses have nowhere to deposit the day's takings or to get cash float for next week.

        A big supermarket can afford to pay for a daily visit by an armoured vehicle, but a small business cannot.

        So accepting cash often means having to keep a huge amount of cash on-hand simply to operate - making them a target for burglary and theft, generally uninsureable even before considering the risk to staff safety.

        That's why they don't take it.

        1. Dagg Silver badge

          Re: I Feel for the small shops...

          Cash is getting expensive to accept because the banks are closing all their branches.

          Then why the hell do they rip you off with card fees?????

          And the ones that really piss me off are those that want me to go online to order from a menu and then enter all my details or worst download an app, No way.

  6. Anonymous Anti-ANC South African Coward

    Now throw CDBC into the mix.

    The fun's just starting.

  7. John Miles

    Not quite to same scale

    But there was a school in UK shut because their internet was down and it took everything needed to operate safely, including fire alarms, out

    BBC Link

    1. Lee D Silver badge

      Re: Not quite to same scale

      As an IT manager for schools - that's a nonsense.

      There's no way that an Internet failure should take out fire alarms - they can't be compliant.

      Also - no MIS access is feasible, but emergency contact details should be printed out and/or otherwise held offline and on-site. You could phone the MIS provider and ask them to send the output of a single emergency report (which all MIS have) to an email address, and then access it on a phone, in extremis.

      Stopping a school running for either of the above is a failure of their basic processes, whether they use third-parties or not.

      What are they going to do in a real fire that occurs in the middle of the school day and burns through the networking cabinets? Are they going to shrug and say "Oh, well, we can't possibly know how to contact parents or see if we're missing any children now". They'd be shut down by the DoE and the fire service if nobody else.

      And no - repeat NO - fire alarm should have any network dependency, let alone Internet dependency. You keep that stuff entirely separate for a reason.

      Something tells me a fire inspector is going to be paying them a visit "real soon now".

  8. Phil Kingston

    Given how well the negotiations went to allow hassle-free roaming in remote areas a couple of years back I can't see them making more headway to revisit that with the added inclusion of metro.

    What does need to happen is for whichever regulator is appropriate to be given the teeth to go and make it happen

  9. Anonymous Coward Silver badge
    Facepalm

    Switch?

    > "government was considering switching critical services to other companies."

    How about learning from it rather than repeating the same problem? Don't switch services, add redundancy from other providers.

  10. PhilipN Silver badge

    99.99% uptime and Short Memory Syndrome

    Beggars belief that emergency calling - which has been with us now how long - 100 years? had no backup in place (few posts above).

    Which traditional system of communications worked? Genghiz Khan had Pony Express type messengers to transit the whole of Eurasia. The English system of hilltop beacons worked pretty well. Otherwise we’re back to jungle drums.

    Welcome to the future and the bigger problem is after the expletives we shrug and go “oh well”.

    1. PhilipN Silver badge

      Re: 99.99% uptime and Short Memory Syndrome

      Apologies for the P.S. but this is interesting :

      https://en.wikipedia.org/wiki/Emergency_telephone_number

      London 1937 first direct dial emergency number.

      But whatever the system nothing beats talking to another human.

  11. Tron Silver badge

    Who carries cash anymore?

    I do.

    Tech is fragile. Much less resilient than physical alternatives. Yet nobody seems to have any form of Plan B.

    In the UK a school had to shut because of a 'system failure'. It's pathetic.

    https://www.bbc.co.uk/news/uk-england-cambridgeshire-67314252

    BT are getting rid of UK landlines that work in power cuts, just as climate change is causing more of them, just so they can save cash, their chums in government supporting them.

    Every service and institution we have seems to be run by greedy, incompetent morons.

  12. Anonymous Coward
    Anonymous Coward

    How did their engineers contact each other?

    They all probably had Optus mobile and Optus NBN at home.

    I'm tipping they use a cloud hosted collaboration and/or email service.

    I wonder if they still carry old skool pagers - even if they do I'd also bet they rely on an internet gateway to send pages.

    Can you imagine the reaction of the techs after they rolled out yet another screwed up manual change or buggy firmware.

    "Oh crap, I better call my boss...... d'oh!"

    The first 3 hours were probably waiting for people to come in to the office....

  13. jeremya

    BGP failure is no the only option

    I think people are locked on the BGP because that's what caused the Facebook outage.

    Another possibility is a private root authority certificate expiring. Suddenly all parts of the network that rely on PKI would instantly fail and this would cascade into automatic provisioning failure.

  14. Anonymous Coward
    Anonymous Coward

    When I first heard of this my immediate thought was the Israelis had attempted to shut down the Garza Strip telecoms and got the wrong carrier or the CCP was carrying out a UAT.

    1. Anonymous Coward
      Anonymous Coward

      I love the down votes, it means I mind my point! Ha Ha

  15. Bluebottle

    Normally it wouldn't have been too much of an issue for me, as internet is with another provider, but a couple of hours into the outage internet went down too - a local power outage that was fixed 9 hours later. So no mobile phone; no VOIP as internet's down. Coffee shop (run by a mad French ex-gendarme) was taking cash as usual and had their bank account details shown for direct credits so that was OK.

    Reminds me of when I worked for an ambulance service, and we stipulated that all infrastructure must be in-house. It all crashed when the Health core router died even though none of our traffic was supposed to go through it, so paper and pen for emergency 000 calls.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like