back to article Square blames last week's outage on DNS screw-up

Square says the widespread outage that hit its payment terminals last week was caused by a DNS failure and not a cyberattack nor an intrusion. During that 14-hour downtime, businesses across the US, UK, and beyond that relied on Square's point-of-sale systems were unable to process customers' credit and debit cards, hitting …

  1. Androgynous Cow Herd

    Offline mode?

    "this mode allows a terminal to queue up transactions for processing later while backend systems are down or can't otherwise be reached. "

    I cannot imagine any way THAT would be exploited....

    Oh, wait...yes I can.

    1. chivo243 Silver badge
      Facepalm

      Re: Offline mode?

      I once supported* POS devices that did exactly that, when connection to the mothership was lost, it began buffering, but there was a limit as I recall. I'm so glad those days are behind me.

      *Went as far as unplugging the offending USB peripheral and plugging it back in again. Rebooting the underlying Windows, or reading the error code over the phone to the helpdesk.

    2. Hans Neeson-Bumpsadese Silver badge
      Pirate

      Re: Offline mode?

      I'm reminded of a friend and a trick he tried to pull back in the early-ish days of Sky Box Office movies on demand.

      When you picked your movie to watch, the box of tricks sitting under the TV would try to request a purchase from Sky HQ (the Sky box had to be plugged into the phone line- we're talking pre-broadband). This had a fallback behaviour whereby if it couldn't phone home it would let you watch the move anyway, and try to register the sale later (better to delay the request than to p*** off the customer).

      My friend found out about this loophole and just unplugged the Sky box from the phone line. It was like this for aaaaages, allowing lots of free movie watching.

      No practical upper limit apparently.

      Then one day his wife announced "I was tidying up and I noticed that the Sky box had got unplugged form the phone, so I plugged it back in"

      Seems like the buffer on the box was very big, because the next bill was somewhat eye-watering.

    3. Greg 38

      Re: Offline mode?

      This affected my business last week and I was thankful it didn't happen during a very busy time. I was able to get the POS tablets to work in offline mode for square after turning off the wireless and increasing the offline sales maximum amount. Of course, this risks a card being declined later once the transactions get processed, leaving us out our merch and the customer gone. This is not something our daily staff would know how to do if I hadn't been on-site. Something similar happened in 2022 also that prevented us from making realtime sales. This definitely has me looking into alternative POS systems.

  2. chivo243 Silver badge

    It's always DNS

    Sounds like the issue is a fat finger... causing the problem, and then another fat finger compounds the problem, and finally a total melt down, too many fat fingers in the soup?

    1. MatthewSt Silver badge

      Re: It's always DNS

      Always lower your TTL first. Means fat finger mistakes don't take as long to fix

  3. Hazmoid

    It is always DNS!

    been there done that, got the blistered ears from users yelling that the Internet is broken.

  4. Ball boy Silver badge
    Facepalm

    Translation

    Square also outlined how it hopes to avoid this sort of meltdown again:

    It claimed it has made changes to its DNS and firewall servers to "protect against the issue we saw," and has taken other defensive steps.

    In English: We changed the admin password from p4ssw0rd to something a bit harder to guess and told the PFY to leave the fuck alone next time

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like