back to article Cleanup on aisle C: Tesco app back online after attack led to shopping app outages

The UK's largest retailer, supermarket titan Tesco, has restored its online operations after an attack left its customers unable to order, amend, or cancel deliveries for two days. A Tesco statement acknowledges disruption to the giant's grocery website and app, claiming "an attempt was made to interfere with our systems, …

  1. Anonymous Coward
    Anonymous Coward

    Tesco said "there's no reason to believe customer data is or was at risk."

    Shame I'm an atheist.

  2. IGotOut Silver badge
    WTF?

    Is this the Dail Mail?

    Posting a random tweet from a random nobody who doesn't have a clue.

    Still he has 5 followers, so I guess he's classed as an "influencer"

    1. Ken G

      Re: Is this the Dail Mail?

      Yeah, I read El Reg for the comments from Anonymous Cowards who have the inside scoop. Anyway Mr BaldTony doesn't say who his employer is or which sector, to give context. I've worked for companies which would have escalated to hell after 2 minutes and others who wouldn't notice for 2 weeks.

    2. Spaceharrier

      Re: Is this the Dail Mail?

      Surely you've heard of BaldTony, the leading authority on incident response and disaster recovery?

      1. Ken G
        Trollface

        Re: Is this the Dail Mail?

        Not to be confused with the character of the same name on The Sopranos?

      2. Andy The Hat Silver badge

        Re: Is this the Dail Mail?

        I believe baldtony is related to Facebook Margaret who is apparently leading authority on global pandemics, virus transmission vectors, nail varnish and whether Greggs is currently open in Wakefield.

    3. Brewster's Angle Grinder Silver badge

      Re: Is this the Dail Mail?

      It does seem unlikely that Tony's firm has IT handling 1.27 million weekly orders across 4000 stores.

      Nor has he twigged that while his IT might brief managers and fellow employees, they wouldn't necessarily give customers the skinny. (Particularly if that meant revealing trade secrets to competitors or embarrassing management.)

    4. Anonymous Coward Silver badge
      Facepalm

      Re: Is this the Dail Mail?

      Especially one who thinks that backups and DR are adequate defences against a DDoS attack

  3. Anonymous Coward
    Anonymous Coward

    Very poor customer relations

    Getting around the fact that Tesco did not seemingly have any Business Continuity plans in place, let alone isolated [safe] backups and Disaster Recovery procedures, they also seem to have forgotten about their customers. No public statements worth a damn, nothing. Just a black hole for nigh on 2 days. If I were CEO, then heads would 100% be on the block in both IT Services and in Marketing and Publicity. It is now 3 days later, yes the website is back, but a lot of data isn't. Most of the special offers that existed on Friday have gone. A customers lists of favourite and frequently ordered items is missing. I suspect people that jumped ship to ASDA et al will seriously consider whether it is worth going back. This is going to cost Tesco Millions.

    1. MJI Silver badge

      Re: Very poor customer relations

      Have you used Asda online shopping?

      There is a good reason Tesco is popular at it.

      Iceland is not bad though.

      1. JDPower666

        Re: Very poor customer relations

        I've used both and never noticed much difference between them. The one difference over the Covid pandemic was Asda had delivery slots, Tesco rarely did.

    2. G40

      Re: Very poor customer relations

      Very fair summary. Am failing to see any reason for the excess of downvotes.

    3. chris street

      Re: Very poor customer relations

      "the website is back," - and you think they didn't have BC in place?

      It seems they did and they worked. Although the PR droids apparantly seemed to be totally absent telling people what was going on, they appeared to recover a total disaster ina weekend. Thats not that bad TBH

  4. mark4155

    And how much ransom did they pay?... Answers on a postcard to the usual address. Toodle Pip!

    1. hammarbtyp Silver badge

      Maybe they were give Tesco clubcard points

  5. djack

    The downtime was inevitable

    All of those saying that this shows a massive failure in Tesco's BC plans are almost certainly wrong.

    This wasn't a random hardware failure - someone (seemingly successfully) compromised a server. Unless their DR system is totally separate, running a completely different software stack, developed in isolation from the main site, then simply switching over to DR is not an option ..all that will happen is that the DR site will then be compromised too.

    The correct way to do things is to take the financial hit - turn it all off. Investigate so you are confident you know what vulnerabilities were exploited, then secure the DR system and turn that on.

    Doing that over a weekend is not bad going IMO.

    1. localzuk

      Re: The downtime was inevitable

      Hell, even simply restoring from a backup, if that's what they were relying on, might take that long. Restoring data isn't instant, that's for sure.

    2. Anonymous Coward
      Anonymous Coward

      Re: The downtime was inevitable

      This is the comment that has cut to the root of it. A compromise that comes from the inside, with an active-active BC set up, leaves you with nowhere 'quick' to go. Very good response by Tesco.

  6. elsergiovolador Silver badge

    Deliveries

    How do people get these apps to work? Last time I tried none of the supermarkets had any delivery slots available for the entire month.

    1. Ken G
      Trollface

      Re: Deliveries

      That's a supply chain issue, possibly related to increased home delivery as a result of Covid or drivers shortages which are a) not happening b) happening worse somewhere else c) definitely not a result of Brexit d) a necessary part of levelling up to a higher wage economy.

      1. katrinab Silver badge
        Meh

        Re: Deliveries

        I'm pretty sure you can drive a Tesco home delivery van with a regular car licence. The lorries that deliver stuff to the distribution centres and stores, definitely you need an HGV licence, and there is a shortage of those people.

        1. IGotOut Silver badge

          Re: Deliveries

          Depends when you got your license. Also you need to take an additional "license" if doing it as part of your job. I. E if you do the the odd run for your employer it's OK, but if your job is a van driver, you need the additional course.

    2. Anonymous Coward
      Anonymous Coward

      Re: Deliveries

      Since the end of lockdown I have had no problem booking "flexible 4 hour" slots with Tesco every 12 days. Usually looking up to nearly a month ahead there are plenty of free slots - and I don't subscribe to their "privileged" booking service.

      The only variation recently is that there appear to be no such slots offered after lunchtime on 5 November. I do fret that my Xmas delivery slot would fall on the 23 December - so that might mean living out of my store cupboard/freezer for a week or so over Xmas.

  7. Lunatic Looking For Asylum
    Flame

    Non story - the press tried to big up to another fuel shortage..

    R4 had disgruntled customers complaining - what a bunch of whiney whingers we have in this country - "Tesco .. can't deliver for me this week / change my order - my kids will starve" - F-OFF go and walk to the shop and push a f*ing trolley round...

    1. Anonymous Coward
      Anonymous Coward

      Re: Non story - the press tried to big up to another fuel shortage..

      "But my anxiety ..."

    2. chris street

      Re: Non story - the press tried to big up to another fuel shortage..

      Which is fine unless you are still having to shield, have mobility issues, have no car, or live a long way from the stores. All of which will stop you from just pushing round a trolley.

      1. IGotOut Silver badge

        Re: Non story - the press tried to big up to another fuel shortage..

        Then there is Occado, Asda, Sainsburys, Iceland, Amazon, deliveroo, that "farm" one advertised on the telly and probably a dozen others.

        1. Anonymous Coward
          Anonymous Coward

          Re: Non story - the press tried to big up to another fuel shortage..

          "Then there is Occado, Asda, Sainsburys, Iceland, Amazon, deliveroo, that "farm" one advertised on the telly and probably a dozen others."

          There is also Waitrose

        2. Anonymous Coward
          Anonymous Coward

          Re: Non story - the press tried to big up to another fuel shortage..

          Try getting a delivery slot when 250000 others are also looking for one.

        3. chris street

          Re: Non story - the press tried to big up to another fuel shortage..

          Yes - I can tell you've not tried getting a slot when you want/need it only 2 days out - I found it impossible and I get to use the reserved slots that the regular people dont as well.

      2. localzuk

        Re: Non story - the press tried to big up to another fuel shortage..

        Shielding is officially over according to the multiple copies of the same letter the govt have sent me.

        1. Pookietoo

          Re: Non story - the press tried to big up to another fuel shortage..

          Officially yeah, but there are still plenty of people at higher risk who aren't prepared to gamble with their lives.

          1. Anonymous Coward
            Anonymous Coward

            Re: Non story - the press tried to big up to another fuel shortage..

            A neighbour was sceptical of my prediction of the effects of the first Covid waves - when I continued my shielding. Now he asks me for my prediction after the government gives a new "all clear".

        2. chris street

          Re: Non story - the press tried to big up to another fuel shortage..

          What that means is "we now have space for you in intensive care" It doesnt mean the risks of Covid are any lesser than they used to be, vaccines not withstanding.

  8. Howard Sway Silver badge

    "an attempt was made to interfere with our systems"

    Well, attempts are made to interfere with systems all the time, it's rather a well known hazard of "being online".

    When they're successful enough to bring down your systems, it's no longer an attempt, your systems have been breached. Much better to admit it, than compound the problem by trying to obfuscate with language like this, and therefore making yourself sound somewhat dodgy.

    1. Mike 125

      Re: "an attempt was made to interfere with our systems"

      I interfered with their system once- swiped a bunch of bananas across this weird lasery beepy scanny thing they had instore. And their bastard system charged me 99p!

      Not going there again.

      1. elsergiovolador Silver badge

        Re: "an attempt was made to interfere with our systems"

        Were these bananas bendy by any chance?

    2. This post has been deleted by its author

  9. Anonymous Coward
    Anonymous Coward

    Who said they were 'hacked' ?

    Lot of assumptions here. Has anyone confirmed they were actually 'hacked' ? (as in, servers access, data access which should not have been) or is it simply they were subject to some sort denial of service attack.

    Lot of badly worded reporting going on in the mainstream media in an attempt to generate a bigger story than there might actually be.

    On the plus side it's good to see how we'd recover a 2 server 10 user business in the event of an attack. All those twitter sysadmins love this sort of thing ...

    1. Pookietoo

      Re: Who said they were 'hacked' ?

      It seems likely that they detected some sort of interference and hit the big red button while they identified and cleaned up the affected areas. Maybe caught a ransomware attack before it had caused irreparable damage (or paid the ransom and secured the systems).

  10. Plest Bronze badge
    Mushroom

    Bald Tony - What a twat!

    Really Tony, you'd sack all your IT staff on a whim just because the whole shebang is down?

    To simply pin all the blame on every IT staff member, well I don't think I want to work your company BaldTony 'cos you sound like a twat! The worse kind of manager I have tried my whole career to avoid working for, the screaming, no brains moronic manager who shouts the loudest to try to use fear to motivate rather than managing a bloody hard situation with people trying their best. If you did that with me I'd walk out there and then thanks mate, as you obviously don't respect your IT team or wish to help them cope and learn.

    1. Yet Another Anonymous coward Silver badge

      Re: Bald Tony - What a twat!

      Or he's a Russian / n Korean / Belgian bot that generates outrage at anything. Like the daily mail

      1. cyberdemon Silver badge
        Trollface

        Re: Bald Tony - What a twat!

        I'd never heard of Belgian Bots.. Do they write for the Daily Mail?

        1. Yet Another Anonymous coward Silver badge

          Re: Bald Tony - What a twat!

          Belgium is the biggest state actor threat to the UK - that's why it was targeted by MI5 hacking operations.

          1. Andy The Hat Silver badge

            Re: Bald Tony - What a twat!

            keep on the lookout for shady characters dipping chips in mayonnaise ...

            1. The Basis of everything is...
              Pint

              Re: Bald Tony - What a twat!

              Chips with mayo is a really good reason to hate Belgium.

              But they do have some most excellent beer, sossies and jenever so on balance I can let them off.

    2. Anonymous Coward
      Anonymous Coward

      Re: Bald Tony - What a twat!

      Yeah - "slaphead needs a slap" was my take as well

  11. mark4155

    +++NEWSFLASH++++NEWSFLASH+++NEWSFLASH++++

    From our Tunbridge Wells News Desk.

    Reporter. Mr. V Meldrew.

    No Embargo:

    Reports from the BBC news website have confirmed the serious gravity of the great "Tescogate" scandal.

    News has reached us that "Rebecca" (surname withheld), housewife, 32 from Yorkshire had ordered on the Tesco grocery site her full weekly shop just minutes before the website crash.

    Rebecca said "I am distraught, there will be no food in the house, hubby will come back from the pits (mine) wanting his tea on't table and there'll be nowt to give the lad. There'll be bleeding hell to pay.

    As for the bairns well I hope I can give them some liquid nourishment. You know what I mean, but were' a respectable family and don't talk about bodily functions in this neck of the woods."

    Thankfully there was one saving grace from this sorry saga, Rebecca got a delivery from Tesco of 120 cans of Pepsi Maxi, it is not sure whether she had ordered the fizzy drinks. Facebook "Sharon37" suggested she go on t'internet and order from Mr. Issa who owns a local corner shop called "ASDA" in Leeds. So all in all a greet ending.

    [PS There is actually some truth in this article, Toodle Pip!]

  12. Long John Silver
    Pirate

    A reasonable expectation?

    My family on weekly basis uses the Tesco home delivery service. We were not affected by this glitch. Had we been, I would willingly set aside grievance had Tesco adequate means of informing customers of the problem and for updating on progress in resolving it.

    I don't expect Tesco or any other similarly reputable online business to be totally free from human error, from technological mishap, or from attack by criminals. What I do demand is immediate openness when problems occur, and, so far as practicable, righting serious wrongs (especially financial) to particular customers in timely manner. On such matters reputation rests: Tesco knows that. Its management must learn from each mishap and take steps to improve resilience.

  13. Anonymous Coward
    Anonymous Coward

    They appear to have lost some user data. My "previous orders" list ok for transfer to a new order. However - the "My Favourites" and "Usuals" tabs are both showing empty lists - instead of the fairly long history of my purchases.

    The search facility functioned as normal today to find things that I order less often for my next orders. Sunday night's delivery arrived as normal - even though obviously it couldn't be fine-tuned on Saturday.

    This incident is a reminder to me - not to leave a "1 banana" place-holder order waiting too long for some sensible additions. I wonder if Tesco have waived the £4 "small basket" surcharge for people caught out by the failure?

    1. katrinab Silver badge
      Meh

      Apparently you were able to refuse delivery and get a full refund.

  14. Ken Moorhouse Silver badge

    See the deadly nightshade grow

    A good way to describe the hack.

  15. Raymond Stone

    Seems to have a large data loss on the site, my order history apart from the last order is totally gone and all saved favourites are also gone, so whatever has happened I doubt the full story is admitted to yet

    1. Ken Moorhouse Silver badge

      Re: so whatever has happened I doubt the full story is admitted to yet

      Let us know when a trolley-load of Spam arrives on your doorstep.

  16. Anonymous Coward
    Anonymous Coward

    Config issue?

    "Data loss" and "search" references make me think of a massive ElasticSearch cluster wobble; not a simple restore.

  17. Cav

    Since when is it IT's responsibility to brief the public?

  18. MJI Silver badge

    Tesco C&C

    All running OK, picked one up today

  19. Anonymous Coward
    Anonymous Coward

    Rumour has it

    That the issue was caused by some EC2 instances being deleted. No confirmation yet whether it was actually a bad actor, or a slip of a finger. If it was a bad actor, it was through a compromised account which had privileges for mass deletions.

    This also lends credibility to the rush to get services back online, while historical data is still missing, or offers from a few days prior being missing, as it's all down to which backups have been restored so far for both the site and customer data

    AC for obvious reasons. While not directly involved with Tesco, we're close enough to hear some of the story coming out from behind enemy lines.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021