back to article UK web host 123-Reg goes TITSUP, customer servers evaporate

UK hosting and domains provider 123-Reg has been struck by a weekend TITSUP (Total Inability To Support Usual Performance) that knocked an unspecified number of VPS (virtual private server) customers offline. The company posted a status message saying that the unspecified issues arose on April 16. 123-reg customer, software …

  1. amolbk

    In related (hoax) news

    http://www.independent.co.uk/life-style/gadgets-and-tech/news/man-accidentally-deletes-his-entire-company-with-one-line-of-bad-code-a6984256.html

    1. Frank Zuiderduin
      IT Angle

      Re: In related (hoax) news

      You beat me to it. Very suspicious...

    2. trollied

      Re: In related (hoax) news

      That was a hoax:

      http://meta.serverfault.com/questions/8696/what-to-do-with-the-rm-rf-hoax-question

      1. Doctor Syntax Silver badge

        Re: In related (hoax) news

        "That was a hoax"

        Are you absolutely sure?

  2. Anonymous Coward
    Anonymous Coward

    Do they outsource their Technical Support ...

    ... from TalkTalk ?

    1. Lith

      Re: Do they outsource their Technical Support ...

      From previous experience of 123reg, they source their customer services from TalkTalk rejects.

      1. fruitoftheloon
        Happy

        @Lith: Re: Do they outsource their Technical Support ...

        Lith,

        On the whole, my experiences of 123-reg over a number of years have been quite acceptable...

        Ymdv.

        Cheers,

        Jay

  3. Pascal Monett Silver badge

    "expresses sympathy for anybody without a proper backup"

    Well that sounds very professional. We screwed up and we're very sorry that you don't have a backup.

    That's not even a Cloud issue, it's a hosting one, but here we go again with the consequences of trusting your data to someone else's infrastructure. Somebody goofed, which happens, and websites were deleted. Okay, human error, statistically unavoidable, but compounded by a total lack of data backup - leaving customers up the creek to provide their own paddle.

    Well it's a choice and it was probably in the T&Cs, but I'll wager there are a bunch of websiteless companies that are, at this point, seriously re-examining their level of acceptable hosting costs and guarantees while looking over other hosting companies' offerings.

    I know I would be.

    1. Mage Silver badge
      Facepalm

      Re: that's not even a Cloud issue, it's a hosting one,

      Cloud is just market speak for certain kinds of remote hosted services,

      1. John Sanders
        Linux

        Re: that's not even a Cloud issue, it's a hosting one,

        Thank you!

        I have been saying for years that "cloud" is just managed hosting with an orchestration panel at best, and a shoddy virtualization platform at it worst.

        Hosting, hosting never changes.

        1. PNGuinn
          Linux

          Hosting, hosting never changes.

          Hosting, hosing never changes ...

          There - FIFY.

          Have an upvote anyway.

      2. grumbley

        Re: that's not even a Cloud issue, it's a hosting one,

        "Cloud Hosting with 123-Reg"

        DOES EXACTLY WHAT IT SAYS!

        We all know that clouds disperse and "VAPORISE"

    2. VinceH

      Re: "expresses sympathy for anybody without a proper backup"

      "Well that sounds very professional. We screwed up and we're very sorry that you don't have a backup."

      INNmaster - who expressed the very sympathy you are criticising - are a 123Reg customer. As such, I don't think they're responsible for making backups of other 123Reg customers' websites.

    3. grumbley

      Re: "expresses sympathy for anybody without a proper backup"

      Absolutely! What a numpty I've been. I was a 123 reg dedicated server customer for a number of years without incident. Then one day they erased my server by mistake thinking it was someone else's. I had no recourse and it was my fault for not having a backup; after all I was just a customer; what rights could I possibly have claim to?

      So I decided to leave, but in the interim I downgraded to a VPS thinking that at least on a VPS there would be a disc image on the cluster so total loss "JUST COULD NOT HAPPEN".

      Well, time went by and I didn't get round to moving everything to a new hosting provider.

      You'll never guess what happened to me at 07:21 am 16/04/2016?

  4. gnufrontier

    The National Enquirer of Technology

    Your reading it.

    1. Steve Davies 3 Silver badge
      Headmaster

      Re: The National Enquirer of Technology

      quote

      Your reading it.

      I know it is monday but...

      You are reading it

      Is a more correct way of saying it. (see ICON)

    2. maffski

      Re: The National Enquirer of Technology

      What is that supposed to mean? My VPS went down at approx. 9am Saturday morning. In 48 hrs there have been 5 status updates. Most providing little information, and what they did provide being so vague as to be useless. For example they currently claim to be restoring VPS images, but won't say how quickly, or what percentage are likely to be restorable, or what criteria are likely affect the chances of your being a recoverable image.

      Things break, it happens, but the communication has been the real failure.

      1. Vince

        Re: The National Enquirer of Technology

        I'm not defending 123-REG (and I'd never defend the customers daft enough to take service with these budget providers for anything vaguely important either)... but out of interest...

        ""won't say how quickly?"

        What are you asking for? The number of KB/MB/GB a second being restored or the number of machines vs total or ???

        "what percentage are likely to be restorable"

        I haven't checked, but assuming backup is part of the agreement anything less than "all of it" means they've been in breach of contract. However, in terms of knowing this, this brings me to a pet niggle.

        When something breaks, the first questions you get are:

        "How long will it take to be fixed"

        "What is wrong?"

        "How long will it take"

        "So when do you think it will be working again"

        "Why can't you fix it sooner"

        ...essentially all silly questions. Normally the focus is on restoring service. Root Cause analysis is not going to happen at this stage, and often you won't know when something first goes wrong, you won't know thus what is wrong, and you won't know how long it will take either.

        That said, the updates 123-REG have given are essentially just vague probably because they're a bit embarrassed, a little bit of PR management and a dash of disinterest.

        What do you expect for "starting from £9.99 a month?

        1. d3vy

          Re: The National Enquirer of Technology

          What do you expect from something starting from 9.99 a month?

          Well, for a start for them NOT to delete EVERYTHING and maybe backups.. Unfortunately they have failed on both counts.

          They don't back up the virtual machines.

          And many of the affected customers choose the budget option for a reason.. They are small companies with small margins keeping expenses down is important.

          1. Vince

            Re: The National Enquirer of Technology

            "Well, for a start for them NOT to delete EVERYTHING and maybe backups.. Unfortunately they have failed on both counts."

            Does that agreement say they WILL back it up (I don't know, I'm not a customer). If it doesn't, then there's no point moaning, that's your problem.

            "They don't back up the virtual machines"

            See above.

            "And many of the affected customers choose the budget option for a reason.. They are small companies with small margins keeping expenses down is important"

            Sure, but then that's part of the calculation you make - there's a place for budget providers, and indeed budget customers, but normally, from experience, budget customers expect premium service, premium resilience and premium support. It's rarely possible to get incredibly low pricing and all of those.

            1. d3vy

              Re: The National Enquirer of Technology

              To be fair they do say that backups are not included... But then they provide backup software which backs up to the VPS itself... Great of you delete the off file.. Not so great if the whole server is gone.

              If it was a physical server this is the equivalent of them in racking it, putting it in a skip and setting it on fire.

              Regardless of whether they provide backups or not, you don't expect that!

              PS. Just spoke to support, if I reimage my server and set it all up again from backups they will overwrite it when they manage to recover the VPS image (I gather this is recovery of deleted data rather than restore from a backup)

              This means that even after I get my e-commerce site back up and transactional again at some point in the (possibly) near future, they will replace it with data from Saturday morning so I'll lose any data generate after that on my restored server...

              1. Skoorb

                Re: The National Enquirer of Technology

                And as anyone who has had the misfortune of being involved with undeleting deleted data knows, you are going to have some bit rot in there. Some percentage of your data will have odd corruptions in it; better hope it's not in a key part of your database or your will end up with a 'restored' service that won't actually run because some file somewhere has a 1 where it should have a 0.

                So even if you do restore from your own backup, they could end up overwriting it with something broken anyway.

                ಠ_ಠ

                More importantly, do they have enough free unused servers to ensure that anyone restoring themselves isn't going to overwrite somebody's deleted data? Any disk that has had data deleted from it needs to be taken offline completely and mounted read-only otherwise things will only get worse.

              2. FlatSpot

                Re: The National Enquirer of Technology

                "To be fair they do say that backups are not included... But then they provide backup software which backs up to the VPS itself... Great of you delete the off file.. Not so great if the whole server is gone."

                My reading of not providing backups is that if I mash up my files from day A to day B then I can't restore back to A but I would expect at the server level some form of snapshot/backup, not a restore to a completely empty box.

                Glad I'm not with them :p

                1. Adrian Midgley 1

                  expects backups not included to mean

                  backups are included.

                  Optimistic, I think.

                  If only on the dictionary front.

        2. Doctor Syntax Silver badge

          Re: The National Enquirer of Technology

          "How long will it take to be fixed"

          To which the answer is "It depends how long I have to spend answering your questions".

          1. d3vy

            Re: The National Enquirer of Technology

            "To which the answer is "It depends how long I have to spend answering your questions"

            Their support department are not restoring the machines...

          2. maffski

            Re: The National Enquirer of Technology

            To which the answer is "It depends how long I have to spend answering your questions".

            No it isn't - the correct answer is I'll publish a one sentence status update every hour or two so I don't keep getting bugged by people asking how it's going.

            This is what management are for, it's not like they can sweep this issue under the carpet, so own it.

            If I leave 123 Reg it won't be because of the failure - all platforms fail eventually. It will be because I wasn't kept informed enough to keep my customers informed.

            Trouble is, I've never come across a cloud/hosting/service provider that did keep people informed.

            1. John Robson Silver badge

              Re: The National Enquirer of Technology

              "Trouble is, I've never come across a cloud/hosting/service provider that did keep people informed."

              My current provider has had a couple of issues over the past few years.

              There was even an IP range change (having to give back small slices in order to get larger slices), there have been data centre level DDOS and a router screwup.

              But I have always had an email from the main technical support within minutes of it kicking off (I was actively online for one of the events, and I hadn't finished bashing out my "what's up" email before theirs hit my inbox).

              Regular updates of what they are trying along the way, then a decent breakdown of the issue after.

              Makes life so much easier if people admit that things go wrong, and deal with their customers as if they could do it as well...

              Who? corgi tech

              I was a fairly early customer and agreed to switch data centres for bonus features (i.e. I pay a lot less than I should) - but it's still pretty good value anyway. I don't use it for anything particularly critical, but I really notice when it isn't there...

  5. x 7

    one of my customers has two sites with 123-reg, both are up an running OK

    1. d3vy

      It's only VPS that are affected.

  6. batfastad

    Bozos

    I put these bozos in the same bracket as Farthosts. I have migrated many customers from both providers over the years. Not so much for reasons of reliability (although both have been shaky at times) but the customer service was always dreadful and lack of technical control.

    I definitely understand choosing cheap providers because of cost - if you won't die over a day or so of downtime then thats a perfectly valid decision to make. But if downtime is going to cost your business serious money, it's probably better to not be using budget mass-market web hosts where you are one account of 10,000 and instead have a managed service, or better still run from multiple providers. Compare the loss due to downtime vs the cost of improving availability.

    One thing I would say is separate out your DNS hosting, domain registration and web hosting. Easier to juggle things around in times of brownout.

  7. Stoneshop Silver badge
    FAIL

    They took the 'script' that was posted on Serverfault

    And went "What does this do?"

    1. David Gosnell

      Re: They took the 'script' that was posted on Serverfault

      Yep, about my deduction. Even wondering what the --no-preserve-root option (rather giving the game away about the hoax SF article) might have been about?

  8. d3vy

    Check your email REG I sent you the whole update email that they sent out when I realised they had deleted all of the VMs and didn't have a backup!

  9. To Mars in Man Bras!
    Trollface

    I Know It's Cruel...

    But I can't help wishing them the worst.

    Many moons ago, I used to teach a web design module at a college. Whenever I got to the lesson on registering domain names and web hosting [with accompanying "who to avoid like the plague" caveats], there would always be a few students who'd proudly announce they'd already registered a domain name and bought web space, in advance of the lesson.

    My heart would sink because, 99% of the time, said domain name would have been registered with 123-Reg, or similar "Get a Domain Name and 25 Years Hosting for Only 99p!!!!!" Mickey Mouse operation and I knew I'd be spending most of my break and end-of-lesson time wrestling with their fucktarded Fisher Price "control panel", trying to configure DNS servers or <shudder!> upload a website via a browser based form.

  10. tin 2

    I know this is not about 123-reg's business practices, nor do I agree with the criticism of their technical support - I have always found them excellent when you can get through (which of course means when there's NOT a big outage), but there's no harm in reminding of this.....

    http://www.theregister.co.uk/2014/04/03/123reg_shrugs_at_complaints_over_sudden_domain_transfer_fees/

    1. d3vy

      While I agree (I've been with them for 5-6 years with no issues) normally their support are great.. In this instance though they seem totally unprepared.

      I'm going to stick with them in the hope that they have learned from this.. I certinally have and will have mirrors of my servers somewhere else too.

      1. Known Hero
        Paris Hilton

        are you a talktalk customer by any chance ?

  11. Chris Priest

    Email from 123-reg.....

    I am writing to you to explain what happened to some VPS services on 16.04.16. This email is to detail what our steps have been. I am committed to open communication with all customers and would like to take this opportunity to explain in detail.

    So what happened to some services? As part of a clean-up process on the 123-reg VPS platform, a script was run at 7am on 16.04.16. This script is run to show us the number of machines active against the master database. An error on the script showed 'zero-records' response from the database for some live VPS. For those customers, this created a 'failure' scenario - showing no VM's and effectively deleting what was on the host. As a result of our team's investigations, we can conclude that the issues faced having resulted in some data loss for some customers. Our teams have been and continue to work to restore. What have we done? We have been working with an extended team of experts and have left no stone unturned. Our teams have been working long into the night to restore as much as we possibly can. We have also invested in external consultants to recover, in the best way possible.

    We have recovery running on the VPS servers and some are restoring to new disks. We have also begun copying recovered VPS images to new hosts and we expect some VPS to be back up and running throughout the night and in to tomorrow.

    1. gerdesj Silver badge

      "An error on the script showed 'zero-records' response from the database for some live VPS. For those customers,"

      This appears to explain that their monitoring system is a bit more macho than the ones I'm used to.

    2. Electric Panda

      So, a bug in their script reported some customers as having no VMs running when they actually did? And the "cleanup" job was to simply nuke the apparently spun-down VMs that weren't actually spun-down but were live?

      Shoddy stuff. VERY dangerous thing to automate IMO, they should have just used the script to generate a report and take it from there.

  12. David Roberts Silver badge
    Coat

    One man and a dog outfit

    Presumably the dog is still keeping the man away from the keyboard?

    1. SolidSquid

      Re: One man and a dog outfit

      Given the circumstances, I think he might have finally gotten to the keyboard and now the dog has to clean up the mess

  13. alain williams Silver badge

    I left them years ago ...

    I used to use them as a DNS tag holder, but departed after yet another screw up. Some companies you learn to avoid.

  14. Anonymous Coward
    Anonymous Coward

    I left them when the started to hold .co.uk domains to ransom. Bad business practice. You might as well go to lowendbox. At least you know what you are getting there.

  15. Chronos
    Coat

    An extended team of experts

    Over-stretched has-been drips under pressure. Yup, sounds about right ;-)

    Sorry, just couldn't resist.

  16. d3vy

    Most recent update : if you have local backups we suggest setting up a new vps and restoring to that.

    Brilliant, I have backups (Many dont because 123s backup solution by default backs up to the VPS) but Im looking at a solid days work to get everything back up...

    1. Blitheringeejit
      FAIL

      "Most recent update : if you have local backups we suggest setting up a new vps *WITH A DIFFERENT PROVIDER* and restoring to that."

      FTFY

  17. Dominion

    So who should I host with?

    Genuine question, but which providers are better for small websites? Fasthosts are rubbish, Heart lost an entire DC a month or so back.... 123-Reg are now proven to be incapable.... Who is actually any better?

    1. Blitheringeejit

      Re: So who should I host with?

      The problem is that no-one will tell you if they have a brilliant hosting company, because they don't want that company to acquire so much new business that they become over-stretched and flaky. I did a decade of trial-and-error (including an early flirtation with 123) before I found my lasting life-partner - and you're not getting their number from me!

    2. Anonymous Coward
      Anonymous Coward

      Re: So who should I host with?

      "Heart lost an entire DC a month or so back"

      Heart Internet has been a part of the same group since 2011. I've no idea how much- if any- of their infrastructure or support they share with 123-reg, though.

    3. Anonymous Coward
      Anonymous Coward

      Re: So who should I host with?

      They're all consumer-level providers - fine for little Jimmy's scout group, but that's about it.

      If you're looking at them, then you probably don't have budget for a proper supplier with a dedicated account manager and SLAs.

      You could always do it yourself - but then you'd need to fork out for a pipe with an SLA ('cos you won't want to use your consumer-grade ADSL) plus a DR site... Which you'd already be doing if you could afford it.

    4. UKShazad

      Re: So who should I host with?

      I've been using Vidahost for 8 years now and never had any issues.

      1. Oliver Burkill

        Re: Haters Gonna Hate, but...

        Thjey were bought by the saem group that owns 123-reg and Heart ant the beginning of the year.

        http://www.host-consult.net/internet-industry/paragon-internet-group-acquired-by-heg/

        1. tin 2

          Re: Haters Gonna Hate, but...

          Crap. You just ruined my year. Been migrating from 123-reg to TSO very slowly for about 3 years now.

          Looks like I'll have to set up my own domain registrar!!!!

      2. To Mars in Man Bras!
        IT Angle

        Re: So who should I host with?

        Domain Reg: Namecheap

        Hosting [VPS]: Linode*

        *I know Linode had their own 'Keystone Kops' period a few months back, and I was first in the queue to stick the boot in, in these comments. But, weighing everything up, I've had 1 day downtime and about a week of crossed fingers, in almost seven years of running a couple of VPS with them. So, in the cold light of day, that's good enough for me.

    5. Anonymous Coward
      Anonymous Coward

      Re: So who should I host with?

      Xen Europe misplaced my VM a few years ago

      1. psychonaut

        Re: So who should I host with?

        i resell for these guys. they host my stuff and about 40 of my business customers too

        shameless affiliate link below, you can go direct if you want, im sure you can work out the url, but Gods honest truth these guys are golden.

        http://billing.host100.co.uk/billing/aff.php?aff=004

        medium sized outfit, superb service, uk based datacentres, uk based support engineers - they all know what they are talking about, they look after the servers. call them up and you are right through to someone who knows what they are talking about, what you are talking about and how to fix the issue.

        1. solex16

          Re: So who should I host with?

          hmm.. host100 have VPS only on Windows

    6. highpriest

      Re: So who should I host with?

      DigitalOcean. They are utterly brilliant. They now have a datacenter in the UK as well (London). The big boys (Azure, AWS and Google) also do good small VPSes but they will be more expensive. Linode is excellent as well but they don't do a 512 MB VPS at $5/mo like DO.

  18. Sgt_Oddball Silver badge

    so in summary..

    Pretty bad but not as bad as hearts balls up...

    And people wonder why I'm paranoid about hosting solutions.

    (for the record I'm about to start on migrating from a server 2008 to 2012 dedicated box. At this point with all the recent vps boxes issues, I still see no reason not to stick to dedicated hardware)

    1. Anonymous Coward
      Anonymous Coward

      Re: so in summary..

      "Pretty bad but not as bad as hearts balls up..."

      Heart has been a part of the same group since 2011.

      Mind you, if you're referring to Heart's power outage that happened a couple of months back- corrupting our MySQL installation in the progress- that was bad, but I'd have said a complete loss of multiple VPSs was worse still.

      1. Dominion

        Re: so in summary..

        The Heart balls up also took out VPSs according to the comments on the Reg story back in February.

  19. lupine

    glad

    i left them completely a year ago thank goodness.

    now use OVH and digital ocean for various VPS and support is better from both.

  20. Dave Bell

    What are we risking?

    I have had a very long period of reliable operation of a domain name registration with email forwarding, over fifteen years.

    This isn't part of this cock-up, for which I am very grateful. My email still works. I have been receiving email forwarded through my domain.

    This is bad enough. If they mess up the Domain Name registration side, they're going to be dead.

    And what happens if they do drop their Doman Name database in the shit? Who will know who has a right to use a name. Tesco, to pick an example, have trademarks registered, and lawyers. It's easy to prove they own it. What proof do I have?

    1. psychonaut

      Re: What are we risking?

      you can have the "a" in "Fuck All"

  21. Anonymous Coward
    Anonymous Coward

    DNS

    And whilst you are looking at SLA's ask them for their SLA on their DNS service. When they suffered a full weekend outage a few years back the response I got was it was free and therefore subject to a best efforts policy. Needless to say I moved every domain elsewhere very quickly and have not touched them since.

  22. vanjaap

    Update email address FAIL

    These guys must host their own email. They want you to email for updates but their email address doesn't work:

    Delivery to the following recipient failed permanently:

    vps-team-123@123-reg.co.uk

    Technical details of permanent failure:

    Google tried to deliver your message, but it was rejected by the server for the recipient domain 123-reg.co.uk by edge01.webfusion.com. [92.60.112.26].

    The error that the other server returned was:

    500 Message rejected

    1. Chris Priest

      Re: Update email address FAIL

      Probably overloaded! I sent one a couple of hours ago and so far it's not been bounced back, but hey, they've probably run a script to process email and deleted it :) (Smiley face as I'm trying to stay positive over this shambles)

      NOTE: Still not had an update on my server tho....

  23. BaudBandit
    Thumb Up

    Shameless plug for VPS's

    I use VPS.net ... fantastic (and inexpensive) VPS's and excellent support ... www.vps.net

  24. Anonymous C0ward

    Out of 4 cheap VPS hosts I've used

    not one has been without billing issues. Sometimes even in my favour! At least I configure them securely enough that one hosting company's admins had trouble getting in to test things when I reported the clock being wrong enough to cause problems (the host box's fault, and seriously who doesn't use NTP).

  25. MattP1821

    In related (hoax) news response

    Sophos have reported this story not naming 123 reg but claim that someone did indeed use rm -d/f in an admin script so have Sophos fallen for a hoax news item or is it actually the case?

  26. philmck

    If an outage like this happened to me (as it has more than once) I would immediately start restoring my backups to a completely new provider, because you don't know if or when the original provider will fully recover, because restoring and DNS propagation take a while, and because better providers almost certainly exist but you have to test them.

    I currently use a VPS at transip.eu in Amsterdam and they have been fairly reliable.

  27. briangreen

    123reg disaster

    Hi guys. I feel sorry for the 123reg victims including myself. Decided to leave them right away, whatever they say. Am now with the company Linux Hosts Ltd. They claim to have the lowest customer account cancellation rate. Many different VPS plans from £10.43. So far so good!

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021