back to article IBM broke its cloud by letting three domain names expire

Back in September, IBM was left red-faced when its global load balancer and reverse DNS services fell over for 21 hours. At the time, IBM blamed the outage on a third-party domain name registrar that was transferring some domains to another registrar. The sending registrar, IBM said, accidentally put the domains in a “hold …

  1. redpawn Silver badge

    _BM

    It wasn't I who let it expire. There is no I in _BM.

    1. smudge

      Re: _BM

      Have you actually worked for them? That is *so* accurate!

      I once worked on a major programme in which IBM were the prime contractors. The IBM managers spent all their time passing the buck, and ensuring that they weren't responsible or to blame for anything.

      I was part of the assurance function, and I quickly learned how to silence a meeting room full of senior IBM managers. Just ask "Who's going to pick this one up?".

      1. Anonymous Coward
        Anonymous Coward

        Re: _BM

        When I joined IBM a wise manager said "at IBM Success has many friends and fathers and failure is like being lost at sea.

    2. GruntyMcPugh Silver badge

      Re: _BM

      "But there is a U in F*ck Up" : -) (as we used to say)

      1. Anonymous Coward
        Anonymous Coward

        Re: _BM

        "But there is a U in F*ck Up" : -) (as we used to say)

        In fact, as with most Fuck Ups there are two Us involved.

    3. CrazyOldCatMan Silver badge

      Re: _BM

      There is no I in _BM.

      Of course there is. Why, only the other day, my dyslexic doctor was asking me about my IBM..

  2. Yet Another Anonymous coward Silver badge

    Not their fault

    The intern that knew how to do the web stuff went back to school and doesn't seem to have a left a fax number to contact

    1. Rohime

      Re: Not their fault

      Fax? you mean Pager?

  3. This post has been deleted by its author

    1. Ken Moorhouse Silver badge

      Re: domains may be registered up to 10 years

      Maybe that's what they did, ten years' ago. But understandably they didn't have a 2017 diary on hand at the time to put the reminder in to renew it.

      1. frank ly

        Re: domains may be registered up to 10 years

        I've had an idea for some kind of computer based diary/reminder application that would give some kind of notification for a pre-set event that was about to happen. Do you think I should patent this?

        1. Korev Silver badge
          Coat

          Re: domains may be registered up to 10 years

          I had a similar idea, maybe we can Exchange some ideas and swap Notes.

        2. BinkyTheMagicPaperclip

          Re: domains may be registered up to 10 years

          That computer based diary/reminder thing is a great idea, but we need a backup plan.

          How about if you fail to do that, the hosting company sends you multiple letters, especially close to the expiration date? To make things easier a system could be designed that doesn't need the postal service, and sends the reminder over the network.

          I'm thinking of calling it e-letters, for electronic letters. Do you think it would work?

          1. Pascal Monett Silver badge

            Re: "How about [..] the hosting company sends you multiple letters"

            I have a domain name registered. I pay for it by 10-year blocks. When the time comes to renew, I get a mail from my registrar warning me about it.

            I don't need a calendar app, I just need to read my mail.

            I fail to see how this can be improved.

            On the other hand, I can very well see how a major company can fuck up on this kind of thing. The guy responsible for registering the domain has left, the registration email account has been discontinued after a reorganization, and the swarm of managers in between have never wondered or even thought of checking how the domain names were managed.

            Thus, the domains lapsed, the functionality was broken, and the managers scurried around like headless chickens until somebody with a clue phoned the registrar and got things sorted out.

            1. Pompous Git Silver badge

              Re: "How about [..] the hosting company sends you multiple letters"

              "I don't need a calendar app, I just need to read my mail.

              I fail to see how this can be improved."

              Exactly. I am in the process of thinking about letting one of my domains expire. I have been receiving reminders for nearly six months now! Get sick of constant nagging reminders? Just fucking renew!

              "The guy responsible for registering the domain has left, the registration email account has been discontinued after a reorganization, and the swarm of managers in between have never wondered or even thought of checking how the domain names were managed."
              Who'd want to do business with such a badly managed organisation? Hint: that's a rhetorical question...

              1. Ken Moorhouse Silver badge

                Re: I just need to read my mail.

                The problem there, of course, is if the email address you've asked to be notified on is on a domain that has expired then you will be oblivious of having to renew it. The domain portfolio I manage is all on auto-renew so, so long as I keep my credit card details, notably the expiry date, up to date before that expires* then things are ok.

                The observation to make there is that we live in a risk-laden world because, if your credit card expires and your notification domain expires, you're scuppered. One good point though is, with Nominet, they do go to some lengths to protect against the loss of domains by their owner. Some years ago I thought I had lost my primary domain due to a dispute with the agent I used at the time (ah, add "trustworthy agent" to the list, asterisked above). Even though it had expired by the time the dispute was resolved I still managed to get it back by liaising directly with Nominet.

          2. MachDiamond Silver badge

            Re: domains may be registered up to 10 years

            My ISP sends me notices every other day for a month when my domains are up for renewal. You might think that somebody would have picked up the phone and called a company such as IBM to find out if they were going to renew the domains. I know I would have. If IBM was going to let some domains lapse, I would gobble them up with all haste and either sell them or set up a competing business.

          3. Anonymous Coward
            Anonymous Coward

            Re: domains may be registered up to 10 years

            Who is left to write to?

      2. GruntyMcPugh Silver badge

        Re: domains may be registered up to 10 years

        Probably put it in the then Project Manager's Lotus Notes Calendar,.... then resource actioned them.

      3. This post has been deleted by its author

    2. LDS Silver badge

      Re: Pay peanuts, get monkeys!

      Maybe the reminder was lost in some Lotus Organizer file...

    3. Doctor Syntax Silver badge

      Re: Pay peanuts, get monkeys!

      "If you do this, it's easy to hedge against forgetfulness for at least 9 years"

      And therein lies the problem. It just increases the probability of forgetfulness in the future. If you persist in getting rid of everyone who knows how things get done there's damn-all chance that when the need to renew comes around there's anyone left who knows it needs to be renewed. And if the company has in the meantime screwed up its payments system if, by any chance, someone does know about renewals, the payment might not go through in time anyway.

      1. Jamie Jones Silver badge

        Re: Pay peanuts, get monkeys!

        No, the point is, you don't pay for 10 years and leave it, you STILL pay every year, topping it up so you've always got a 10 year buffer...

        1. Anonymous Coward
          Anonymous Coward

          Re: Pay peanuts, get monkeys!

          If I were IBM I would offer to pay upfront for 100 years.

    4. Paul Hovnanian Silver badge

      Re: Pay peanuts, get monkeys!

      "Domains do not have to be renewed every year."

      This is true. On the other hand, my registrar had a requirement (or strongly suggested) that domain owners should check their Whois entries at least annually. One needs to keep contact information up to date in addition to making the occasional renewal. Smart companies will set up an e-mail account specific to the domain maintenance function (and NOT through the MX record attached to the domain in question) and assign the monitoring of this to whoever has the oversight of this resource.

  4. Nolveys

    Sounds like it's time to fire the person who fired the person who fired the person who was responsible for domain renewal.

    Unless that person is in India, in which case everything is going according to plan.

  5. Ken Hagan Gold badge

    Redundancy?

    So they were relying on a set of domain names and they are *all* renewed together, so that they all expire at once if someone forgets.

    Can we add this one to the RAID-is-not-a-backup list?

    1. Jamie Jones Silver badge

      Re: Redundancy?

      What's more, it appears they were relying on an external service for their internal communicatons to work.

      If your name expires, or your internet link goes down, or terrorists shut down the .com servers, the only things that should be affected are the external connections.

      Your own domain should be anchored in your DNS structure so your internal network isn't potentially vulnerable to an external action by a third party. Simply speaking, if someone "on the internet" can screw up your internal servers, you're doing it wrong.

  6. Conrad Longmore
    Coat

    In the old days..

    In the old days they would have been even more screwed. When domains expired they used to drop straight away and could be reregistered almost instantly. That changed a bit more than a decade ago. If you think that getting your domain out of REDEMPTIONPERIOD is a pain, trying to get it back from some anonymous domainer who wants a fortune for it is even harder..

  7. Missing Semicolon
    FAIL

    SoftLayer are not a Cloud Provider

    They are an old-fashioned VPS hoster. And an expensive and inflexible one at that!

  8. Unicornpiss Silver badge
    FAIL

    This may not have happened..

    ..if IBM wasn't so keen on cutting staff recently. Not only does this lead to situations like: "Oh, Jones was in charge of that, wasn't he?" "Jones was let go a year ago." ..but also doesn't really inspire much extra effort to notice such things and be proactive when your employees' morale can only be measured with a microscope.

    In the end it's likely to have cost them more in the long run than any short term savings.

  9. nil0
    Terminator

    RotM

    At least when the machines rise, we'll only have to survive a year at most before some vital domain lapses and humankind can re-emerge.

    1. Yet Another Anonymous coward Silver badge

      Re: RotM

      So if IBM build the terminators they will be made redundant before they reach you

      HP terminators will fall apart as the left and right legs are split into different companies

      MSFT terminators will suddenly get converted into fluffy 3d bunnies by the fall creative update

  10. Duffaboy

    Outsourcing

    Never saves you money in the long term

    1. Anonymous Coward
      Anonymous Coward

      Re: Outsourcing

      But in the short term I make millions :-)

  11. Martin M

    Idiots

    Even given the incompetence of this particular company, it stuns me that they do not have an automated monitoring system watching the critical domain expiry dates and raising a P2 ticket a month before and a P1 ticket a week before. And also daily generating P3 tickets that get automatically shut by a different system, so you know if either the watcher or the watcher's watcher fails.

    1. Anonymous Coward
      Anonymous Coward

      Re: Idiots

      If they use Spiceworks and whatever domain my current employer is using, Spiceworks regularly reminds them that their domain is expired. Leads to a mini heart attack on my part until I verify we're still live.

      At a former employer, we had a nifty (and very short) domain name that was grandfathered in, and would not have been allowed under the (then current) rules for domain names. I interpreted that to mean if we allowed it to lapse, we were hosed.

      The boss insisted on renewing annually and as close to the renewal date as possible. I always asked if I should plan on not having a job the following year, since he obviously was not planning on being in business in 12 months...

    2. Anonymous Coward
      Anonymous Coward

      Re: Idiots

      If you worked with IBM you'd realise that even if they had such monitoring system the tickets would quickly be closed by someone worried about the time required to complete the work causing an SLA breach.

      Naturally, outages would fall under a separate team so closing the tickets quickly doesn't result in any problems for the SLA drone...

  12. Stevie Silver badge

    Bah!

    I'll wager that among all those people surplussed over the last year are a bunch involved with keeping the books straight on matter like this.

    This is the way cost-cutting works. Keep paring back until something falls apart, then put that part back again while making excuses.

  13. Anonymous South African Coward Silver badge

    Outsourcing hard at work :)

  14. mako23

    Expect more of this to come

    I suspect the person who was aware about the expired date was fired during the many head reduction programs on European and USA workers.

    1. Anonymous Coward
      Anonymous Coward

      Re: Expect more of this to come

      It was me !

  15. MachDiamond Silver badge

    Online = out of business

    If you are relying on a cloud provider for your business, you may want to consider this issue. You have no way to throw money at the problem to get it fixed and may have no way to work around the problem. Just because you have gone with a large well known computer firm, it's still not a guarantee that they aren't going to screw up. It is almost guaranteed that when they do, it will be spectacular.

    Remember when there was a banking network problem and stores like Tesco couldn't ring up a sale?

    1. Anonymous Coward
      Anonymous Coward

      Re: Online = out of business

      The problem here is IBM - same service level as the worst outsources, but charge more and wrap doing things in so much process that you never do anything.

      As for your Tesco example, if only Tesco was small enough to not be affected by such problems. Or more accurately, small enough that no one wrote stories about it happening.

      If you're small enough to be able to do most things in-house, I envy you the simplicity Meanwhile other companies expand and make systems work across providers for.a large number of services from banking to Internet access.

  16. Anonymous Coward
    Anonymous Coward

    In the early days

    Working for a FTSE 100 firm a bright spark realised the obvious domain name was available and registered it to himself. He then sold it to his employer for a small sum, he wasn't greedy but I think his bonus was quite good that year.

  17. Anonymous Coward
    Anonymous Coward

    One year at a time?

    Whois reports network-layer.net is renewed for one year and the registrar is CSC Corporate Domains, Inc.

    1. Anonymous Coward
      Anonymous Coward

      Re: One year at a time?

      Have used CSC for managing a five figure number of domains. A little expensive for small numbers of domains but it's a fully featured service for managing domains with different company accounts and administrative permissions across the portfolio.

      To manage to not see the reminders in a well setup CSC account you'd need to get rid of a lot of people, have a crap email platform and a lot of idiots. IBM have done well to only lose three domains....

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2020