back to article IBM's SoftLayer is having a meltdown – and customers aren't happy

An extended outage over at IBM's Bluemix SoftLayer portal has customers fuming – and they say there is no way for them to tell Big Blue about it. A Reg reader whose biz runs more than three dozen servers within the SoftLayer cloud told us the platform has been out all day and the management portal website has been inaccessible …

  1. Anonymous Coward
    Anonymous Coward

    "A Reg reader whose biz runs more than three dozen servers within the Softlayer cloud.."

    I don't mean to be mean, but what on earth did you expect?

  2. Nolveys
    Windows

    Thank you for calling IBM technical support.

    Please listen carefully to the entire painfully slow recitation of our menu options as they are generated at random.

    If you would like to talk to a qualified IBM technical support representative then please contact your local unemployment office, fiverr.com or Craig's List. If you would like to talk to an Indian prison convict who is working out of a repurposed textile mill that's constructed mostly out of asbestos then press 1.

    If you would like to leave a message, voicing your support for IBM hiring US citizens to replace the US citizens we recently sacked then press 2, wait for the tone and say "Thank you Mister Trump for supporting IBM, a truly patriotic and American company. I am very glad that I voted for you because you put America first".

    If you would like to hear Ginni Rometty and the board of directors laughing as they throw money at each other then press 3.

    If you wish for our sales department to lie to you, press 4.

    If you would like The Hold Queue To Nowhere then press 5.

    If you would like to attempt to contact technical support through our web site then please visit "http://sap12.abandoned-data-center-14.ibm.com/barelyfunctional/" and enter your secret code. If you do not have a secret code then search your IBM hardware, software, manuals, promotional material, empty envelopes and packing material for it. If your secret code is rejected and you are wondering if it's because you are using the wrong code or if it's because our web site is crashing then try to enter it again. If the web site acts in a way that does not resemble the way it acted any of the other times you tried to use the web site then either you have the wrong code or our web site is crashing. If the web site responds in the same way it responded in prior attempts then you are accessing some other company's web site, check the URL and try again.

    Press 0 at any time to talk to a human...sounding program that will repeat the aforementioned options in a condescending tone.

    The menu option recording will now end and the line will remain dead until you make a selection. Thank you.

    1. Anonymous Coward
      Anonymous Coward

      Re: Thank you for calling IBM technical support.

      hmmm.... no mention of the word cognitive or Watson in the menu, that's what must be wrong!

    2. Sil

      Re: Thank you for calling IBM technical support.

      Well done!

      Are you an ex IBM employee by any chance ;-)

      1. Nolveys
        Meh

        Re: Thank you for calling IBM technical support.

        Are you an ex IBM employee by any chance ;-)

        No such luck. I have worked with quite a few IBM servers though, such a strange mix of excellence and complete garbage. The rack mount ThinkServers were built like tanks, were easy to service and the insides were incredibly clean. Then there were the tower ThinkServers, that were made of corrugated plastic, felt like they were made of thick jello and the insides looked like they were laid out by some very large bird.

        Recently I had to activate an extended warranty on a new ThinkServer, which, in spite of being sold by Lenovo, entailed traversing some of the older and more forgotten parts of Gormenghast^WIBM Support. The machine had a sticker with a 1-800 number that lead to a phone system designed by M. C. Escher, with doors that lead to black voids from which voices would say things like "It's so dark, I don't know where I am. Please help me." and signs that read "Please Ignore The Pleas Of The Damned".

        I eventually made it to a recording that pointed me to a web site that was clearly the front end for some SAP abomination. The "html" forms were fascinating, typing in a text input field caused a delay while the individual characters were sent to The Great Unknown and then, eventually, came back and were echoed into the input field via some javascript disgorgement (I assume). The thing would regularly time-out simply moving individual characters from and back to the browser. Clicking buttons would, when it didn't time out, cause seemingly random and completely bizarre error messages to be displayed. "SYS0014A722FE-00-97125: Ph'nglui mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn."

        I eventually got to the part of the web site where I had to enter the secret code. The code came from a sticker on the machine with an anonymous alpha-numeric string on it (one of many). The code came in two parts, the serial number and the "mystery number". The code on the machine was something like ME17573729-4572381276FEA94483902-FOO3315751. So the serial number was "FOO3315751" and the mystery number was "FEA" (4572381276[FEA]94483902). Of course, when I eventually found these numbers and got to the place in the web site in which to enter them they didn't work. Someone had entered the wrong warranty and it was incompatible with the serial and mystery numbers.

        I really hope that the damn thing doesn't break while under warranty, I have no idea how to summon someone to fix it.

        Then there was the time I worked under an ex-IBM sales person[1], evaluating the suitability of IBM Cloud services. The advertising material made it sound like IBM Cloud would solve all our problems, but everything it referred to was too poorly-defined to actually figure out what the hell they were talking about. It took me weeks to eventually get through IBM Sales to a tech who could answer my questions, and the answers amounted to "None of the sales literature means anything, this is all vanilla RedHat VM stuff."

        I had actually worked with the ex-IBM guy many years earlier[2] and it was interesting because almost nothing he ever said actually meant anything. I'd never heard anyone talk like him before...until I spoke with IBM Sales. Lots of conversations like this: "This setup is dangerous. If one hard drive goes we will loose significant data and be dead in the water." "(sly smile) It's all good, baby. You just gotta, just, you know. It's like, you just, it's all good, baby."

        No, I've never worked for IBM. However, I have caught a few brief glimpses into that bizarre world.

        [1] Maybe I'll tell you about that once the criminal fraud proceedings have concluded.

        [2] Massive investor money bonfire, huge legal fallout, leans placed on common areas of buildings, preventing normal residents from getting to their apartments, etc. The CEO spent the last months in the server room, shredding documents day and night. He then moved to a different country, shaved his head and started a cult. I swear I'm not making this up.

        1. Anonymous Coward
          Anonymous Coward

          Re: Thank you for calling IBM technical support.

          Sounds about right. I've worked under ex-IBMers, contracting for an ex-IBM division. I bet they still use the same 50+ year old database systems, and rely on outside contractors using Excel to get anything done. Above my pay grade! Job security! Keep your head down! mmm, sounds like Soviet Russia.

          This was all US government pork barrel crap. A total waste. Drain the swamp already...

          1. Anonymous Coward
            Anonymous Coward

            Re: Thank you for calling IBM technical support.

            Not outside contractors. Interns. Hundreds of them. Know feck-all about IT, but can run rings around you when it comes to Excel spreadsheets......

        2. Adam 1

          Re: Thank you for calling IBM technical support.

          > completely bizarre error messages to be displayed. "SYS0014A722FE-00-97125: Ph'nglui mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn."

          I see what you've done. You've accidentally changed your preferred language to Welsh.

    3. fredesmite
      Thumb Up

      Re: Thank you for calling IBM technical support.

      Thank you . Now which button do I press to get a factory job building something IN AMURIKA ?

  3. Anonymous Coward
    Anonymous Coward

    The Cloud..

    Other peoples computers you have no control over, and the US spooks have full access to.

    1. Anonymous Coward
      Anonymous Coward

      Re: The Cloud..

      I guess they have no access to your's then, you poor naive fool.

    2. Anonymous Coward
      Anonymous Coward

      Re: The Cloud..

      The whole "cloud is not secure" line doesn't make any sense. A company like Google has an army of PhD's working in security, literally wrote the book on many security domains, use non-public/non-commercial technologies, have guaranteed security SLAs. When you use their cloud, you basically have the world's best security consulting firm managing your infrastructure. No way the average company can match the hyper scalers' security.

      1. eldakka

        Re: The Cloud..

        "The whole "cloud is not secure" line doesn't make any sense. A company like Google has an army of PhD's working in security, literally wrote the book on many security domains, use non-public/non-commercial technologies, have guaranteed security SLAs. When you use their cloud, you basically have the world's best security consulting firm managing your infrastructure. No way the average company can match the hyper scalers' security."

        And in the face of a single NSL, it doesn't matter what level of encryption the cloud provider uses or how many PhD's they have on staff.

        1. Anonymous Coward
          Anonymous Coward

          Re: The Cloud..

          Notice how the big tech companies have reacted to the government asking for any data lately. Wholesale rejection of whatever the government wants from a data perspective regardless of why they want it. The major IT companies, Apple and Google in particular, understand that the reputational damage associated with divulging data is substantial and they are not going to do it. The average business or organization probably isn't going to get into a lengthy lawsuit with the federal government over data requests. If they have an NSL in hand, they'll likely just turn over data vs getting into a multi-million dollar lawsuit and hiring lobbyists to work Congress on their behalf. The huge IT providers are in a better position to fight back there as well.... The only system or data which is perfectly secure is the system or data which doesn't exist, but the cloud providers lower the odds of data breach by a wide margin. They are certainty not worse than the average company throwing up a commercial firewall and some commercial anti-virus software with off the shelf encryption. If you are concerned about the NSA looking at data, the commercial security software and firewalls will not even slow them down.

  4. Halfmad

    Social media for support

    What? Really people are happy to accept tweeting or posting on a FB page as a method of contacting support? Talk about keeping customers at arms length.

    1. Flakk

      Re: Social media for support

      Seriously. It's like people have never heard of an SLA before. I'd guffaw over any IaaS agreement that did not provide me a list of proverbial "throats to choke", starting with my sales exec's boss and ending with the NOC.

      I did business with SoftLayer for about four years, before the got gobbled up by IBM. I never had problems like these. It's a shame.

      1. eldakka

        Re: Social media for support

        And in the language of SLA's, a PLANNED outage is not an outage that breaches an SLA or counts against uptime or as "an outage".

        You can guarantee 5 9's uptime (99.999%), but:

        1) That still allows 8 hours unplanned outage a year;

        2) if you require a 4-hour complete outage once a week for maintenance, that does not count against the 99.999% figure for uptime as it is a planned, scheduled outage.

        That is the legalese around SLA's and the definition of uptime and outages.

        Read very carefully, don't assume that what you consider "an outage" is the same thing as what the contract defines an outage as - or that the people doing the actual contract negotiation even understand what the difference is.

    2. Anonymous Coward
      Anonymous Coward

      Re: Social media for support

      We live in a instant gratification work ...

  5. scrubber

    The salespeople and evangelicals told me...

    ... when you put stuff in the cloud it is immune to outages.

    1. Anonymous Coward
      Anonymous Coward

      Re: The salespeople and evangelicals told me...

      That's a different cloud though. You're thinking of a provider with actual cloud scale facilities (and API's to support their services) rather than a regional hosting company, brought by a dinosaur falling behind in "the cloud" and then left to fester.

      With Softlayer^H^H^HIBM, an outage is just an example of why you should pay more money to IBM.

      If you were an existing Softlayer customer and wondered how IBM would develop the business, you've seen the future.

      If you're in Softlayer AFTER IBM brought the business, I have a bridge that I can sell you to greatly improve your reliability.

    2. fredesmite

      Re: The salespeople and evangelicals told me...

      you simply can't move something to "the cloud " .. apps have to be cloud aware from scratch.

    3. Anonymous Coward
      Anonymous Coward

      Re: The salespeople and evangelicals told me...

      Yes - but they only say things like that BEFORE your PHB has handed over an eye-watering amount of money for a service you're never going to get.

  6. Anonymous Coward
    Anonymous Coward

    Serial number

    The code on the machine was something like ME17573729-4572381276FEA94483902-FOO3315751. So the serial number was "FOO3315751" and the mystery number was "FEA" (4572381276[FEA]94483902).

    FFs -- everyone knows that.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like