back to article After IBM SoftLayer fails to scrub bare-metal box firmware of any lurking spies, alarm raised over cloud server security

Cloud providers renting out bare-metal servers must make sure they scrub every last byte of writable storage on their boxes between deployments, infosec outfit Eclypsium has urged. Otherwise, malicious customers could stash spyware and other malware in motherboard flash memory that secretly activates when the next user of a …

  1. Hans 1

    Cloud providers cannot trust third party hardware unless it is open hardware with open firmware, nor can anybody else.

    Cloud providers offer full access to proprietary hardware with proprietary firmware, nobody knows what the thing is doing, and their customers can break out of their garden if they can get their mits on 0wned firmware for the device.

    "I only trust firmware I doctored myself."

    1. Korev Silver badge
      Big Brother

      And $CLOUD would also need to verify the open hard/firmware in the documentation is the same as what is delivered...

      1. Hans 1

        Not only that, say, $CLOUD uses open hardware and firmware, they can trust that .... AND, unless they provide that to their clients as well and allow their clients to edit and flash firmware, the clients still cannot trust.

        That is the whole point of open hardware and firmware -> at any given time, I know EXACTLY what that piece of hardware is doing.

        1. LDS Silver badge

          "at any given time, I know EXACTLY what that piece of hardware is doing"

          Good luck with "open hardware", especially, waiting for open blueprints of every hardware piece... and every microcode/firmware they could run.

          1. Anonymous Coward
            Anonymous Coward

            Re: "at any given time, I know EXACTLY what that piece of hardware is doing"

            .. and having access to the skillsets to properly review this for vulnerabilities. The Obfuscated C Contest is but a hint of what depth of competency you need to catch questionable aspects, ditto for the hardware. That translates directly into money.

            It isn't easy to avoid having to invest trust somewhere in your supply chain. Unless your budget is paid for by the taxpayer, you will have to make choices at some point.

        2. steveayre

          Reflections on Trusting Trust

          You might want to read Reflections on Trusting Trust by Ken Thompson. Even if your hardware, firmware and software is open you do -not- know what the hardware is actually doing. As soon as you introduce a third party you cannot trust the compiler or the manufacturer to do precisely what you asked without adding something else. And even if you bootstrap the entire system yourself from logic gates up there is still a chance that malware could infect your compiler and introduce an extra payload (as has happened in real cases).

    2. LDS Silver badge

      What the difference with open firmware?

      Are you going to download and disassemble it, and compare it with source? If you want to check hashes only, you can do it with proprietary firmware as well. If you reflash firmware, it doesn't matter if it is open or not.

      Unless you don't trust the OEM, but that's another matter. You can ban Huawei, for example <G>

      1. Hans 1

        Re: What the difference with open firmware?

        Are you going to download and disassemble it, and compare it with source?

        Ever heard of "build from source" ?

        Huawei ? They traded with Iran and are now a USian target. Not that I would trust their hardware, neither more nor less than any other thing non-open out there, mind ... if I cannot doctor the firmware from the source, then I cannot trust.

        1. s151669

          Re: What the difference with open firmware?

          Ever heard of login, c compiler and Ken Thompson? So much for building from source.

        2. LDS Silver badge

          "Ever heard of "build from source" ?"

          On hardware you don't fully control? In any given instant there's nothing to ensure the system is running the firmware you built.

          While it is true you should not blindly trust anybody, if you don't trust anything and anybody, you'll end up not trusting yourself too.

          Good luck, anyway - you'll have to collect your silicon from a trusted soruce, ensure it wasn't tampered with, check the "open" foundry equipment, than check your photolithography equipment has not been hacked (good luck in finding an "open" one, since very few make them...), etc. etc.

          1. Hans 1

            Re: "Ever heard of "build from source" ?"

            On hardware you don't fully control?

            I am not advocating for $_CLOUD. I am merely saying $_CLOUD is folly, because you do not know what is running where, where your data is stored, if and where it is copied to ... granting customers full access is dangerous, because they can get firmware that can bork the system .... only a matter of time until that happens.....

            While it is true you should not blindly trust anybody, if you don't trust anything and anybody, you'll end up not trusting yourself too.

            Now, one moment, logical fallacy, here, sir. I trust myself. I know I will never be 100% certain that a certain piece of hardware in my data center, manufactured to open designs, running open firmware that I compiled and flashed onto it will run exactly as intended, sure.

            But compare that to closed hardware, with closed firmware, I even grant you running a partially open OS (GNU/Linux with binary blob drivers), running on a third party system somewhere, where others may have access to the underlying hardware, because multiple customers happen to use instances on one same physical system to which we are all given full access to the OS running ( in our instance ) ... folly, I tell you, folly. Ever heard of break out of sandbox vulns ?

            Even closed hardware, with closed firmware in my data center cannot be trusted not to leak data. Sure, no other choice now, but the minute open hardware that is shipped with open firmware and that performs reasonably well, I will jump ship in no-time, as any sensible person would.

            We should always strive for perfection.

            In short:

            1. $_CLOUD is utter folly

            2. Closed hardware/firmware/software is folly*

            3. Closed hardware/firmware with partially open OS (GNU/Linux with binary blobs) is better

            4. Open hardware, running open firmware and software (free of binary blobs) in your data center is as close as you can get to perfection. You can vet, or have vetted, each and every line of code that gets compiled and run.

            You can say compiler could be 0wned, sure, but that could potentially happen to any and all.

            You can say manufacturer could add a backdoor with some hidden ROM firmware somewhere, but that could potentially happen to any or all as well. And you can check the chips on the board, if need be.

            Why you are preferring a far less trusted system is beyond me. You can say what you want, open stuff is better than closed stuff - like Democracy, with freedom.of information act, is better than USSR.

            * Cf article on this site about Dutch authorities kicking Microsoft who REPEATEDLY denied copying sensitive data to servers in the US, even when faced with the evidence. Even MS does not know what Windows and Office are doing.

            * Intel ME, anyone ?

  2. petethebloke


    "...grin and bare it".

    Honestly. My old man's jokes were better than that.

  3. Thomas 6

    "We suspect the big players do this anyway..."

    Ouch! Sucks to be IBM in 2019.

    1. Hans 1
      Thumb Up

      If you use proprietary firmware, your life sucks as well ... ;-) Just like mine, agreed, but things must change!

  4. Anonymous Coward
    Anonymous Coward

    So don't trust hardware you don't own ?

    When's egg-sucking for grannies on ?

    1. eldakka Silver badge

      Re: So don't trust hardware you don't own ?

      So don't trust hardware you don't own ?

      ...and any hardware you aren't in physical possession of.

  5. muhfugen

    What good does reflashing the BMC's firmware do? The only (supported) way of doing that is from the BMC itself, and if it had been compromised, then it could just ignore the new firmware and report a successful reflash. I really doubt they're removing the EEPROM/flash from the motherboard and flashing it and resoldering it.

  6. This post has been deleted by its author

  7. Anonymous Coward
    Anonymous Coward

    Problem Solved

    Maybe IBM should start using HPE servers with Silicon Root of Trust where a series of fingerprint authentication takes places (iLO ASIC, iLO, UEFI BIOS, ROM & OS Bootloader) to ensure FW has not been altered. If it has, it then provides recovery to a good known state

    1. TechDrone
      Black Helicopters

      Re: Problem Solved

      And a great way for HP to enforce that only their spare parts, upgrades and OS's can be run on their kit too.

      Or am I being too suspicious of their motives?

      1. Anonymous Coward
        Anonymous Coward

        Re: Problem Solved

        Nothing to do with OS. How would that be any different for any other server mfg? If you're CPU dies on a Dell or Lenovo do you run to Fry's and get a replacement? How about memory?

  8. Ken Hagan Gold badge

    Jumpers (for goalposts?)

    Perhaps we should go back to hardware that doesn't let you flash it unless a jumper is set one way. That would make it impossible for a remote user to "maintain" the system because they don't have physical access. I can see that being able to download a program that flashes your BIOS is convenient if the system has been sold to the great unwashed (ie, me) but the trade-offs of security and convenience are exactly reversed if the system owner is a VM provider.

  9. IanMoore33



    You are just a visitor

    1. Anonymous Coward
      Anonymous Coward


      Evidently not if you flash the BMC. Now it’s yours. Am I the only person hearing the sound of perpetual free bare-metal cloud machines?

      1. LDS Silver badge


        The hardware is still in someone else's room with full access to hardware... how much is it "yours"?

  10. a_yank_lurker Silver badge

    An argument for

    An on premise server farm. With the cloud you are renting space on someone else's hardware.

  11. fredesmite

    Softlayer is GARBAGE

    We used Softlayer for a IBM openstack project .. they use supermicro garbage ..the worse availability ever . I

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2022