back to article If malware wants to bury deep inside your Lenovo or Gigabyte servers, they can just ask Vertiv's insecure BMC firmware

A pair of vulnerabilities in BMC firmware used in servers built by Lenovo – and in Acer and Penguin Computing boxes using Gigabyte server motherboards – can be exploited to hide malware deep below the operating system, hypervisor, and antivirus. Said spyware could lurk out of sight and undetected by the OS, security tools, …

  1. whitepines
    Alert

    Who else is sick of these proprietary, unremovable, unreplaceable, often hostile black boxes in their machines? Where this gets really scary is that even the firewall systems probably have a BMC of some sort, so even blocking the BMC at the edge isn't always feasible. Plus, these kinds of hacks can be utilized from a single intrusion on the host machine itself, to install an APT no one is looking for. One that will survive the normal cleanup process after the intrusion is detected.

    Something that can survive an OS reinstall / hypervisor reload is not to be taken lightly unless your org has the tools, datasheets, binary files, schematics, etc. to reload the mainboard ROMs externally before reloading the OS / hypervisor. Very few do.

    I'm personally excited about OpenBMC, as a Linux shop we've used it with OpenPower machines and it's a big step up from AMI etc. A little rough around the edges but it sure beats this kind of mess, especially with the peer review model already having caught fun security holes that the proprietary vendors just ignored and hid for years (CVE-2019-6260 and mates).

    1. Hans 1
      Thumb Up

      Who else is sick of these proprietary, unremovable, unreplaceable, often hostile black boxes in their machines?

      Me!

    2. _LC_
      Stop

      Intel ME

      Not just a backdoor, but an entire system running in background for surveillance.

      1. Aitor 1

        Re: Intel ME

        Yep, a full operating system.. runnning on your CPU, not for your benefit.. yet YOU pay for it.

        1. Anonymous Coward
          Anonymous Coward

          Re: Intel ME

          You might be thinking of "active management" features from the CPU designer, and confusing it with BMCs.

          BMCs are typically stand-alone modules (now evolved to single chips designed into the main board). They often run a stripped down Linux, with no external indication. The vendors are often cagey about the software stack, implying that they wrong it from scratch, and rarely complying with the original license requirements.

          1. whitepines

            Re: Intel ME

            The ME with AMT is basically just Intel integrating BMC functions within their silicon. I don't care if the BMC is off die, on die, or suspended in a pocket universe near the chip, it's still a BMC and, for security, we should be able to fully control and manage it as the miniature network-connected computer it is.

            Anything else is planned obsolescence in the best case, and downright malicious in the worst.

    3. phuzz Silver badge
      Thumb Up

      "Who else is sick of these proprietary, unremovable, unreplaceable, often hostile black boxes in their machines?"

      Well, I have to actually administer servers that are further away than the next room, so personally I'm all for some kind of 'lights out' management system.

      Being able to reach the console of a server before it's booted has saved me from having to drive for an hour and find a monitor/keyboard to plug into a machine in a rack on multiple occasions. HP's iLO even allows you to mount an iso as a virtual CD and install an OS from scratch without having to be anywhere near the server.

      That said, unless you're buying cheap servers, usually the management system has it's own network port, and it's up to you as the sysadmin to keep that management network locked down.

      1. whitepines

        So do I. In fact I do this daily across multiple geographically diverse datacenters. What I don't want, and cannot trust, is some random piece of closed, proprietary software that can do all kinds of nasty things in the background as this article shows (and multiple other CVEs also attest to). Remote access is fine, but let us manage the BMC the same way we manage our other Linux systems -- don't go pretending it's "just part of the hardware" when it's just about anything but that.

        Hence why I like OpenBMC and why OpenPower (which ships with OpenBMC) has been great for our uses. YMMV -- if you're a Windows shop, you've got bigger problems to worry about than the system firmware at any level!

      2. Hans 1
        Windows

        Well, I have to actually administer servers that are further away than the next room, so personally I'm all for some kind of 'lights out' management system.

        Being able to reach the console of a server before it's booted has saved me from having to drive for an hour and find a monitor/keyboard to plug into a machine in a rack on multiple occasions. HP's iLO even allows you to mount an iso as a virtual CD and install an OS from scratch without having to be anywhere near the server.

        Yeah, I totally agree, handy, that ... but it does not have to be a black box that you cannot control, nor configure freely, inspect, and update

        1. phuzz Silver badge

          "it does not have to be a black box"

          Unlike the BIOS/UEFI, or the CPU microcode, or the firmware on the network card, or the RAID card, or on the backplane, or even the firmware of the drives themselves?

          1. _LC_

            Granted, but something like the Intel ME is working autonomously in background, virtually undetectable. It can even bypass sniffers on its interface.

      3. Anonymous Coward
        Anonymous Coward

        " HP's iLO even allows you to mount an iso as a virtual CD and install an OS from scratch without having to be anywhere near the server."

        Nice!

        That is the same technique that is thought to have been used to spread the Stuxnet worm.

        Place an .iso containing executables on a hidden, encrypted partition on a U3 enabled flash drive to abuse Microsoft's autorun vulnerabilty.

        https://en.wikipedia.org/wiki/U3_%28software%29

        [PDF]

        http://www.irongeek.com/downloads/Malicious%20USB%20Devices.pdf

      4. Anonymous Coward
        Anonymous Coward

        @phuzz & I get paid to sit on my A***, screw your security

        As a real administrator, part of your job should be making certain that the systems you maintain are secure, if you have to get out of your chair to do it then perhaps that is part of your job too.

        Any remote management tool that cannot be removed/disabled is inherantly insecure, so what if it is on a seperate nic they are both software controlled and just waiting to provide a bridge into your so secure maintenace network.

        KVM (keyboard,mouse&video) streamed over IP on completely seperate network and hardware, locally enabled as required,is okay if you must but nothing running when not needed and never from the same hardware or you are just asking for it to become a bridge.

        Mind you, if you didn't say anything about them putting in Lenovo hardware given the many articles here then perhaps you have not been keeping up with the field.

    4. vtcodger Silver badge

      Not firm enough?

      I think what we're dealing with may be WFIIPS (We'll Fix It In Production Syndrome). Back in the day, "FIRMWARE" was firm -- programmed into blown fuses on a chip ... or maybe hard wired into the silicon interconnections. Upgrading it required prizing out and replacing a DIP package. Or maybe soldering an upgraded chip onto the motherboard. Not surprisingly, the firmware did very little, was pretty primitive, and was virtually never upgraded.

      Nowadays firmware is exceedingly flexible. Too much so perhaps. I think maybe too many "firmware" changes are done to fix stuff that should have been better tested or omitted entirely. I think there is a case for firmware should do less, should do it better, and should be much harder to alter. Not solder in a new chip harder, but maybe something along the line of no upgrade allowed until someone has inserted a physical key and turned it to "Service Mode" harder.

  2. JimmyPage
    WTF?

    Interesting legally, rather than technically ...

    I wonder how many MegaCorps have bought kit and had a T&C somewhere in the deal that required a declaration by the supplier/manufacturer that the kit was secure from such bright ideas as BMC ?

    Or, more to the point, I wonder how many will from this point on ?

    Now that's a popcorn-fest waiting to happen ....

    1. whitepines

      Re: Interesting legally, rather than technically ...

      If the legal folks do it right, that means no Intel kit (ME is effectively an even more invasive BMC than the one described here, and is required on all Intel desktop / server products) and AMD is legally questionable due to the PSP. Even ARM and Power would get in on the fun, since they also need BMCs.

      A better way of doing this would be to mandate the org has source code and control of the BMC, so they can strip it down to basic "boot the platform" tasks if they want vs. "spy on the owner for DRM, enable network access to everything, etc.".

      1. J. Cook Silver badge
        FAIL

        Re: Interesting legally, rather than technically ...

        Or Cisco Kit. (CIMC) Or Dell Kit. (iDRAC/DRAC). Or HP (iLO). or just about every other major server builder with IPMI support (Supermicro uses the affected BMC in their complete server models for their IPMI support.)

        If you want mitigation for the problem, look at configuring the BMCs to use a dedicated network port and either leave it unplugged, or put it on a heavily restricted network if you have a need to use it.

        1. whitepines
          FAIL

          Re: Interesting legally, rather than technically ...

          either leave it unplugged, or put it on a heavily restricted network if you have a need to use it.

          How does this help against the described host-based attacks that leave a nearly untraceable APT in place that can quietly re-infect the host? Or brick the servers entirely?

          As an analogy, sticking your fingers in your ears and singing "God Save the Queen" may make it so you think no criminals are about (since you can't hear anything), but it doesn't make you immune to physical damage from said criminals. By focusing so heavily on the network port, you've only managed to disable one possible way in, not actually dealt with the APT threat posed by the shoddy coding and poor security posture most proprietary BMCs offer. Oh, and have you made sure your BMC actually obeys the request to listen on only one network port? Some don't, and of those there exist some that wait some hours before starting to listen on the second port...

  3. Claptrap314 Silver badge

    1999 called

    We were crypto-signing microcode patches to CPUs no later than the original K7 (Athlon). Maybe the K5? (1996) What in the **** is someone doing releasing firmware in 2014 that doesn't require signatures?

    At best, that's criminal (as in, people should go to jail) stupidity.

    1. whitepines
      Facepalm

      Re: 1999 called

      Oh good. So your solution is ... more blind trust of the vendor with no SLA and no legal / financial penalties for firmware that goes against the interests of the machine owner (bugs, etc. or just plain DRM)? How'd that work out with the ME, one of the most locked down and crypto signed bits of semi-malware in wide release? Hint: repeatedly hacked, only the good guys locked out of "their" machines. Sound familiar?

      If you think this is the solution, I guess you're also good with signature checks requiring the machine only boot Windows 10 (or the latest version of Windows, configured for your safety by non-reversible firmware security updates) and Windows also only allowing Microsoft-signed / authorized apps? Whoops, guess you need a (paid) license to make anything that can run on PC now.

      1999 called all right. Project Palladium was rejected last decade for very good reasons. Let's not bring it back alongside the Clipper chip, mmkay?

      1. Claptrap314 Silver badge

        Re: 1999 called

        If you have EVER installed software with elevated privileges that you did not decompile with a decompiler you compiled yourself on a compiler you compiled yourself.... Then you have "blindly trusted" someone else's code.

        The issue is about how hard it is for a third party to inject code into your system. Best practice has been to require the code be signed.

  4. Anonymous Coward
    Anonymous Coward

    I've got an idea...

    How about any kind of update to anything lower-level than a hard drive requires the pushing of a physical, hardwired button? As in, the electrical circuit needed to update the firmware isn't powered up until the button is pushed, and then only for 10 minutes? And button would be behind a screwed in cover on the computer. Thus updates to a BMC, BIOS, etc. would require a tech to be physically present and intentionally updating them, preferably through a dedicated USB port (not visible to the OS), to make it harder for malicious software to infect the update.

    I'd love to see this on home machines as well. Imagine having to trick a user into taking the case off the computer and plugging an infected USB stick into a hidden port in order to infect firmware. I'm pretty sure that would be the end of firmware malware as we know it, aside from anything installed in the supply chain.

    For that matter, how about another button, or the same one but push-and-hold for 20 seconds, to wipe the firmware and reinstall from an onboard ROM (proper absolute never-rewriteable ROM) chip, to allow for a true restore to factory defaults?

    1. _LC_

      Re: I've got an idea...

      Great idea, which unfortunately collides with the interests of our "shepherds". ;-)

    2. Charles 9

      Re: I've got an idea...

      How funding all the expensive trips to perform all these resets when not if they become necessary. There's a reason remote administration became a thing: electrons cost a LOT less to move (especially over long distsnces) than a man: to the point firms may prefer to gamble.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like