Enjoy the holiday weekend, America? Well-rested? Good. Supermicro server boards can be remotely hijacked Tens of thousands of servers around the world are believed to be hosting a vulnerability that would allow an attacker to remotely commandeer them. The team at Eclypsium says it has discovered a set of flaws it refers to as USBAnywhere that, when exploited, would potentially allow an attacker to take over the baseboard …
How many of the administrators of the vulnerable servers even know that the system has a Supermicro motherboard ?
How many of those even read security vulnerability notices ?
And how many of those know how to update firmware ?
And of those how many will bother ?
My guess is that at least 50% of the currently vulnerable systems will still be vulnerable in a years time !!!
Icon for the administrators who do not do the security updates ============>
Whoever hooks the management port up to the internet, without even putting a firewall in the way, kinda deserves whatever happens to them.
Mind you, whoever does that probably has equally terrible security practises everywhere else, so a BMC issue is the least of their problems.
There may be no firewall if you are on the same network segment but I highly doubt there isn't at least a rudimentary firewall on the 'Internet connection'. Then again some plonker could have just forwarded all ports to the system with the vulnerable BMC... I doubt anyone would be that stupid though :0
This post has been deleted by its author
I want to like AMD BTW (even though I've been fucked by them hard with the HP laptop of fire, going mad when that broke and the fan wasn't awlays on on the next laptop, not even needing a book to have it on ALL THE TIME because idling on a surface will cause it to shutdown for thermal reasons, the fx-8350 "8 core" crap ect)
You're mad at AMD because HP can't design laptops??
FWIW the intel based HP laptop I once had was a piece of shit too.
Really interested in the new Zen stuff, just trying to get over aforementioned fucking
Apart from being released too early (the early motherboards were a bit.... fun) the Zen stuff is really good. Plus being able to swap my 1600X for a 3xxxx and get the performance boost is a boon too.
Been building computers for years, had similar issues with both Intel and AMD over the years. Have had some complete rubbish, but also some amazing stuff, in both camps.
It wasn't the fastest, but my bought on day of release 'Bulldozer' box is still going strong; My current i5 box is a bit 'funny' in some situations, whereas the previous core 2 was bullet proof. The first Zen box I got was a bit flaky, but after a drunken accident took it out I bought better quality stuff and that's been super stable ever since.
Or am I missing something?
Well, you're missing the fact that most successful IT attacks against organizations of any significant size involve penetrating the corporate network through any one of the zillions of holes already punched in it - dodgy WiFi routers, employees with VPN connections and careless browsing habits, SSRF attacks, and so on - and then pivoting and escalating.
USBAnywhere is a pivot-and-escalate dream come true.
The "egg model" (corporate network hard on the outside but soft on the inside) has failed. While it protects from a huge number of everyday attacks, it nearly always fails quickly when confronted with a dedicated, knowledgeable attacker. Even security firms (e.g. Hacking Team) get taken by penetrate / pivot / escalate chains, often by lone, amateur attackers.
Our Supermicro supplier up to magbe a year ago used to ship boxes with the default login and dhcp enabled... So the minute it found power and an ethernet connection it was wide open - you didn't even need to hit the power button.
Machines destined for me got fixed pronto (so I could install an OS from my desk),but I realised for other machines, their 'owners' perhaps didn't even know they had IPMI!
Depends on the model but in general yes - however by default ipmi will jump to the other ports if the dedicated port isn't active.
ALWAYS _check_ your ipmi settings before letting a server loose on the network.
Of course some helpful individual could connect to all those open ones and issue a "power off" command.
Even if it's accessible only inside a network something into the network can be compromised in other ways, and this becomes another easy prey for lateral movements and persistence, even more so because it's on systems mostly used as servers, and some probably as network appliances as well.
I can't see how this can be done without port forwarding between the router and the server and or some similar form of NAT/PAT.
Or are there really people that mount servers 'Directly' connected to the Internet... Really, it's actually not so simple to do.. You would have to have more than just basic knowledge in order to do so, and if that was the case you would presumably know about the risks involved... Or am I missing something here?
I wouldn't be surprised if some bright mind did exactly that to allow for "remote manglement", especially some small installation with no on-site admin, and no VPN to access the internal network... or "just in case if the VPN doesn't work".
It does impact some "embedded" micro-systems (i.e. the SYS-E300 series) as well, that may have been configured that way easily.
I can see how. Compromise one target in a network, pivot from there. Plenty of networks out there that don’t separate vlans or that don’t use firewalling between those vlans.
It IS disturbing though that so many seem to have the interface exposed to the Internet directly.
Or are there really people that mount servers 'Directly' connected to the Internet...
Yes, it's not that uncommon. NAT is an abomination to be avoided if possible. At a previous job, we had a whole /24 to ourselves (legacy from when addresses weren't scarce) and all servers were on public IPs (IPv4). We did have a firewall between them and the internet though ;-) Go to any "pile em high and sell em cheap" web hosting provider and you'll find your server is on a public IP.
Really, it's actually not so simple to do
Actually it is trivial to do IF you have the public IPs. It's actually easier than putting them behind NAT.
Even if you only have the one IP and need to use NAT, it's usually easier to forward "everything" to one internal device than it is to setup lots of port forwarding rules. Port forwarding is more flexible, and offers less scope to mess up the firewall rules - such as failing to change the default "allow everything" rule in a Draytek.
Really, it's actually not so simple to do
It's absolutely trivial if you give it a public IP.
Also by the way there's billions of reasons to use IPMI not just related to no onsite admin - if you run thousands of servers you want a way to manage them, to provision them - it's the public IP *plus* flaws that's at issue here. They're both fixable.
From looking at the packet capture linked to in GitHub it looks like it's using the standard emulated CD iso like used in the old U3 enabled flashdrives back in the Windows XP days.
isolinux.bin
Also, I see several references to Samsung in the capture?
0x0050: 2020 2020 2020 2020 5361 6d73 756e 6720 ........Samsung.
Well, yes, that is what it is designed for.
While there are clearly some vulnerabilities here I can't help but feel that it is being over-egged in significance. In substance it amounts to an authentication bug, screaming "look what I can do" is hardly evidence of further flaws when those are documented features of the system.
Bear in mind the wider context here: these are professional grade server boards. People then pay a premium for these BMC equipped boards. If you opt to use a shared rather than dedicated IPMI port one of the first things you are asked is what VLAN you would like the remote managent on to keep the traffic segregated. Even if you never use it (possibly because you bought an inappropriate board in the first place) the vulnerability doesn't arise because it needs a prior login during that power cycle and before the BMC is rebooted.
So yes, there is a flaw here which has been patched. But do not start down the "dumb lusers don't understand this" road: it doesn't affect them and if for some reason they are using what are advanced professional-grade tools it is ultimately their responsibility to mitigate the well understood risks of these fetures.
Let me help:
BMCs are designed to be connected to physically isolated internal networks, or at the very least a separate dedicated non-routeable VLAN on an internal network.
It was mentioned in the article, in passing, that BMCs are rarely connected to the Internet. That is why. Yes, it would be much better if this crappy firmware were fired into the sun and replaced by a quality open source product designed from the ground up with a sensible security model in mind. Yes, it would be much better if operators demanded that Intel and AMD provide the necessary documentation to properly write and install such software into the BMC and other system components in place of the delivered binary-only firmware. Yes, it would be much better if this software could be customised to disable unneeded components to reduce the attack surface. No, that's not what's been happening, and everyone knows this stuff is total garbage which is why it's never connected to a globally routeable network. It's not an excuse, just a fact. You can use a steak knife to cut off your hand, but they're designed to be used to cut food and everyone knows that.
Shouldn't be surprised I suppose.. my last SM IPMI update(~5 years ago) part of the instructions was to wipe the configuration, I suspected that included the network configuration meaning it would no longer be accessible on the network after rebooting. But I tried anyway just in case. And sure enough yes the IPMI went offline at that point and I didn't have connectivity to it again for another couple of years (next time I went on site, fortunately had no HW failures in the meantime). Add to that the terrible documentation SM has on when firmware is updated, what is fixed etc(release notes seem more common for them on their newest stuff from the looks of it).
I replaced my personal SM server (which otherwise worked OK as in no failures anyway, my SM experience goes back to about 2001) last year with a Dell R230. For work stuff historically I use (since ~2006 anyway) HP, but in this case HP didn't offer a configuration that I wanted so went with Dell. Has worked well so far anyway. My personal server is at a co-lo and runs a half dozen VMs, though maybe will add more VMs got tons of capacity now.
Back to SM..
Since this article mentioned "X10" I wanted to see what the current situation is, so I poked around for an X10 board with IPMI
first web hit was this board:
https://www.supermicro.com/en/products/motherboard/X10SLM-F
seems recent "Single socket H3 (LGA 1150) supports Intel® Xeon® E3-1200 v3/v4, 4th gen. "
downloaded the IPMI firmware package, and at least in this case they give release notes and a list of fixes, but in the "IPMI Firmware Update_NEW.doc" file they still say in big red letters(had higher hopes given the "NEW" in the name)
"NOTE !!! Uncheck preserve configuration box during flashing (very important step for FW to work properly). All settings will be reset to default."
I suppose if you are using Windows, DOS or Linux on the bare metal that may be ok, but for me anyways running vSphere there are (as far as I could tell anyway) no vSphere related tools for IPMI config on supermicro.
At a bare minimum there should be an option where you can at least populate some basic configuration such as network configuration so you can connect to the IPMI after it resets. Hard to believe this situation is unchanged years later. I have had seamless upgrades on HP iLO and Dell iDRAC (used Dell at a company back in 2009-2010 too) on every single attempt, not a single issue over the past ~13 years. Before that I was mostly a SM customer (had a few hundred systems at one point) and firmware updates were basically never applied as the process and documentation was quite scary(I believe often required DOS floppy disk on systems that had no floppy drives, and the remote KVM/virtual media abilities did not exist at the time), and SM themselves warn you not to upgrade anyway(still warn you even today). Their processes and documentation are only marginally better today.
A while back I looked at the IPMI update procedure for Citrix Netscaler, and it was just horrifying https://support.citrix.com/article/CTX137970 ), I believe Citrix uses Supermicro as well. I tried once getting IPMI working on Citrix but ran into a wall pretty quick(I think the certs were the same on every device which caused browsers to freak out, known issue at the time anyway), just use serial console and network PDU.
the IPMI went offline at that point and I didn't have connectivity to it again for another couple of years (next time I went on site
Wait... WHAT?
From within the OS of a running system you can access the attached BMC/IPMI to change the network settings, reboot it, etc. Whatever you need to do.
running a hypervisor on the system
Fair enough, I didn't make it quite that far through the wall of text.
I would hope that it would prevent any client OS running on it from accessing the base hardware? They also mentioned the lack of tools for Vsphere so they have looked into it.
VMWare/ESXi has a "management console" that is basically a Linux VM but with privledged access. On the management console, you can run ipmi configuration utilities. Here's a fellow who shows how to do so with SuperMicro specifically:
https://www.cryptomonkeys.com/2016/12/supermicro-ipmi-reset/
It seems elemental that any organisation would have at least one outer DMZ where all ports are closed by default (block all) and specific rules to open required ports.
If the organisation takes credit card payments in the UK, it’s a PCI requirement and no QSA should sign off a site without checking the actual firewall config not just the policy document.
IDK about this being a huge issue qua security hole.
If a server's BMC is accessible from the public Internet then the server's owner has worse problems than what are described in this article.
This seems like some of the "holes" described last year where the attacker has to achieve root level access to a box to exploit. If an attacker has root on your box then how likely are they to exploit obscure holes rather than just do bad stuff, stop syslog, clean up the logs and restart syslog?
Isnt this a bit like saying "if you leave the door to your safe open people who are allowed into your home and are THEN left alone in the room with your OPEN safe "might" take something from it, therefore you shouldn't buy a safe"....
What makes this even more silly, is how many people have pointed out that server admins might not even realise which servers have these boards in them without hitting network inventory data. Yet "someone" from, I guess CHINA "might" somehow magically arrive in a companies server cluster room and would then somehow magically know what the companies own techies dont and would somehow get physical access to the server to exploit the issue.....
TBH, this just seems like a less moral facet of the US trade war with china, you know, like the BS over the Huawei products JUST after they "coincidentally I am sure...." overtook APPLE in global phone sales with an even better model JUST about to launch.....
What tech companies SHOULD have taken from the Huawei fiasco is that NO company in ANY country is safe as long as it relies on US hardware, software or firmware
ARM needs to move completely out of the US so it can sell its chips where ever the heck it likes and other tech companies around the world need to work on having open source, openly traded equipment with no link to the US
Then the rest of the world can carry on advancing whilst the US clings desperately to its ever decreasing stranglehold on the worlds tech industry
Leave US tech companies ONLY selling in the US market and to its owners in Israel with everyone else selling EVERYWHERE else
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
