back to article Supermicro spy chips, the sequel: It really, really happened, and with bad BIOS and more, insists Bloomberg

Following up on a disputed 2018 claim in its BusinessWeek publication that tiny spy chips were found on Supermicro server motherboards in 2015, Bloomberg on Friday doubled down by asserting that Supermicro's products were targeted by Chinese operatives for over a decade, that US intelligence officials have been aware of this, …

  1. Andre Carneiro

    The fact that Supermicro hasn’t gone down the path of litigation in 2018 is quite interesting in itself, wouldn’t you say?

    1. Throatwarbler Mangrove Silver badge
      Meh

      Having this story stay in the news, substantiated or not, seems like it could be bad for Supermicro; they may consider the risk of negative publicity to be too high, especially with a lot of attention on Chinese vendors at the moment. The last thing they need is to find themselves on the same blocklist as Huawei.

      1. Anonymous Coward
        Anonymous Coward

        It could drop the price and there may be even more ordered worldwide because they are slashing their margins. Can a server phone home if it is behind a three firewalls ?

        1. Wellyboot Silver badge

          If the firewall hardware is compromised at a similar level anything is possible.

          1. Anonymous Coward
            Anonymous Coward

            If anything is possible, let's just assume that all Western governments are actually on the FSB payroll.

            Or maybe they're all lizard people from Alpha Centauri in disguise.

            Or actually, since the universe is a simulation, they're all Agent Smiths.

            Because, hey, why stop at the ridiculously implausible, let's just go one little step further.

          2. William Higinbotham

            https://www.infoworld.com/article/2653167/fbi-worried-as-dod-sold-counterfeit-cisco-gear.html

        2. Orv

          It depends on how careful your filtering is. There are lots of side channels that are commonly allowed by firewalls that can be used to exfiltrate data. ICMP can be used to move data. Data can also be disguised as DNS lookups.

        3. Stoneshop Silver badge
          Holmes

          Piggyback

          If the server has legitimate outgoing connections, a bit of steganography can piggyback the data to be exfiltrated onto them. It would the target (proxy) server(s) in a less-restricted area to also be compromised in some way, to be able to split off the steganographic load and forward it to Outer Elbonia.

        4. Clive Galway

          If the firewalls only block inbound connections, then sure, 3 or a million, makes no difference

        5. Claptrap314 Silver badge

          Maybe, but if you use seven, then there is no way you can be traced...

      2. low_resolution_foxxes Silver badge

        I suppose you neef to consider plausible reasons why Bloomberg would report this:

        1) it really happened, so it should be well documented and reported to the authorities with a response (it is not)

        2) PR sabotage (for a competitor)

        3) Short attack (plausible given Bloomberg's core finance trading market), especially considering the key US rivals with major stock MCAPs

        4) supermicro may not be cooperating fully with the NSA/GCHQ, this is a warning slap they should play ball

        5) legitimate concerns that Supermicro are run by several Taiwan executives and make all their products in China, and would be potentially open to Chinese govt bribes/access

        6) Supermicro involved with AI/cloud, known areas US and NSA want to keep an eye on

        7) they accepted some issues with dodgy revenue recognition in their accounts (daft trivial stuff really, like the shipment was delayed 5 days but they booked the revenue in the last month of the quarter)

        8) overall they seem trapped in the middle of the US-China trade war. They have apparently agreed to move all their manufacturing to Taiwan to make this problem disappear, perhaps even moving some chip manufacture to the USA (Trump did push these kinds of policies to encourage US jobs).

        1. Sanctimonious Prick
          Coat

          I guess #4

          1. Jaybus

            Maybe, but #5 is a good guess too.

      3. RM Myers Silver badge
        FAIL

        especially with a lot of attention on Chinese vendors

        Supermicro is a California based company. The last time I checked, California was still in the United States

        1. Old Used Programmer Silver badge

          Re: especially with a lot of attention on Chinese vendors

          Probably not if you are a true member of the trumpen proletariat.

          1. RM Myers Silver badge

            Re: especially with a lot of attention on Chinese vendors

            Well, I have heard of people who want to split California into multiple states, and people who want to make California into its own country, but I hadn't heard of anyone who wanted to make California part of China (okay, Pooh Bear may be an exception).

        2. low_resolution_foxxes Silver badge

          Re: especially with a lot of attention on Chinese vendors

          Yes, but the chips are mainly manufactured in China.

          1. RM Myers Silver badge

            Re: especially with a lot of attention on Chinese vendors

            I was referring to Throatwarbler's post in the thread, where they speculated that Supermicro might be blocked like Huawei. If US companies who have manufacturing in China can be blocked, Apple better start worrying.

    2. A Non e-mouse Silver badge

      Proving a negative is very hard.

      1. MiguelC Silver badge

        Well, if I'm accused by Bloomberg of eating babies for breakfast and I sue for libel, it's up to them to show some evidence that I really ate babies for breakfast....

        1. Malcolm Weir

          Not at all. It's up to them to show that they had reason to believe you really ate babies. Subtle distinction, but they don't need evidence of the eating, just evidence supporting the reporting of the eating ("Reliable source Fred -- who's a spook, so we can't divulge his name -- says you ate babies").

          But even worse, this isn't a case of someone accusing you of eating babies, so much as someone saying your babies are being eaten by The Bad Guys. Supermicro isn't being accused of malfeasance, China is (and Supermicro is the victim).

          1. This post has been deleted by its author

      2. A.P. Veening Silver badge

        True, but forcing Bloomberg to prove a positive shouldn't be that hard (if Bloomberg can, of course).

      3. DJ
        Mushroom

        Prove it.

    3. Malcolm Weir

      No. It's very hard to prove the "negative" that there _weren't_ spy chips added. And all Bloomberg has to do is say they had reason to believe that there reporting was true, and they're then off the hook even if it was false! Remember, the gist of the story is that The Bad Guys (aka "China") maliciously added the chips without Supermicro's knowledge, so where's Supermicro's beef?

      1. A.P. Veening Silver badge

        But Bloomberg is making the allegation, so it is up to Bloomberg to prove/show a positive.

        1. John Robson Silver badge

          US vs UK law...

          - In the US, if someone accuses you of lying about them in print and sues you, they need to prove that what you said was false.

          - UK libel law reverses the burden of proof: when suing someone for libel, it's up to the defendant to prove that what they said was true.

          So if the publication happened online, could it not be said to have been published in the UK, and therefore they should sue in the UK, which places the burden of proof on the accused, not the (allegedly) libelled.

          It is basically impossible to prove that no SM shipments have ever been intercepted and had a spy ship added.

          1. Imhotep Silver badge

            "In the US, if someone accuses you of lying about them in print and sues you, they need to prove that what you said was false."

            Since you added the "in print" qualifier, I would add that the standard is much higher than simply proving something was false. That is why almost no one can successfully sue the media in the US.

    4. Orv

      They could be trying to avoid the Streisand Effect.

    5. thames

      Bloomberg were laughed at throughout the technical press in 2018, even if the general media took them at face value. The one person they were willing to cite as a "source" was interviewed by a security podcast and said that he didn't find Bloomberg's claims credible and that they had misrepresented what he had said. They were pretty comprehensively discredited back then, at least among the segment of the audience who would form Supermicro's customer base. There wasn't a lot of reason for Supermicro to sue then.

      Since 2018 not a single solid piece of evidence has emerged to support Bloomberg's claims. Back in 2018 Bloomberg relied on second hand information from various unnamed officials in Washington and their one named source said he had been misrepresented and that he didn't believe Bloomberg's story.

      This time their story is once again various unnamed officials in Washington, and their one named source just turns out to be someone who heard the same second hand story they did. In other words, it's just 2018 all over again with even less evidence this time.

      Where's all these Supermicro boards that have hidden chips in them? If they're out there and been discovered, why aren't we hearing any first hand reports? If the US really had any of these boards they would be holding the biggest IT security press event of the decade with one, rather than feeding stories anonymously to compliant reporters. They're not shy about showing off Russian rootkits to the public, so where's the evidence in this case?

      I'll believe it when I actually see some credible evidence in the hands of a neutral party who has a convincing chain of custody for it. Until then, I'll just put it down as Bloomberg being their usual less than credible selves again.

    6. Anonymous Coward
      Anonymous Coward

      Never pick a fight...

      I thought the quote "Never pick a fight with a man who buys ink by the barrel and paper by the ton" was from Mark Twain, but the internet disagrees. Regardless of origin, it's pretty apposite in the circumstances.

  2. aregross

    And what about Huawei?

    I've wondered ever since Huawei was singled-out as another baddie.... Where was *any* evidence about them spying? I've looked and looked and waited and waited (expecting El Reg to come out with it first) for any sort of smoking gun to say "Look Look, See Here, this is what they've done" but nothing.

    Where did the Huawei story start in the first place? If there's *any* evidence to show, why hasn't some news site revealed it?

    1. low_resolution_foxxes Silver badge

      Re: And what about Huawei?

      Huawei overtook America's largest company (Apple) in sales revenue and were promptly put on the ban list. Xiaomi recently achieved the same feat, within weeks the US put them on the naughty step.

      Presumably Huawei probably doesn't install NSA back doors to let the US spy on Chinese citizens. Hence, by twisted logic, Huawei are a threat to US intelligence and security,because they don't handover user data.

      1. find users who cut cat tail

        Re: And what about Huawei?

        > Presumably Huawei probably doesn't install NSA back doors to let the US spy on Chinese citizens.

        They can't. There is no space left. Already too many Chinese backdoors there…

    2. teknopaul Silver badge

      Re: And what about Huawei?

      Agree. I highly doubt that Huawei has done anything spy related that was not mandated by a govt, either us or prc or whoever has jurisdiction.

      You cant blame the soldiers for the war.

      & PRC would have a good argument that they were just defending themselves if they dropped a bomb on Florida.

    3. khjohansen

      Re: And what about Huawei?

      My two bits - Huawei were getting into Backbone Networking (5G hardware) in a BIG way - thusly they were a) taking market shares from Cisco et. al. b) denying NSA/GCHQ/5eyes access (allegedly!) c) p***ing down the leg of "western tech supremacy" ...

  3. DS999 Silver badge

    This is so stupid

    If it was real there would be publicly available videos of the boards showing them being torn down and the spy chip removed. They can't simultaneously claim both that the hacked boards were widespread and expect us to accept zero public proof.

    1. A Non e-mouse Silver badge

      Re: This is so stupid

      Extraordinary claims require extraordinary proof.

      The problem is that the NSA have been caught with their hands in the cookie jar so the burden of proof has come down.

      1. Doctor Syntax Silver badge

        Re: This is so stupid

        Would it be too uncharitable to wonder if there's another story about to break with NSA's hands in the cookie jar and they need a create a distraction?

        1. low_resolution_foxxes Silver badge

          Re: This is so stupid

          I dunno, NSA's been on the end of a hiding since they accidentally leaked all their 0-days to Russia and the malware scene.

    2. Malcolm Weir

      Re: This is so stupid

      Yep... And remember the last go around had the servers in an Apple-owned data center, and the comment from the Apple IT people about how they'd have noticed traffic -- the alleged traffic -- because they don't put the IPMI/management NICs on the same network as the application NICs, and they monitor their management network.

      Which sounds, you know, professional.

      True story: I was overseas when I got an urgent call from our cybersecurity consultants... they'd detected traffic from our internal network heading to an Indian IP address. The FBI were alerted. Tension was rising.... until I pointed out that the IP address in question was part of a block that we'd sold a couple of years previously, and in fact this was only an ancient desktop PC trying to reach a long-decommissioned print server whose address was now allocated to an Indian telco!

      (And this, people, is why NAT is your friend: using a public IP address range for an internal network is a Bad Idea. See also IPv6...)

      1. wwwd

        Re: This is so stupid

        Oh my.

        NAT is NOT yours or anyone's friend, it fundamentally breaks the end to end communication model of IP. Using a "public"range for an internal network is NOT a bad idea, either. Having no stateful security (which some NAT implementations happen to have as a by product of their packet mangling) is a bad idea.

    3. Cuddles Silver badge

      Re: This is so stupid

      Indeed. It's particularly telling that all the support for the claims comes from security experts saying that it's plausible that something like this could be possible, or how they've seen and studied other examples of hardware compromise in the past. Because yes, attacks like this are technically posisble, and examples have actually happened. But the thing about those similar attacks that have been previously seen and studied is that they have, in fact, been seen and studied.

      Bloomberg aren't claiming that this is a theoretical attack route that someone could be exploiting. They're claiming to have seen the actual physical chips used to do it. They're claiming that multiple different organisations, private, governmental and spy agencies, have all detected the activity and investigated the actual hardware, over a span of many years. It's not enough to show us some unrelated expert who says it might be possible to install a chip somewhere. You claim that lots of people have seen the actual hardware used to do this, so where is it? Put up or shut up.

  4. David Pearce

    Absent evidence

    The Trump administration would have been so pleased to show proof that China was spying and never did.

    1. Robert Carnegie Silver badge

      Re: Absent evidence

      It's rather broad, and implausible, to say that Chins never ever spies at all.

      But as for the Supermicro business, I thought I remembered that there was indeed photographic evidence, of a tiny spot on a circuit board that was alleged to be a microscopic chip unconnected to the rest of the computer, that nevertheless would send your private data all the way to China by itself, sneering at your airgaps.

      Why this amazing chip wasn't being used to make cellphones work as well as that, remained unexplored.

      It probably was a blob of glue or something.

      The key insight is that the internet is s series of tubes. If you don't understand that then you don't understand anything.

    2. Anonymous Coward
      Anonymous Coward

      Re: Absent evidence

      Trump and evidence is an oxymoron. Whatever he says must be true - according to him and his followers. No need for facts.

  5. Anonymous Coward
    Anonymous Coward

    The impossible bus

    Yeh, you have a tiny chip that intercepts data and lets them compromise the running software.

    So all it needs is 64 data pins in, 64 data pins out and the address pins in and out, and it intercepts the data without slowing down the processing, so it must get mighty hot, better slap a big heat sink on it, oh and it has network access, an RF shield.... illustrated with a picture of an 8 pin chip.

    As I recall, it wasn't the lack of named sources that made it impossible, it was the impossible nature of the claim. Specifically the intercept claims required a big bus between RAM and the processor, that would be very noticeable complete with a mass of tracks on the circuit board. Can you imagine the QA inspector not noticing that the circuit board doesn't match the scheme they're inspecting against? Can you imagine the guy checking the cooling doesn't notice this hot chip?

    So now its back.

    Add Mukul Kumar, saying vague concepts as to how you might go about it, doesn't pass muster either. Describing how you *might* set about things while simultaneously claiming you've been shown the chip itself in an unclassified briefing....

    So concrete:

    "I HAVE physically held evidence in my hands,"

    Turns into a hypothetical 'if':

    "IF you create something that should not be there, the x-ray inspector can discover it," he said. "But IF it's changing the shape and size of existing silicon slightly, that's harder to inspect. It's not difficult to add a little circuitry and have no material difference in observability."

    Sure whatever Bloomberg.

    1. Anonymous Coward
      Anonymous Coward

      Re: The impossible bus

      Unless it was IN the chip, piggybacking (or rather, belly-riding) on existing circuits, too small and slight for x-ray to detect, and switched after the QA check while in transit...

      1. Anonymous Coward
        Anonymous Coward

        Re: The impossible bus

        Which, "in the chip" or piggybacked "under the chip"?

        Which chip has all the necessary access and yet isn't mounted with solder-flow pads so you can layer under it?

        "too small" small and yet with all these connections into and out of it.... you mean thin?

        It all smells a bit "PR copywriter" to me. Still does.

        Googling Mukul Kumar name doesn't reassure me either.

        1. Anonymous Coward
          Anonymous Coward

          Re: The impossible bus

          Whatever happened to switching out the chip with a compromised one that looks the same and has the same connections? As for where, a network controller is the most likely place, allowing very low level subversion. Given processing demands today, a high-end network controller likely has direct memory access and plenty of computational oomph.

          1. Def Silver badge

            Re: The impossible bus

            An even easier (for a given value of easier) option would be to hijack the chip design during the development stage.

            If company X decides to create a new product which requires new IC components to be designed and manufactured, at some point those designs need to be send to a foundry for production. As soon as the designs are received they could be forwarded to nefarious types to tinker with. Send back engineering samples to the original specification, but swap in a modified design when the chip goes into production.

            If new minor revisions are submitted, pump out those for a few months until the changes can be updated - if necessary - and then continue pumping out the hacked designs.

            How many processor designs are actually verified after production has started? Checked for manufacturing defects, sure. But not the designs, I bet.

            However unlikely it maybe, it's not an impossible scenario.

            1. Anonymous Coward
              Anonymous Coward

              Re: The impossible bus

              Clearly no understanding of ASIC design/verification/manufacture/test. Or are you just making up fan-fiction for your next 8chan/kun drop?

              1. Def Silver badge

                Re: The impossible bus

                And yet instead of taking the opportunity to educate and correct me you decided to take the infantile route with condescending and snide comments. Well done. Your mother must be very proud.

                1. Anonymous Coward
                  Anonymous Coward

                  Re: The impossible bus

                  And there's the problem today: People with zero knowledge willing to create baseless conspiracy theories and post them on the internet.

                  1. Def Silver badge

                    Re: The impossible bus

                    If you think what I posted was "a baseless conspiracy theory", you need to get out more. (Or possibly stop reading 8chan/breitbart/et al yourself.) What I wrote was a simple hypothesis postulating a possible scenario about how something could happen.

                    At no time did I claim it was what actually happened, nor did I try to claim some deep insider knowledge or secret source as to where this "theory" came from. I don't know if you noticed, but I also completely failed to reference the shape of the earth, the anonymity of Q, the 5G/Corona connection, or any other random nonsense found elsewhere on the Internet.

            2. Antron Argaiv Silver badge
              Black Helicopters

              Re: The impossible bus

              Isn't here already an Intel Management Engine waiting to be used for nefarious purposes?

          2. Anonymous Coward
            Anonymous Coward

            Re: The impossible bus

            Just a reminder here..... Bloomberg original story complete with *external* tiny chip:

            https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-china-used-a-tiny-chip-to-infiltrate-america-s-top-companies

          3. Anonymous Coward
            Anonymous Coward

            Re: The impossible bus

            You mean changing the address of the server that the existing backdoor communicates with

          4. Anonymous Coward
            Anonymous Coward

            Re: The impossible bus

            Even considering that it's far from trivial to create a chip that looks exactly identical to the real one - with added circuitry, its own CPU, memory, both ROM and RAM to run its own software, while keeping up to the same specifications of the real one...

            Spying on the network level would give you a bunch of useless encrypted packets. If that's what you want, for some reason, it'd be much easier to plug something outside of the premises, on the cables.

            Because DMA does not mean DMA to /all/ of the system memory. And even if having access to /all/ of the system memory, modern OSes randomize their memory allocation. And on a modern system with memory amounts counted in hundred of GBs, or TBs, that tiny, invisible chip would not have instantaneous access to all of it, at once, and be able to pick and send only the juicy bits out. And copying the whole of it would not exactly be impossible to detect.

            And don't forget: it's not *one* system doing that, according to Bloomberg. It's *all servers produced by Supermicro*.

            I can accept the existence of a targeted device installed on a handful of subverted servers, even more easily since the CIA did it, But all of them? Most of the traffic on the internet now would be Supermicro servers dumping their memory to China 24/7.

    2. EveryTime

      Re: The impossible bus

      It only takes a few pins for a serial flash, and that's the type of flash used on a management processor.

      But there is no evidence that the modification ever took place!

      I could have come up with a much more credible story than the Bloomberg story e.g. using direct editing of Gerbers to support a COB/DCA modification under an existing component.

      But, again, there is no evidence that any modification ever took place!

    3. vtcodger Silver badge

      Re: The impossible bus

      Can you imagine the QA inspector not noticing that the circuit board doesn't match the scheme they're inspecting against?

      I know that's a rhetorical question. And there is some validity. But the short answer is "Yes, I can imagine that." Assuming that there actually is a QA inspector visually examining the output, they are probably looking for missing or improperly aligned components, gaping cracks, solder blobs and the like, not stuff that looks proper. As long as the board passes functional tests, my guess would be that it ships. (And yes, I've worked in places with serious hardware QA).

      1. Loyal Commenter Silver badge

        Re: The impossible bus

        My understanding is that it passed the QA, because they slipped an extra component into a multi-layered PCB. Reading between the lines, that's another chip underneath an existing chip. How is QA going to spot that? Especially if it's surface-mounted and there's no tracks showing on the other side of the board.

        Let's also bear in mind, that if this has actually taken place, the "bad actors" in this case are almost certainly backed by the Chinese state, so won't exactly be short of resources to make sure such things are well hidden.

        It's not going to be like chipping a PS1.

    4. naive Silver badge

      Re: The impossible bus

      It is an interesting thought experiment to design such an evil "chip", since it won't be a 74LS00 tacked on somewhere. Given that the system will be running Windows, Linux or VmWare in most cases, how can a system be inserted which is capable to a) Spy on OS memory and b) Send interesting data using the correct ethernet interface controlled by the OS to some secret data collection server.

      Such a chip should be somewhere in either the PCI controller path or the memory management module, since it should be able to control the bus, to collect and send data. To make it work on an existing MB design, it could be possible to insert PCI and memory controller chips which have a complete OS themselves to perform these tasks. Even when such modified standard components can be inserted into the assembly line, it is still a challenge to extract valuable information from a server with 256GB memory. That could only work if the OS itself is modified to assist in this. Virtualization poses also a hurdle, since VM's do not have direct hardware access.

      1. doublelayer Silver badge

        Re: The impossible bus

        If I was designing something like this, my first attempt would probably be to compromise the firmware or bootloader. If I rewrote the default image to include a rootkit and modified the chip slightly to only start using it after a certain number of boot cycles, that might work. It would have access to most of the hardware, but it would be memory-resident in such a way that the running OS wouldn't detect it. It's not perfect, especially if there are updates to the firmware which this has to handle somehow, but it could function while avoiding a QA scan--a slightly modified chip which was already there and the first cycles are using the correct firmware image.

    5. Pete B

      Re: The impossible bus

      Can you imagine the QA inspector not noticing that the circuit board doesn't match the scheme they're inspecting against?

      I can well imagine our QA inspectors not noticing if the board was actually a turnip, based on some of the "passed QA", "Returned Faulty" products I've seen.

  6. Neil Barnes Silver badge

    From a position of complete ignorance

    It strikes me that if we're speaking x86 parts here, isn't messing with the Intel Management Engine the way to do it with no obvious physical addenda to the circuit board?

    Like others, I have a hard time believing in the need to supply extra chips to the PCB with all the attendant risk of discovery, when Intel have kindly provided exactly the claimed abilities built into the system by design.

    1. Adrian 4 Silver badge

      Re: From a position of complete ignorance

      And quite likely with some of its code provided by the by the US agencies

      Or a backdoor that got owned. As we constantly warn might happen.

    2. Anonymous Coward
      Anonymous Coward

      Re: From a position of complete ignorance

      Intel chips are neither invisible nor omnipotent, so simply don't fit the Bloomberg description,

      1. Claptrap314 Silver badge
        Pint

        Re: From a position of complete ignorance

        Think of the keyboards!

  7. YetAnotherJoeBlow Bronze badge

    No silly, that is not Chinese espionage, that is just Intel's ME software...

    1. Anonymous Coward
      Anonymous Coward

      https://libreboot.org/faq.html#amd

      https://libreboot.org/faq.html#intel

      Summary: A backdoor with total access to and control over the rest of the PC. The ME/PSP is a threat to freedom, security, and privacy, and the libreboot project strongly recommends avoiding it entirely.

      1. Charles 9 Silver badge

        The problem is, that's NOT what corporate bigwigs want (they want control for themselves), and they're the ones making the big orders...

      2. needmorehare
        Facepalm

        Not a backdoor

        Intel ME provides IPMI and DRM services for desktops. That's like saying HP iLO, Dell iDRAC or Supermicro SIM are backdoors. These are all features so that people like me can monitor the health of the hardware, add security checks outside of the control of the operating system and remediate problems remotely.

        You can disable Intel ME entirely and disable the ME coprocessor if it suits your security model, just as you can use server hardware without IPMI features.

        1. whitepines
          Alert

          Re: Not a backdoor

          You can disable Intel ME entirely and disable the ME coprocessor if it suits your security model, just as you can use server hardware without IPMI features.

          NO. No you can't. That's the whole problem. AMD has their PSP, Intel has their ME, ARM makers usually go one step further and lock the entire boot process on chips like Snapdragon, leaving a couple of alternatives like Power and RISC-V, and not much else.

          What I find most interesting is the claim that Intel's network was invaded. That could range from anything to accessing a few employee workstations to stealing the keys to the kingdom (the ME signing keys), at which point a grain of rice type flash device could absolutely hijack the Intel ME, much like the Solarwinds mess.

          If the (theoretical) attackers then used standard HTTPS traffic with a hard-coded range of IPs, the malware could probably communicate through most firewalls, especially in a Windows environment where activation requires this kind of communication to be allowed in the first place. The ME is more than powerful enough to support this kind of advanced malware...

          1. Anonymous Coward
            Anonymous Coward

            Re: Not a backdoor

            There are plenty of secure environment out there which are not connected to the internet and where even Windows does not need to call home to be activated. Those environments would detect one of their servers trying to connect to the outside world. In other words: there would be evidence.

            1. Anonymous Coward
              Anonymous Coward

              Re: Not a backdoor

              Then how does it protect itself against security exploits that get discovered later on? Or are they simply left vulnerable to evil tech attacks?

              1. Orv

                Re: Not a backdoor

                The usual practice in air-gapped networks is to use physical media to transfer updates. Of course this has to be carefully done to make sure no malware comes along for the ride.

              2. ckm5

                Re: Not a backdoor

                The are not usually updated as places like what the OP is describing consider all electronics vulnerable and isolate them completely from pretty much everything.

  8. Anonymous Coward
    Anonymous Coward

    hello kettle meet pot

    Is this really any different than AMD's PSP or Intel's ME, a processor running on almost all x86/x64 based computers worldwide that can not be audited. One FISA court order from the NSA and a firmware update later and full access to the full contents RAM of on a machine is possible.

    1. mevets Bronze badge

      Re: hello kettle meet pot

      Lets say I can remotely trigger a ramdump; around 256G on a medium server. Now what? Send 256G across their network, and hope that nobody notices? How does the trigger-er know when the right moment to snap is? It could take many thousands of snapshots to find the info you want; so 100s of T of undetected network traffic..... There are much more precise ways to spy than that.

      On the other hand, if I could use my nefarious little circuit for some other purpose, like a general outage.

      Through {some magic} I could trigger a widespread outage so I could rob a bank or something. Yes that is the plot of 1969's The Italian Job. But is far more feasible application.

  9. 45RPM Silver badge

    I don’t know what the truth of this story is but Bloomberg isn’t exactly the most reliable news agency out there. It’s slightly above The Sunday Sport - but it’s marginal.

    Wake me up when a reliable agency covers this story - The FT, Reuters, The Guardian, BBC - or even ElReg covering the story directly rather than the story about the story.

    1. John Brown (no body) Silver badge
      Coat

      "Wake me up when a reliable agency covers this story....or even ElReg"

      Oooooh....Someone likes to live dangerously! The implication that El Reg is not a reliable agency could be an account losing comment :-)

      1. 45RPM Silver badge

        I think that ElReg enjoys its RedTop masthead, and all that it carries by way of implications of unreliability.

        It’s not my fault if ElReg frequently breaks the contract implied by the colour of its masthead and indulges in good quality journalism.

  10. alain williams Silver badge

    Rather than looking for dodgy chips ...

    or altered BIOS why not monitor IP packets coming from the machine ? If it is to talk to its spy masters how else will the machine do so ?

    Most have firewalls that filter incoming traffic, but a really high value site should also look at outgoing packets - so this should have been caught.

    The trouble with this sort of story is that you cannot trust what anyone is saying.

    1. Charles 9 Silver badge

      Re: Rather than looking for dodgy chips ...

      Defeated by multi-hosting. Combined with encrypted packets, destination filtering becomes useless by way of hiding behind an IP set that also happens to house legitimate traffic. The evil traffic gets lost in the nouse.

      1. Ken Moorhouse Silver badge

        Re: The evil traffic gets lost in the noise.

        The generator of the evil traffic would have to be canny enough to determine that there was a substantial amount of noise to slipstream into, and to remain silent otherwise.

        To counter that presumably a test-bed consisting of a suitably noisy network could be benchmarked for quiescent traffic and this filtered out before the server was introduced into it. Anything remaining would be due to the server's presence.

    2. Anonymous Coward
      Anonymous Coward

      Re: Rather than looking for dodgy chips ...

      I work at a high security site, we block all communications in and out, only allowing a select set of servers to receive and transmit out of our network. We also have higher security zones with zero Internet access, zero remote access.

      We have some supermicro systems of various ages. These systems do not communicate outside our network, zero communications outbound from them have been detected to the Internet, they also run Linux.

      Other systems that, some dells for example, that we have, also running Windows, have 1000's of blocked attempts from the system to connect to the Internet, to Microsoft / azure (telemetry).

      1. Alumoi Silver badge
        Joke

        Re: Rather than looking for dodgy chips ...

        Yeah, but Microsoft are the good guys, and we need them to be aware of what our computers are doing in order to preserve us from terrorists/child molesters/boogieman-of-the-day.

      2. Malcolm Weir

        Re: Rather than looking for dodgy chips ...

        This, this, a thousand times this!

        Competent security professionals don't rely on "oh, well, we bought it from someone we like", they rely on monitoring and architecture. Don't trust anyone... its not exceptionally hard to do!

        (The hard part is usually figuring out who to trust -- "trust everyone " and "trust no-one" is fairly easy.)

    3. Danny 2 Silver badge

      Re: Rather than looking for dodgy chips ...

      There's a guy works down the chip shop swears he's Elvis...

      but he's a liar and I'm not sure about you.

      1. yetanotheraoc

        Re: Rather than looking for dodgy chips ...

        If you buy enough chips at the chip shop, one of them will look like Elvis.

        Although I've also been told it's not true, and people who think they see Elvis actually suffer from Pareidolia.

  11. Ken Moorhouse Silver badge

    Collusion

    If this story is true then - as I see Alain Williams has recently pointed out - there would need to be collusion with other manufacturers in the chain.

    Are they saying that all Router manufacturers are part of the clique?

    Does Wireshark and its predecessors have code in it which turned a blind eye to these packets? Surely packets of one form or another (not necessarily IP structured) has to be the vehicle for transmission of this data through the wires to its supposed perpetrators. Even long-latency steganographic techniques should be detectable within the time-frame being discussed.

    1. Version 1.0 Silver badge

      Re: Collusion

      Sure, but "collusion" with other manufacturers could simply be manipulation of other manufacturers. So we're noticed this "hack" but how many others are there that we haven't discovered? The simple answer is that nobody knows for sure ...

    2. Wellyboot Silver badge

      Re: Collusion

      Compromised hardware will beat any software working above layer 1. The difficult bit is managing to get the compromised hardware installed at every step in the path where packets may be inspected.

      1. whitepines
        Devil

        Re: Collusion

        Compromised hardware will beat any software working above layer 1. The difficult bit is managing to get the compromised hardware installed at every step in the path where packets may be inspected.

        With HTTPS for Microsoft services being strongly encouraged in Windows environments, and outside of special SSL MITM setups that may violate various laws (e.g. EU privacy laws I think prevent this), there is already a tremendous amount of uninspected traffic passed through an average corporate firewall. A few packets for the backdoor here and there in that sea of opaque traffic will quite frankly go unnoticed. Even in a MITMed environment, a smart backdoor would see that MITMing during the SSL handshake and immediately deactivate itself, making detection even less likely.

        1. Anonymous Coward
          Anonymous Coward

          Re: Collusion

          "Uninspected" does not mean "undetected". And you seem to be assuming a level of opacity here that simply does not exist on the secure environments that would be prime targets.

          Secure servers' traffic is not sent to the internet from the same network as what Bill from Accounting uses to update his Facebook page. The other systems they communicate with are known and identified. And even if you can't see the payload inside a TLS connection, the remote server identity (assuming the network is somehow connected openly to the internet, so not terribly secure in the first place) is visible by its certificate during the TLS handshake. So it's not as invisible as you think it is when border controls are properly set-up.

          1. whitepines

            Re: Collusion

            is visible by its certificate during the TLS handshake.

            On TLS 1.2 and earlier, yes, assuming SNI isn't used. TLS 1.3 fixes this.

            To be very clear I'm not saying the Bloomberg allegations are correct. I'm simply pointing out the fact that it not easy to detect this type of malware, and especially so if there is any kind of targeting of the supply chain attack (to weaker security environments) or intelligence in the payload (i.e. don't activate unless other SSL traffic spotted on the network).

  12. Steve Graham

    Coincidentally, the original "confidential briefing" given to Bloomberg by the three-letter agency (allegedly) occurred just after a large budget cut made to projects investigating Chinese electronic skullduggery.

  13. Anonymous Coward
    Anonymous Coward

    I have it on good authority that my local fox is a bit shady. I don't have any actual evidence but 50 cats who did not want to be named saw the fox doing something shady though I can't tell you exactly where it was or provide proof of this but these cats were part of the neighbourhood fox watch so I think they can be trusted on this information. I'm also not sure why none of these 50 cats have not leaked the information of where or how the fox was being shady. You just have to take my word on all this and at this current time I can't offer any help with your hen houses.

    1. TimMaher Silver badge
      Pint

      Management problem.

      Have you tried herding 50 cats?

      Impossible.

      There’s a beer for the fox ————->

  14. Tim99 Silver badge
    Trollface

    It's probably too hard

    Now that systemd is nearly an OS, just upload a modified networkd to GitHub, no one will ever find it. The snag: Spooks being able to find someone who understands unmodified systemd...

  15. Trigun Bronze badge

    Well this is a bit annoying. My R&D server at home is a super Micro from 2010-ish. It could be be reporting on my network to the Chineese government!!!! It might also explain why there has been an epidemic of narcolepsy over there :D.

  16. major_paine
    FAIL

    Chip or no chip, allowing servers in a DC to talk to the outside world is just bad practice

    Talking to the outside world is a carefully controlled process and it's performed by dedicated hardened gateways which include TPM (look it up). What's more.. any deployment classified Impact Level 4 (in the old classification, I dislike the new one) or above requires air tight architectures which in many cases must use diode-type gateways which as the name states, only allow traffic one way. This is why security experts also mix vendors for the different perimeters.

    *takes architect hat off*

  17. StrangerHereMyself Bronze badge

    Suprised

    I'm somewhat surprised that the U.S. didn't hit back at China by banning the Chinese manufacture of components used in DoD or Federal agencies

    Instead they're waiting to see China's hacking capabilities? I'd have thought that they'd already demonstrated their supply chain infiltration capabilities.

    China needs to be held accountable for this by quietly pressuring manufacturers to leave the country and to set up shop somewhere else in Asia.

    1. A.P. Veening Silver badge

      Re: Suprised

      China needs to be held accountable for this by quietly pressuring manufacturers to leave the country and to set up shop somewhere else in Asia.

      Where in Asia? Most of it is China's back garden.

      1. StrangerHereMyself Bronze badge

        Re: Suprised

        Vietnam, Philippines, Laos.

        It may be their backyard, but it's not THEIR yard.

        1. Malcolm Weir

          Re: Suprised

          Don't forget Taiwan, which has the advantage that they don't like the PRC much...

  18. Anonymous Coward
    Anonymous Coward

    Trust?....You need to trust YOUR OWN technology!

    If users who depend on unknown technology (including SuperMicro) were to use their own private cipher before ANY messaging enters the channel......then all of this "embedded chip" problem IS MOOT.

    *

    DON'T TRUST ANY CHANNEL!

    *

    As with the message below for Jeremy Fleming and Ben Wallace:

    *

    kDgdGRaveb4boVS76NUfuPal01aHaXQvePIFKL0Z

    UvgFqZuJmpMRiv6BM9CJCvMzeViHANifyDYxmTyT

    QBYxUlEtCb2DExWfSz0VaRoJ8PWRupOPGZqJYxQh

    QHcp85av0T2rYdmJcXOhuBKLmVKP6RatMzw5Mx0v

    o9uRyjQjWxq3ORsVKj6XKbAVgN6Dq1i96pu9aZGR

    ULu5o5mDOpG7qFS3e7cDeNQhuz0bK5qJaxcTSh2b

    ET2Du5c9QJ0B69UDe3Y7SXAxYdgPgdwhKnM5OTS9

    WtoJcJq7CJoZWtOpKtEVizMLUBQHIlwFS1GFgVaZ

    efkxG9gl0nCrsNItUtspSLaX4Lglshoh0L25498Z

    glGxATaZmr0duXKvMrclSFqvu1U5YNar85MpkpcP

    gDo1e9yNKniv89ifu5szqb01QbGZ0zM9kxUNUx4p

    w1OhYFo1Ipqpun6LqTkzezSHCBi7Uti5svixcHER

    otSrsj4x03KNkLGRaHaNkF6d2joXwnKBsHyX4FG7

    65mnybizCn29ab8Fe1AL0Norcj2h2zenQbaZOjcV

    83ijcnUNWnGNsfKDybkpyTsvkjYZ

    *

    1. doublelayer Silver badge

      Re: Trust?....You need to trust YOUR OWN technology!

      We don't think the chip happened, but if it did, cryptography doesn't fix anything. The theoretical chip would attack the source, not the network. It would take the unencrypted contents of memory and report that, which means your cipher, good or bad, would not affect it in the slightest. Please pay attention to where in the chain an attack happens before you declare thee cipher to be the panacea for any and all security problems. Unless you plan that everyone will manually encrypt their messages on paper before laboriously typing it in, in which case I'd encourage you to stay in the 1950s where nobody needs to send anything but short text messages.

      1. Anonymous Coward
        Anonymous Coward

        Re: Trust?....You need to trust YOUR OWN technology!

        Quote: "...manually encrypt their messages on paper..."

        Ever heard of "air gap" or "thumb drive"? Obviously not!

        1. doublelayer Silver badge

          Re: Trust?....You need to trust YOUR OWN technology!

          Again, you're missing the point. A hardware exploit attacks the source, not the network. If the source can be compromised, then the data leaked is the original. If you want to air-gap your machine and physically move data across for transmission, that could work, but it won't fix most of the problems that an embedded chip can create. Your assumption that the only point of a system is to send messages is simplistic. Among other things, do you expect someone to walk back and forth with a thumb drive to encode and decode each step in the process of, say, querying a database?

          Machines which could be compromised usually either are air-gapped already or won't be. The air-gapped ones are usually safe. The ones which aren't are the problem. The solution is monitoring of traffic from devices that have the ability to connect, with encryption being useful but not solving every problem. One good example of a problem it doesn't solve is compromised hardware at the endpoint.

  19. Pascal Monett Silver badge
    Coat

    Pics or it didn't happen

    Bloomberg's credibility is zero as far as I'm concerned. So they're doubling down ? Still no pictures ? It's hookum. Hogwash.

    I will believe this very dubious story if you show me pictures of the modified mainboard, with a bright red arrow pointing to the part that has been added, and specify where you found it.

    It's classified ? How can it be classified and widespread at the same time ?

    I'm tired of this bullshit story.

    Mine's the one with a phone that can take pictures.

  20. Danny Boyd Bronze badge

    Stubbornly stepping on the same rack

    Jeez, Bloomberg thinks it's not enough it beshat itself in 2018, it goes for encore. Do they want to became a common word like "xerox" or "kleenex" as in "this is a crock of bloomberg"?

    No evidence then, no evidence now, just blah-blah. I lost my respect for Bloomberg in 2018 after this load of crap was first aired, and now I see I made no mistake then.

  21. boggles

    I've worked in the hardware industry for over 20 years and visited factories in China and Taiwan many times, often with large Western European customers doing security and quality audits.

    Mainboard production runs are not usually done in tens or hundreds, but many thousands at a time (its not worth the job setup time to do small runs) The factories have the strictest ISO process controls, and much of the QA is automated and performed by computers, not humans. So it would be necessary to hack the process control and QA software system and pay off key members of production staff involved. Making a change to a handful of products mid run would be even more difficult than changing an entire days work of say 5000 boards.

    So lets imagine an entire production run of 10,000 units gets changed, then there is the distribution. The boards end up in finished systems or sold as boards through the distribution channel and potentially some large direct customers. Distributors sell the systems all over the world.......

    ....and after 8 years no-one can provide evidence of one of these "altered" mainboards ?

    The cover up operation for this "hack" must be a bigger conspiracy than the altering of the boards themselves would have been :)

    Personally i think its anti China trade propaganda, with a goal to sow doubt and put pressure on to move more tech manufacturing back to the USA, I guess we will never find out just who is behind it....just another example of fake "news" in our media today.

    I remember the last time this story reared its head, it won multiple awards at DEFCON as the security industry joked about how daft it was......

  22. RLWatkins

    "If you can think of it, there are bad guys already doing it."

    I realize I'm quoting fiction there, but save for the fantasy genres good fiction stands on its plausibility.

    And since we (US) have done it, why assume, or worse, hope, that China wouldn't? After all, they learned the trick from gaffed Cisco routers, among other things, then they learned how to use, and then used, that very backdoor themselves.

    Like Apple Computer Co. so frequently says, "Everyone wants to be us." And we don't stint on the lessons.

  23. Mr Dogshit

    Why Supermicrobe?

    Surely most people buy Dell or HPE?

  24. Starace
    Devil

    Who benefits?

    Seems a bit of a coincidence that the supporting 'evidence' comes from people with affiliations with a company that includes commodity whitebox servers among its products.

    Or do the words 'Altera' and 'a major semiconductor company' not point in a very particular partisan direction?

  25. Anonymous South African Coward Silver badge

    Dang, damned if you do, damned if you don't.

    A good firewall will be able to record outgoing/incoming traffic.

    If you are able to filter traffic per server, you should be able to see which server are making spurious DNS/UDP/whatever traffic to the outside world (traffic which are not generated by user action), and take it from there.

  26. Potemkine! Silver badge

    Show me the ship

    And the analysis made by an independent lab.

  27. Loyal Commenter Silver badge

    Bruce Schneier Featured this on his blog:

    On Saturday:

    https://www.schneier.com/blog/archives/2021/02/chinese-supply-chain-attack-on-computer-systems.html

    Interestingly, when the story first broke in 2018, his take was that, whilst it sounded plausible, he didn't actually believe it. It seems that more has come out since, but details are necessarily scant, because it's spook vs spook stuff.

    Also, just a couple of weeks ago, he had this on his blog:

    https://www.schneier.com/blog/archives/2021/01/injecting-a-backdoor-into-solarwinds-orion.html

    Which appears to be a practical implementation of Ken Thompson's famous backdoored compiler exploit.

    It struck me, when reading both, that one is essentially a hardware analogue of the other - in other words, unless you can personally verify every single stage of the physical process of building hardware, you can trust none of it, just as, unless you can verify every step of the process of building software, including the veracity of the toolchain (and the tools used to build that), then you can't trust any of it either.

    The only way you can find out is either with a microscope (in the case of hardware) or with byte-by-byte examination of the object code (in the case of software). With the complexity these days of both, it really is a case of, unless you already suspect something and have some clues about where to look, such changes are practically invisible to casual inspection.

    This leads me to wonder whether this is actually a use case for "AI" - hardware / object code inspection for anything that looks awry as a pre-pass before humans take a look. You'd have to make sure it was well-trained to keep the false positives down, whilst at the same time avoiding too many false negatives.

    1. Claptrap314 Silver badge

      Re: Bruce Schneier Featured this on his blog:

      The simple answer is, "No, it's not". It's really not that hard to track down "added" hardware.

      Motherboard manufacturing runs on really, really tight margins. The em interference from any added items would have substantial effects. Much more believable to talk about a compromise of the BIOS than this magic chip nonsense.

  28. martinusher Silver badge

    If it was a chip.....

    I was skeptical about Bloomberg's story because it described a 'tiny chip'. It also quotes unnamed security sources that only talk in generalities, the sort of thing that might just fool the general public, politiicans and even IT professionals but not something that would impress engineers with knowledge of hardware, processor boot sequenecs and the like.

    The part was 'the size of a grain of rice' which means that at best it was a serial flash chip of relatively low capacity. Remember, it had to be 'non obvious' so Supermicro could (say) double the boot memory size and then use a hidden part to lose half the image (the other half being the naughty stuff), it would show up in the bill of materials. I could spend all day specualting about what such a part was but unless it was produced then I just have to assume that this entire story was a hoax.

  29. Claptrap314 Silver badge

    Same song...

    Reading the first two pages of comments, I'm seeing quite a bit more comments in favor of this ridiculous story.

    This story is the tech version of anti-vaxxers.

    To summarize: this story is immediately ridiculous to any of the tens of thousands of us who have spent significant amount of time in the field of microprocessor or board design or test. To those with experience, the immediate response is, "who seeded this **** & why?"

    But now, a few years later, it's back. This time with slightly-more-credible-sounding quotes.

    Anti-vax. That is all.

    1. yetanotheraoc

      Re: Same song...

      "To those with experience, the immediate response is, ""who seeded this **** & why?"""

      In the case of fictitious embedded micro-processors, I find the who and why easy to guess. In the case of vaccines, or 5G, or other conspiracy theories, it's not so easy. Maybe it's just a way to keep people mentally off balance, so they are easier to manipulate with some other story when the stakes are real.

  30. CAPS LOCK

    If these devices really existed someone would have found one by now...

    ...and would be going on about it. LOUDLY...

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021