Decoding the Chinese Super Micro super spy-chip super-scandal: What do we know – and who is telling the truth?
Is this going to turn out like the fake MSG scare stories...?
"That Chinese stuff: bad!"
I imagine some people may decide it is beneficial to source hardware from Taiwanese foundries, given that politically, that area is not mainland Chinese, Russian, or American/5-9-14-eyes controlled. Of course, you would still need to ensure that your supply chain has not been subverted in some way, given the example of the NSA and Cisco.
Assuring that your hardware has not been Trojanned is a hard problem, which the vast majority of people don't need to worry about. I have every sympathy for those whose job such assurance is.
So many birds with one
off the record briefing stone. Draw attention from whatever exploits you're making with Intel ME. Prepare the way for "Nice little motherboard business you've got there. Pity if something happened to due to manufacturing off-shore/not co-operating with us. Remember the Super Micro incident".
Does it matter if the Bloomberg report is true or not ?
Maybe it was indeed all orchestrated fake news, the scenario itself is however very probable in case we not aware of potential supply chain issues.
Parties who have to rely on foreign IT components, be it hard or software, should take into consideration to whom they want to be exposed, there are only two or at best three choices, except for Europeans, who do not make much IT gear, so everyone is free to spy on them.
I generally find Bloomberg to be a decent source of biz/finance news. But for IT stuff, they know 2 things, jack and shit.
How do we know this isn't just us Government making up crap because sanctions are not going to work
Perhaps the most surprising thing about all this is learning that there are BMC chips that have their firmware stored on a physically separate chip, with exposed connections between the two, and nobody thought that this might possibly represent a vulnerability.
"layers of lawyers"
Kieran, I love the alliteration, but...EWWWWWW!
What am I missing here?
"A third thing to consider is this: if true, a lot of effort went into this surveillance operation. It's not the sort of thing that would be added to any Super Micro server shipping to any old company – it would be highly targeted to minimize its discovery. If you've bought Super Micro kit, it's very unlikely it has a spy chip in it, we reckon, if the report is correct. Other than Apple and Amazon, the other 30 or so organizations that used allegedly compromised Super Micro boxes included a major bank and government contractors."
How is the contractor/attacker in any way in control of which MBs get sent to which customers. Surely that is entirely in the hands of Super Micro? Are these particular compromised MBs only available to certain customers? If so, how?
Re: What am I missing here?
I think you're missing the section in the Bloomberg piece about how Supermicro is well known for customization...
My analysis is that these attacks were targeted at SKUs specifically known to be of interest, not the general SKUs that you see on the Supermicro website.
One questions seems to have been missed
Why was Elemental targeted?
I haven't even seen anyone question why they would be targeted or what they do.
In fact AWS Elemental are a Cloud and on-premise Video processing, storage and monetisation company - i.e. they are in the TV world.
For the life of me I can't see why China would be that fussed about getting access to what TV show is being played on a random four letter TV station.
Does this mean that the BOFH-on-the-street should be more worried that a fundamentally random company got these motherboards?
Or, I am sure some will say, that it was to get into Amazon - so the Chinese knew that Amazon might want to buy Elemental before they did in order to design and insert components, with all required connectivity, not just the chips, sell them and have them shipped and used before Amazon started their due diligence. That would be impressive!
Re: One questions seems to have been missed
As I remember, elemental had a contract with an American intelligence agency, although I don't think they're still doing that. However, that could have made them a target at the time.
Re: One questions seems to have been missed
For the life of me I can't see why China would be that fussed about getting access to what TV show is being played on a random four letter TV station.
Well, FEDRAMP is hosted largely on AWS, so that isn't exactly a random four letter TV station, it's the primary access point for civilian US federal agencies cloud presence.
As for bloke on the street targeting, I know a number of peers, as well as myself who could be targeted, due to the PRC hack of OPM and the downloading of our security clearance investigation files.
One upside of that is, now, we can get a security clearance in China.
From the course views of chip locations and traces, it's probable these are CMOS wedge devices, pre-pre-execution environment for the BMC, root kitting it at a hardware level, before the BMC CMOS gets loaded. That bypasses checksums, signing, etc.
You'll never buy bullets from China
I agree it's impossible to be certain who is accurate/misled/mistaken/lying through their teeth. Unless you have actual knowledge of this event, first-hand, you are guessing.
But here's the thing: this is an obvious and highly effective means of espionage, for which a highly technicallly capable nation state, one which lacks checks and balances, and which is well motivated to spy upon foreign governments, militaries and corporations, and which is an ever-increasing source of computers and computing components to the rest of the world, is the perfect source. China, in short, has both powerful motive and ample means.
Consider that no one with serious security concerns brings a computing device back from China (or if they do, it's quarantined, stripped, analysed and then incinerated). Consider that China's spyware has been busy for many years siphoning data from western firms and governments. Consider that even consumer grade devices have been found 'phoning home' with personal data from their owners. If any nation could build the necessary hardware into a speck 100-μ on a side, no thicker than a hair, would you seriously bet against China's best? And bear in mind that China is desperate to become the world's next hyperpower, and that arguably only US technology and economic strength could stop them.
Add it all up, and whether the SuperMicro story is total bollox, or 100% true—it hardly matters: of bloody course the Chinese will be trying this kind of trick, and it would be frankly amazing if they haven't already succeeded here and there—and perhaps already on a large scale. How many devices get national security examination or Amazon/Apple level audit? How many ways, how many places, how many disguises could there be for a cleverly designed sequestration/exfiltration nanobug?
Off the cuff, I'd say that for the next few years this battle is already lost. Amazon will probably say anything to deny that its cloud has been compromised, but I remain confident in saying that if you trust vital data to anyone's cloud, you are a fool.
I said a while ago that in due course, nation states and their allies will bring in-house the manufacture of hardware and software for critical components and infrstructure. Expensive as it is, what choice can there be? Soon enough, computing components will be like ordnance. You may make missiles yourself, or you may buy them from the USA; but you'll never buy them from Russia. Or China.
China along with several other nation states are quite capable of dissecting the Intel ME and discovering how to remote control it, no need to modify any hardware
All they need is to brush up on Minix....
no need to modify any hardware
You know, belts AND suspenders.
So they've found the first spy chip.
How many more to go?
What about Quality Control
Having worked in the industry on thing pops out at me. Supposedly security agencies caught this. All production has Quality Control. Visual inspection is done, at least on a spot check basis. One checks to see if there are any parts that have fallen of during manufacturing, shock and vibe. A missing capacitor, no matter how small would be caught. Given that this happens, any extra chip would also be flagged by Quality Control. I could see quality control catching this and forwarding the issue to security agencies, not the other way around.
To add this chip it would have to be to intercept the board in shipment ad replace it, not in manufacturing. This seems like a very unlikely scenario to me and therefore I lean towards Super Micro, Apple, and Amazon statements being more credible.
Re: What about Quality Control
From the Bloomberg report:
"A U.S. official says the government’s probe is still examining whether spies were planted inside Supermicro or other American companies to aid the attack."
QC at Supermicro (or its documentation) would be the number one target for this, I would have thought.
Right at the end of the Businessweek article it says "In the three years since the briefing in McLean, no commercially viable way to detect attacks like the one on Supermicro’s motherboards has emerged—or has looked likely to emerge"
Even based on the little we do know, that is bollocks. Elsewhere in the article they say "the implanted chips were designed to ping anonymous computers on the internet for further instructions, operatives could hack those computers to identify others who’d been affected". So there is a commercially viable way of detecting the chips. Good old-fashioned traffic monitoring.
Sounds more like all those old chain emails that used to go around about viruses that "nobody could detect", encouraging you to forward the email on to as many people as you could to warn them. GCHQ and NSA probably have enough taps on enough lines to do this for us.
The Chinese Have Come a Long, Long Way Since Chairman Mao Encouraged the Establishment ...
of small backyard steel furnaces in every commune and in each urban neighbourhood.
Perhaps Trump should quit bitching about the Chinese stealing US technology and do a bit more stealing of his own, this time Chinese technology.
Are SuperMicro systems going cheap?
I have been thinking about buying a server and sticking it in a CoLo, for offsite backups. Maybe now is the time?
I don't care about the SuperMicro share price -- are their servers going cheap now? I don't care if the Chinese can copy my data -- the rest of the world has it already thanks to my government.
I'm not worried, all they will find on my system are pictures of naked vegetable, phoar look at those onions without their skins on.
No nuke sub schematics here.
If China had the means to design and manufacture such complex "spy chips" then it would also have been able to manufacture lookalike replacements for legitimate chips on the board such as the BMC chip itself. This would not only have been more difficult to detect (as there are no unexpected additions or changes to the board), but far easier to implement involving fewer people. All that would need to happen is to substitute the bogus chip for the real chip as supplied to the manufacturing factory which could happen anywhere along the supply route, or been done via a "black bag" operation substituting the stock in the warehouse of either factory or supplier. Nobody at any factory need be aware of any changes. No highly difficult modification to the PCB layup (putting a chip between fibreglass layers would require a different and completely non-standard manufacturing process for the PCB - everyone working at the PCB factory would know what's going on).
It simply makes no sense that such a highly complex and detectable method involving scores of people would have been used when a simple component substitution would have done the job far better and cheaper with far less probability of detection and no 3rd parties ever needing to know that it has happened.
Re: Occam's Razor
If you're in the business of verifying that the motherboard contains what it should you'd hi the parts off, mill of the packaging and check the die with a verified sample. Perhaps the genius of a component masquerading as a passive, is that nobody would bother.
Re: Occam's Razor
"... been able to manufacture lookalike replacements ..."
Maybe they have. This device is just the decoy.
Re: Occam's Razor
a highly complex and detectable method
Depends on who you want it undetectable (or at least nearly undetectable) for. On the manufacturing side you actually need just a few people who know the details: the ones modifying the schematic and the PCB layout, and creating the manufacturing manifests for the board etching/sandwiching/populating machines. It's the ones that handle the boards after they've shipped (building them into systems, reflashing, further inspection, etc.) that these mods need to be hidden from.
Also, for everybody else in the manufacturing chain these are just like any of the other customised boards destined for a particular customer.
Re: Occam's Razor
mill of the packaging and check the die with a verified sample.
If the spooks are capable of making their own silicon as the allegation suggests, then the die itself can be made to look little different to the genuine product. In many cases a complex chip contains its own CPU and microcode - the only difference between the real and bogus chips being the microcode in it's on-chip ROM which will show no visible difference.
Re: Occam's Razor
" ... and creating the manufacturing manifests for the board etching/sandwiching/populating machines ...
This is like saying that a car manufacturer could start making flying cars without any of the factory workers noticing except those operating the machine that bolts the wings on. PCB manufacturers do not *have* any machines for sandwiching chips between layers. It is not a normal PCB process. I think all the factory workers would know about a brand-new machine and completely different workflow.
You would have to be an idiot to have Super Micro's BMC accessible from the Internet.
Re: No Worries
And you would have to be an idiot to believe that it won't work with the BMC not having a direct internet connection.
The US has fabricated intelligence information before to start a war. I think there is just as high a probability as anything else that this is fabricated, or exaggerated, and Bloomberg has been duped, just like most major US media outlets and Congress were duped into going all-in for the Iraq war.
Frankly I'm amazed the cloud guys even bother with BMC
Surely AWS don't bother with a baseboard controller on their servers...i.e. go to the aggravation of allocating an IP, a subnet etc just so some lucky NOC noggin can web or SSH in once in 18 months when the server seems squirrely? I mean their volumes make this deeply unlikely - you'd need hundreds of people to monitor the server estate in that manner. I posit that AWS's ask SuperMicro for custom server designs that don't even bother with the chip.
So while I can well believe the Chinese might attempt something like this, I'm also skeptical that the attack exists as described.
And of course, are we really to believe that the Chinese assumed that AWS's networks would simply allow the rogue chip to phone home for instructions? Those things must be locked up tighter than a gnat's sphincter.
To reiterate: yes I believe the Chinese state has motive, means and opportunity. And as they say on the UK cop shows I watch, they "have prior". I'm just not convinced about this particular attack.
Re: Frankly I'm amazed the cloud guys even bother with BMC
Err... plugging a cable into the IPMI port and then let it grab an address with DHCP is not exactly the most complex provisioning task. By contrast, the NOC noggin is far more likely to incur higher costs by accidentally prodding the wrong box than the total cost of the cables, switches and infrastructure required for this.
Sad to say, but this story is probably true...
1) Everyone knows that China loves to acquire/steal/copy other countries intellectual property. They aren't the only ones who do this, but I think I am safe in assuming that most Regenistas would agree that they at least have the public reputation of being the worst actor in this area. There is a reason why if I were to use the phrase "Chinese knock-off" when talking about a product, pretty much every Regenista would pretty much know what I was saying
2) Like every sigint/intel community, the Chinese want to know what target governments/militaries/companies/organizations are doing.
3) Unlike most sigint/intel community members, the Chinese have a very large portion, and perhaps most of the sub-assembly and component supply chain in their own country.
4) The Chinese government is very authoritarian, and will openly throw up regulatory/legal/political roadblocks against companies that don't do what the government wants. I've seen this first-hand, and of course there are barge-loads of news stories about this.
5) The Chinese culture also has a higher-than-normal tolerance for bribery.
6) Thanks (once again!) to Edward Snowden, we know that both the U.S. and British intelligence communities have the capability, operational authorization and much past experience with intercepting and backdooring electronics shipments when they are in the supply chain. It is pretty much certain that the other members of the 5 Eyes agreement either have this capability in-house, or they subcontract this kind of work through relevant requests to their U.S. and British partners.
7) Given their reputation for corporate and governmental espionage, I would assume that at the very least France and Russia also have and use these capabilities in-house.
So why wouldn't huge, powerful and increasingly global China do this? It's not like China is morally like a giant version of Sweden or Switzerland. They have no compunctions about being very tough and aggressive versus potential regime opponents, ethnic and religious minorities, journalists and whistle-blowers, companies that want to invest in or export to China, neighboring countries that are not overt allies, etc.
So given that China and others are almost certainly doing this, then I guess the real issue is what do IT and tech professionals do about it? One, make sure that you have elaborate network monitoring, data and firewalls if you are dealing with strategically valuable or sensitive commercial or government information. Your average small/mid-sized business can get away with basic antivirus and email filtering, but once you are dealing with data where you can look at it and objectively say "I could see how country X would like to get this info to give an advantage to their government or corporate sectors.", then you should probably start pressing your management to get very serious about network security.
Two, give as little cover as possible to national security/intel agencies that try to get involved in standards-setting. Assume that they do not want secure IT for anyone but themselves. In fact, truly secure IT would tend to dramatically reduce their usefulness to political and strategic decision-makers, and therefore negatively impact the future of those agencies and the careers of their employees.
Three, if you are dealing with the kind of valuable information that I discussed above, and you have the resources to really audit and inspect your hardware, do that.
A couple of other points
...because similar motherboards were in use "in Department of Defense data centers, the CIA’s drone operations, and the onboard networks of Navy warships."
Uhhhh, nope. Chinese-manufactured motherboards cannot be sold to US governement agencies, especially military or intelligence. Stuff has to come from one of a trusted list of countries called TAA (Trade Agreements Act, FAR 52.225-5). China is not on this list.
Just try to find a disk drive not made in China. Thailand maybe? Been there, had to find that. Did.
..."the middlemen would organize delivery of the chips to the factories."
Wow. Just wow. This is classic misdirection. The Intelligence folks are trying (and largely succeeding with Bloomberg, el Reg, WaPo, etc) to focus attention on one vulnerability, namely surreptitious factory modifications. But there are more, so many more. A few hints:
o Connectors. Yes, those boring black thingies with wires going in, and out. Embedding a chip within a connector requires no BMC changes, is difficult to check even with Xray, and completely unobservable. And since there is exactly one connector model that fits in exactly one place, no wastage. Profit!
o Firmware "adjustments" as many others have suggested. But where? Not just SMI flash but ... power supply flash. RAM controller firmware. CPU firmware. Or simply radio transcievers embedded in the board (or connector) that introduce firmware changes at boot time. (Where is the transmitter? Hmmm...maybe power supply RF emissions?)
o Known zero-day vulnerability in....CPU (obviously), BMC firmware (also obviously), but: Ethernet chips, memory controller chips (like RowHammer), PCI bridge chips, etc
o NSA black bag ops. However, at the scale of Amazon, Apple, etc, probably not cost effective. Except see my next point.
One thing not mentioned by El Reg, is the scale of procurement by Amazon, Apple, etc. Unless merely for development purposes, these companies purchase servers by the container-load (where the container is pre-loaded with racking, switches, power, servers, etc). Thousands of servers per container. The assembler might be persuaded to mung things up. It seems a remote possibility but the supply chain risks at this point (well after the motherboard factory) have not been addressed in the press that I can see.
+1 for El Reg and very well reported. Thanks.
Re: A couple of other points
Wonder if there's a peer exploit putting a little cell phone in, say, the power supply, to get to the outside world from an air gapped system. Yeah, I know, a bit harder in a shielded room, but sure seems like thinking one could "phone home" with IP traffic is a little optimistic if the target had any sophistication or was of any value at all.
What a lot of FUD
If Super Micro construction had been compromised, well you'd think one of these boards would have escaped to an outraged owner who'd have provided a photo of the 'unknown chip'
As it stands "There's been a massive hack" - and then "a massive coverup" - and all involved wish it to go away (except for a load of third parties who've both leaked the details and hidden the evidence)
Re: What a lot of FUD
Clearly you have little understanding of the problem.
If you ever get the chance to take the top of a "Real" computer......... good luck on identifying even half of the components without highly specific engineering drawings.
When the two sides stories sound credible...
... they are likely to be misled by a third party. So who stands to gain from making such noise? About how the US shouldn't be buying Chinese stuff but make their own, and if things happened to go wrong, the media is dishonest?
Especially given the timing just before the midterms, would be highly surprising if some at the GOP were not behind this story. I mean if the story was real, what are the odds of the loudmouth administration actually keeping quiet about something that would drive their agenda perfectly?
I reckon this is a cautionary tale
As others have noted, where are the modified boards? Why are seeing pictures showing some random small thing next to a pencils/pennies? How do we reconcile the denials against the story?
As has been pointed out, an attack like this is plausible. It could be done. It might have worked if it was done. But it alternatively might have been detected early, and that detection resulted in nothing happening.
So my working hypothesis is that this is a cautionary tale: beware of your supply chain.
And for that, it doesn't matter (to the teller of the tale) whether all the details are 100% factual, because they're just there to jazz up the story. Apple, Amazon, a bank, CIA spy videos... even Supermicro. The point of the story is not that Something Happened (to Apple/Amazon/the bank/whoever), but that Something Could Have Happened.
Consider this: is it plausible that *if* China was surreptitiously tinkering with a motherboard that they would have succeeded first time out? Why does the Bloomberg article talk about various different types of spy device, without ever explaining why there are the variations?
So we have a report of several generations of spy chip with no explanation of what the second and subsequent ones were for (they can't still be for the Amazon boards) and how they were identified, and we have no exhibits of the compromised items even though there are several generations of spies implying several generations of targets.
And we have NO information about the "phone home" mechanism which is, apparently, teh whole point of the spy chip. And I'm not just talking about the absence from the Bloomberg article: NO ONE has publicly reported and described suspicious activity resulting from the nefariousness. No security notices have been released suggesting a list of IP addresses to block. And, err, there are non-public infosec channels that exist to disseminate advice to people in the US defense industrial base, and if there was corroboration, I'd expect it to leak.
So I think this is a hypothetical attack being reported as an actual attack at the behest of the US government as part of an effort to help prevent actual attacks using this sort of technology.
From the Bloomberg article:-
“Having a well-done, nation-state-level hardware implant surface would be like witnessing a unicorn jumping over a rainbow,” says Joe Grand, a hardware hacker and the founder of Grand Idea Studio Inc. “Hardware is just so far off the radar, it’s almost treated like black magic.”
The real question in all this is: Who is the logistics super-genius running all of this, and when are they going to get poached by Amazon?
Amazon and Apple are not going to chew off the hands that feed them. I place more faith in the Bloonberg report than their denials....
You only have to look at poor FDTI and their range of RS232 chips.
Tell me none of those went into servers and other equipment.
The very people making the accusations like the NSA have been caught planting malware into Cisco hardware. The BMC have their own port and embedded operating system and web console seperate from the motherboard. I had one myself for a Sun server they called it Lights Out Management.
Nobody in their right mind would have this unprotected without behind a vpn / firewall. I can tell you this gets brute forced.
The level of effort for little gain is a dead give away. If the BMC isn't even connected then this hack is useless too. I am not sure if it's part of the motherboard or a daughterboard as the Sun server's was.
I can only presume it's seperate and "isolated" although it gives you full access to the server terminal like a keyboard and monitor. To configure the bios and power cycle etc.
But what is for sure they think because it's a remote console, and mention malware plant, they think people will automatically believe them. I want to see packet inspection logs !!
Truly irrelevant whether or not the story is true--it is a warning that we must heed, that our electronic civilization is subject to manipulation by others.
We tend to delay security until we are burned, partially due to cost considerations, and partially plain failure to understand that a risk still exists even if no cases have yet occurred.
Billions of dollars worth of software has been written offshore by programmers not restricted by law from stealing it and perfectly positioned to make unknown modifications. Such code changes are seldom easily spotted if at all. Similarly forged chips can contain code and unknown functions.
It is long past time we abandoned passwords and 4-digit PINs for anything of any real importance, and time to recognize and build protections into our sourcing systems against spyware hard and soft.
If we await our first "big" loss, the cost might destroy us.
"It is long past time we abandoned passwords and 4-digit PINs for anything of any real importance, and time to recognize and build protections into our sourcing systems against spyware hard and soft."
Ever heard of the phrase "the cure is worse than the disease"? At least the "big" loss isn't a guarantee, but killing ourselves trying to guard ourselves from a risk may well be a certain death. In which case, it's better to just roll the dice. After all, we've been fapping around with passwords for decades...because there just isn't any better alternative that employs nothing but the human brain, especially for faulty brains. AFAIK, it's a physical impossibility: what man can create, man can re-create. It's a problem that's been known since before World War II (based on the writings of E. E. "Doc" Smith, who had to resort to science fiction to find a foolproof solution).
Did anyone actually verify any motherboard?
Why there's tons of speculation on who's right and who lies instead of actually getting samples of those MBs and actually inspecting them. I understand Bloomberg might not have experts, but there are plenty of security companies in US and Europe.
They should be itching to get their hands on samples and tell the whole world they actually found it and know its insides.
Believe no one. Check the facts.
Re: Did anyone actually verify any motherboard?
From what I've read, they're supposed to be custom boards, meaning they're under contract and probably considered trade secrets. IOW, samples won't be forthcoming due to legal trouble.