nav search
Data Centre Software Security DevOps Business Personal Tech Science Emergent Tech Bootnotes
BOFH
Lectures

back to article
Decoding the Chinese Super Micro super spy-chip super-scandal: What do we know – and who is telling the truth?

Windows

From Amazon's denial

> "It’s also untrue that AWS knew about servers containing malicious chips or modifications in data centers based in China".

Umm, what about outside China then? Like in the USA?

Breaking out the popcorn. The fallout from this one will be entertaining.

8
3
Silver badge

Re: From Amazon's denial

The story said Amazon had only found servers with the spy chips in their Chinese datacenter. Apple found it in a limited fashion as well. These weren't across the board "wow, we have to replace everything in all our datacenters".

It seems both companies do some very robust checking of the boards they are shipped, since it was noticed rather quickly. It goes without saying that 99.99% of companies don't even check the boards in their servers to look for a tiny component that doesn't belong. The main reason Amazon and Apple did was because like other major 'cloud' companies such as Google, Facebook, Microsoft etc. they design their own boards, and want to verify what they are shipped matches their design.

That's why the spy chip was apparently disguised to look like a passive component, hoping it would be ignored as something to address RF or electrical issues.

2
4

Re: From Amazon's denial

The first case was from the acquisition. So yes, Amazon probably knew about it happening on US soil.

1
2

"At no time, past or present, have we ever found any issues relating to modified hardware or malicious chips in SuperMicro motherboards in any Elemental or Amazon systems."

I assume this means that modified hardware or malicious chips were working just fine? No firmware issues like Apple's?

4
1

'Bloomberg Terminals, with many layers of editing and fact checking, and it has a zero tolerance on errors: it is inconceivable that it would publish a story this huge that wasn't watertight.'

A bit of a bold statement, no ?

5
6
Silver badge

Not bold, simply logical

Bloomberg charges a LOT of money to traders for their information, including early access to their stories. They aren't going to risk a multi billion dollar business on a story they aren't 100% confident in. That doesn't mean they can't get fooled if the CIA threw its full weight behind trying to fool them, but I think it is more likely that Apple and Amazon's denials are either wrong or were somehow coerced.

Most likely wrong - if only a few people in each company learned of the issue before contacting the FBI, and they were then told not to tell anyone else, yesterday's denials are easily explained. Press contacts Apple / Amazon spokesperson for comment. They contact various executives asking "do we have any comment on this story" and the executives all know nothing about it and neither do their underlings they talk to - because the odds of finding the three or four people who do know about it are tiny - so the spokesperson reports back that the story is false.

The employees who apparently went to the press (perhaps they were worried other US companies without their resources would be unknowing victims if the government kept the story hush hush forever) aren't likely to fess up to others in their company now - because it would be obvious they had leaked info to the press about it and the company would think "if you leak this, you might leak other stuff, so here's your severance package goodbye"

23
3
Gold badge

it is inconceivable that it would publish a story this huge that wasn't watertight.'

A bit of a bold statement, no ?

MrBlack,

I suspect "watertight" here is journalese. You can't guarantee that every story you publish is correct. Even historians with the benefit of multiple sources and hindsight can't do that. What you can do is check that your story stands up and is watertight. So you use multiple sources and do some checking on them, to make sure they've not got obvious motives to lie - or links between them that suggest a conspiracy. Once you've done all that you go off to the lawyers and make sure that your methods will stand up to getting sued - i.e. you can say you did all possible checking.

Often the lawyers will then send you back to do more homework. With something like this you'll obviously have to satisfy the editor, but also upper management, on the grounds of the financial risk to the company.

After all that, you then approach the people the story is about to see if they're willling to respond. And then do any more checking on what they say. If they deny everything then you have to meet an even higher standard of proof internally, because the legal risk has just shot up massively.

At some point you've then done all you can do - and then have to decide whether to publish or not. If the victims are denying everything, then you either trust your sources and research and risk it - or do nothing. If you do nothing, you may never find out the truth. If you pubilsh, there's a much higher chance the truth will come out. But obviously a much higher risk of getting sued, or being made to look really stupid.

17
0

Should we be worried ?

We recently had our pet cat `chipped' by a vet of Chinese ethnicity and she now spends increasing amounts of time curled up right next to the broadband router!

68
1
Silver badge

Re: Should we be worried ?

You should be worried, your cat is being affected by wifi radiation and will soon get superpowers like being able to leap 5x its height and fit itself into even the smallest box you have left lying around. If your cat already can do these things, it is too late....run!

36
1

Re: Should we be worried ?

This sounds like a great ignoble prize research topic: (chipped) animal behaviour influenced by external RF sources, via the nice warm neck (or wherever the chip got put) syndrome. What you need to do is set up another (identical) router a few feet away and alternate which one has it's WiFI transmitter on. Correlate with cat's favourite resting place...

11
1
Silver badge

Re: Should we be worried ?

"and fit itself into even the smallest box you have left lying around."

But you won't know if it's still alive until you open the box.

11
1
Silver badge
Boffin

Re: Should we be worried ?

Have you had shipments of tuna, salmon and catnip arrive at your door even though you're completely sure your cat has not touched any of your computers? As those implanted chips are small and low power they need to be really close to the access point to be able to connect, that's why you will find your cat right next to it.

5
0

Only just passes the plausibility test for me...

I agree it is completely feasible that a board maker could have it's design compromised. However if you look at a motherboard it is not straightforward to add any components at all. There is not a lot of room, chip embedded in the fiberglass is likely to overheat running at CPU speeds and burn itself away, getting access to the right tracks will be non trivial. It would probably also upset the fine balance of power management that modern motherboards don't have much tolerance for.

Overall if I am to believe the theory I would expect this to have been achieved by the motherboard designers, not just bodged in at the factory. This idea that its factory changed I find completely implausible.

If you are doing this in the highly managed environment of an AWS (for example) datacentre, the network traffic is so highly managed it seems unlikely that even if data is capable of being siphoned, your ability to trigger the siphon and retrieve anything is highly unlikely to be successful. It almost certainly would be unable to be contacted directly even with insider help. This leaves the possibility that it is trained to "look" for certain data streams to activate. Again how to exfiltrate, particularly if done in bulk across a whole datacentre, I can only imagine it would have to insert the data into what appears to be legitimate traffic, a sort of steganography. Trying to get anything coherent out a a vast number of servers operating in parallel seems both impossibly hard, and highly likely to be detected.

Overall my assessment is that this is likely to have been rasied as a potential attack vector, has been validated by the various anonymous sources, but likely has never been attempted at scales as described.

9
9

Re: Only just passes the plausibility test for me...

You are right about a lot of this, but have missed a few points:

It would indeed burn itself out and use too much power if running at CPU speeds. It doesn't need to. If the story is correct, it only needs enough processing to inject code into a serial line. That takes a lot less power. After this, the CPU handling the BMC handles all the work.

It probably wasn't (if it exists) created by the factory. Instead, the plans would have been created elsewhere, and a slight modification to the process would be necessary. I don't know much about the organization of Chinese motherboard factories, but if I had plans that were almost identical, I assume the factory could build them just as well.

The point about monitoring internet traffic is a good one. I don't have a great explanation for how that worked. The best I can come up with is that you could set up an image on such a system that could interact with the firmware and exfiltrate information into that VM, then hide the data as it is sent out from that VM with other expected traffic. Still, that's hard. If it actually exists and was used (it could be a sleeper system for some purpose), perhaps some network traffic systems aren't as thorough as we hope.

7
0
Silver badge

Re: Only just passes the plausibility test for me...

If you are doing this in the highly managed environment of an AWS (for example) datacentre, the network traffic is so highly managed

Piggybacking via steganography on entirely legitimate data connections to an AWS cloud.

1
0

Signal conditioning chips

Aren't chips (ICs) as we normally describe them but passive surface mount filters. It is extremely unlikely that power is available on any pads they are soldiered to.

The whole thing is bollocks.

1
12

Re: Signal conditioning chips

"It is extremely unlikely that power is available on any pads they are soldiered to."

There is this thing called phantom power. You use the data signal for power. I know this sort of thing was being done in the early '90s, the company I was working for did it for MIDI devices. Could suck enough juice from the MIDI signal lines to power the microcontroller and the rest of the circuitry, no need to plug in a wall wart.

7
1
Silver badge

Re: Signal conditioning chips

Also, these would not necessarily have been placed where a filter would have been, but somewhere with a +5v line nearby, for example. Who says the motherboard design was not slightly altered to accomodate them? Only an expert familiar with the non-tampered layout would notice it.

9
1

Re: Signal conditioning chips

Would take an expert to make the design change too, I would expect tampering like this to be invisible from a visual inspection, and only potentially possible from a photographic comparison particularly if it is intentionally obscured in the layers, or using another surface mount chip as cover.

2
1
Anonymous Coward

Re: Signal conditioning chips

Yep. It got stolen by the computer industry and is now called Power Over Ethernet. It isn't actually using data signal for power, it just uses the principle that you can run DC and AC on the same cable and separate the two at the other end with a capacitor. POE uses separate cores but it uses the same current driven technique. I love it when rock and roll infects the rest of the world !

Either way, power is power. As long as you what you suck is less than what will cause the receiving end to fall over, you are good to go.

1
2
Anonymous Coward

In an era of trade wars and when demonising China is a major populist dog whistle this may have another layer of onion skin to unpeel.

One might be labelled a conspiracy theorist to voice it but a story, true or not, that ties Chinese government hacking and off shore manufacturing by an American company in with a major security breach could be spun as a powerful warning to on-shore your manufacturing facilities and to "buy 'murrican".

2% off Apple's share price isn't just loose change down the back of the sofa, it will have made people panic and the 'drip drip' of anti Chinese manufacturing news could eventually force a rethink by some major players which could have severe implications for the Chinese economy.

14
2
Silver badge

The MAGA angle

2% off Apple's share price isn't just loose change down the back of the sofa, it will have made people panic and the 'drip drip' of anti Chinese manufacturing news could eventually force a rethink by some major players which could have severe implications for the Chinese economy.

Which is exactly what El Trumpo wants is it not?

Quite where they are going to find all the workers needed when say Apple brings their iDevice production to the USA is another matter entirely. Perhaps all those Coal Miners who have been led up the garden path by Trump and his 'Good Clean Coal' promise when mines and coal fired power plants have been closing at a faster rate than before he got the nuclear button codes.

18
3
ROC

MAGA angle indeed!

With all the known Chinese cyber attacks (siphoning data in all imaginable ways - IoT, routers, cell phone apps, etc), IP espionage, South China Sea aggrandizement, internal persecution, and external persecution of the likes of the Dalai Lama, they manage to demonize themselves quite well without external help.

If this is some "plot" to get more critical electronics manufacturing (consumer would be nice, too...) moved from their jurisdiction to about anywhere else in the world (not Russia of course, if they could even do it...), then I am all for it.

4
1
Rol

The idea that these companies denials could be later banked by a gaggle of lawyers acting on behalf of investors is the only argument in favour of believing their statements to be true.

On the other hand, their defence would no doubt revolve around the fact American intelligence services more or less written those statements for them.

6
1
Silver badge

Executives of large companies think short-term, not long. Shoring up the share price today is more important than preventing a collapse in a few years time when they will have cashed in on their stock options and moved on.

Hence the denials.

4
3
Anonymous Coward

It's simpler

"A fourth thing is this: why go to the bother of smuggling another chip on the board, when a chip already due to be placed in the circuitry could be tampered with during manufacturer, using bribes and pressure?"

As in the subject. It's simpler to add a chip rather than have a secondary, backdoored, CPU line.

3
2

Re: It's simpler

Agreed. Also, adding extra circuitry in the larger processor is probably easier to catch during root-cause analysis than something you never knew existed.

3
1
Anonymous Coward

Re: It's simpler

"As in the subject. It's simpler to add a chip rather than have a secondary, backdoored, CPU line."

"Agreed. Also, adding extra circuitry in the larger processor"

It's unlikely they're talking about a different processor, merely a different firmware for the same processor, so it's just a matter of reflashing something, which is technically much easier.

2
1

Re: It's simpler

The only case where this makes sense is if the attacker knows the server is going into a classified environment where as a matter of course every single bit of code on the motherboard will be wiped and reloaded before the server is installed. The hardware technique would allow the server to be re compromised after it was thought to have been scrubbed.

3
0

Entirely plausible

Completely plausible. The best proof would be an actual Supermicro board with the spy chip. My only question is, why do it and get a bad rap? My bet is that China has multiple approaches and this got caught. Moreover, China may have realized that the US was aware of the spy chip. So the US went public with it.

3
1
hk

Cui bono

I really wonder what's to gain from industrial espionage on companies like Amazon or Apple. The surely are advanced in operating their data centers with advanced technology, custom kernels and partially silicon. But that's still a far cry from some super secret formula.

3
1
Silver badge

Re: Cui bono

I really wonder what's to gain from industrial espionage on companies like Amazon or Apple

It's hinted at in the article - its not Apple or AWS they are targeting, but the end users of those systems. Compromising the manufacturing of their systems means you can get compromised machines into places that would otherwise be hard to get compromises into - and thus give you another attack vector into some "quite well defended" territory.

Also, as to the "why not just adapt another chip". Well if the manufacturer sticks a JTAG clip onto the flash ROM to put new firmware into it, your separate chip can sit there all safe and sound - and un-noticed. And don't forget that if true, this was done by people with access to the skills and technology to make it happen - it's not like you or me "hacking" a built board, it's the people who make the boards using a slightly modified design. A chip buried in the layers would be invisible, and if buried underneath an existing chip would even be (more or less) invisible to x-rays.

6
1
Anonymous Coward

Re: Cui bono

Azure? Google Cloud?

0
1
Anonymous Coward

Seems like Rube Goldberg approach to spying

There are tons of legit chips on the board. Instead of adding one which will be certainly discovered sooner or later, why not use one of the chips that are already there?

Just fake a vulnerbility in one of the legit chips, and if discovered, then it is just a bug, like so many others. Make it a hardware bug like Spectre or a well hidden firmware bug.

To me, adding an extra chip to do spy is a Rube Goldeberg solution to spying. The same goal could be achieved easier, with less chance of discovery and with better deniability when discovered. Just don't fix one of the vulnerbilities discovered during development of the board. Maybe even hide it a bit better. And if discovered: no biggy, because bugs happen - to everyone.

7
5

Re: Seems like Rube Goldberg approach to spying

That doesn't work. If anything modifies those chips later (I.E. the manufacturer updates something), your bug is destroyed. If the chip is tested, you are discovered. And you can't easily make new holes in the thing because you didn't design it. With a separate chip, the manufacturer updating a chip can't kill you, a test of a component cannot find you out, and you can use all those existing chips to hide yours, which can be really tiny and be set under another chip.

2
1

Re: Seems like Rube Goldberg approach to spying

Not so: if (as another poster has described) you have an SPI memory with a "secret" bank and a regular one, and the sneak chip flips between the two, why would you not also have it treat the JTAG interface in a similar sneaky fashion (i.e. write the new code to another bank)?

Remember: the implication is that these are custom designed parts to do the job, not commodity parts being used to carry malware.

4
0
Anonymous Coward

Re: Seems like Rube Goldberg approach to spying

It risks being spotted in a detailed x-ray, especially if the target is state-trusted agents who will be wary of such trickery.

In which case, your only bet is something so oblivious-looking that it gets overlooked.

0
0
Silver badge

A component inside PCB layers...

... would be possible to implement but would have to be thermally very innovative: an active device like this one dissipating power into a sandwich of low-conductive fiber-glass panel would require some copper nearby or at least vias to evacuate the heat, making it more visible.

8
3
Silver badge

Re: A component inside PCB layers...

I'm not so sure. I don't think the chip does very much most of the time on average, excepting the immediate bus-hugging logic I certainly wouldn't expect it to run its internal processor (if it even has one) at anything like "CPU" level speeds. Besides, all it really needs anyway is a good connection to the internal ground plane - using that for thermal dissipation at least partially instead of a conventional heat sink is already standard practice for certain types of designs...

8
0

Re: A component inside PCB layers...

Multi-layer PCBs have power and ground planes - entire layers that are pretty much entirely metalled with copper. Quite some heat capacity there. This answers the question elsewhere about where the power comes from - directly underneath through a via.

In fact, embedding a device into the middle layers of a PCB is genius - likely to evade optical inspection.

6
0
Anonymous Coward

post-truth times

there's absolutely no way to come even _close_ to the actual truth, i.e. what happened, and whether it actually happened. There's simply too much at stake for all parties (allegedly) involved - billions of USD combined with "national security" (and red buttons) and, of course, the battle is the world top dog. So, the best we can expect is a blockbuster movie (I bet the screenplay's already half-completed). Sad times.

3
3
Silver badge
Joke

"Apple's denial is typical Apple. Reflecting its superiority complex"

There goes the big exclusive apple were going to give you (!)

6
1

Good article

Thanks for breaking it down. I had no idea Bloomberg hacks are renumerated due to market moves either.

18
2
Silver badge

Having some experience of chip and pcb design and having worked with contract manufacturers I highly doubt Bloomberg's story. Sure its not impossible but it would be a damn site easier to compromise the firmware, IME or just use a flaw like Spectre or Meltdown.

4
4
Anonymous Coward

Second chip added, bollocks

This is a nation state with silicon foundries.

Why would you add an obvious second chip with limited code, processing and pins, when your can add a co-processor to the silicon of the BMC (either legit made in China or a clone slipped in to supply chain) if you're being cheap, you would add a second die during packaging and tap on to the bonding wires.

This is critical news for every "friendly" country and all business, but we can't be arsed to show any proof, even dodgy dossier WMD level chicanery.

All part of the info wars going on, just hope it stays virtual.

Tossers, the lot of them, nearly all of them in nearly every country, those who think they're better than the plebs, fcuking people over for their own ends. A pox on you all.

15
2

Re: Second chip added, bollocks

"all part of the info wars going on" Upvoted for that - whether you are right or wrong.

5
0
Silver badge

Article: "Why not switch the SPI flash chip with a backdoored one – one that looks identical to a legit one?"

Who knows, maybe this has also been done on some other motherboards...

5
1

Presumably because the contents of the SPI flash are easily verified- and something more sophisticated customers would actually do.

1
0
Anonymous Coward

Iiiii don't know… SEC and FBI should investigative Bloomberg (the company and the person) for market manipulation (I wonder how much they shorted on this news) and destabilizing Western society (there must be a Russian link somewhere here). Of course, heads should roll and sanctions imposed.

1
10
Silver badge
Unhappy

Trump is trying to divert attention from his Russian master, look over there---> It's Chinese hackers!

13
4

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

The Register - Independent news and views for the tech community. Part of Situation Publishing