back to article 'Rowhammer' attack flips bits in memory to root Linux

Last summer Google gathered a bunch of leet security researchers as its Project Zero team and instructed them to find unusual zero-day flaws. They've had plenty of success on the software front – but on Monday announced a hardware hack that's a real doozy. The technique, dubbed "rowhammer", rapidly writes and rewrites memory …

Page:

  1. Morrie Wyatt

    It's not the fault of Linux

    To be fair, this is an exploit of the underlying hardware, not a flaw in linux itself.

    After all, you can slow down a hard disk disk I/O by yelling at it. (the accoustic vibration affects the read heads positioning, multiplying latency.) This is just another case of rattling the hardware until somthing breaks.

    1. thames

      Re: It's not the fault of Linux

      I don't know if they changed the article since your post, but the second paragraph does say this will work with any operating system. It's a hardware level exploit, targeting a feature of the x86 architecture (page tables). The proof of concept they created used Linux, because that's the operating system the researchers use and are familiar with.

      This hardware level exploit is a form of "writing to arbitrary memory" except this one does not require a vulnerability (bug) in the operating system in order to work. There are probably other variations on this which are possible once people put some effort into it.

      This is a problem that Intel, AMD, and the PC hardware vendors need to address. It's not something that can reliably be fixed in software. At best, you can tweak some firmware settings to increase the time for the exploit to succeed, but that's not a real solution.

      For anyone with vulnerable hardware, you're pretty much out of luck. The one mitigating factor in your favour is that it may be hardware model dependent rather than an across the board exploit.

      1. Paul Shirley

        @thames

        It probably can be fixed in firmware by adding the right guard pages around critical structures but knowing where those pages need to be will require ram manufacturers input, component databases in each os and probably won't happen.

        What's more likely is they'll need to reduce power saving tweaks (like under volting and reduced refresh rates) and everyone gets lower battery life on their laptops. No idea why they think more than a tiny minority of desktop machine are using ECC.

        1. Metrognome

          Re: @thames

          Agreed on the ECC point.

          I have yet to see ECC RAM fitted to any desktop from the humble Dell Minis all the way to powerful CAD workstations, enthusiast LAN party gaming rigs and everything in between.

          In fact, outside of Xeon CPU's there's almost nothing for the desktop. (There's a few for lappies and embedded but not for desktop).

          Haswell and Broadwell have only just started offering ECC support.

          1. Gordan

            Re: @thames

            "In fact, outside of Xeon CPU's there's almost nothing for the desktop. (There's a few for lappies and embedded but not for desktop)."

            FYI, most AMD chipsets still support ECC, whether it is officially listed on the motherboard spec or not.

          2. Phil O'Sophical Silver badge

            Re: @thames

            I have yet to see ECC RAM fitted to any desktop from the humble Dell Minis

            My old Dell Precision 390, bought 8 years ago, has ECC RAM. As far as I remember it made no significant difference to the price.

            1. Anonymous Coward
              Anonymous Coward

              Re: @thames

              Having fabricated numerous machines from scratch, my experience of buying ECC ram has been the opposite. Maybe you got lucky with a deal at the time you bought your Dell, or more likely a period a long time back when RAM became ridiculously cheap for a while.

              Every time I've spec'd a new machine, the ECC option has always been significantly more expensive than non-ECC. Without a use case to justify the additional expense, I've always gone with the cheaper non-ECC option, and spent the saving elsewhere in the system such as a faster, more reliable, hard drive.

              Am going to grab a copy of the POC code, to see if the Corsair ram in my Mac Pro workstation is vulnerable. Although its a lot higher spec to your usual budget DDR boards, I suspect its probably still vulnerable.

            2. Metrognome

              Re: @Phil-O-Sophical

              What you described there, was quite a powerhouse for the time.

              Quadro Nvidias, SAS interfaces, 8 GB RAMs and the first Core2Quads and all that circa 2006!

              1. Phil O'Sophical Silver badge

                Re: @Phil-O-Sophical

                Well, mine had 1GB RAM and a 2.13GHz Core2Duo, but it's got more RAM now, and extra disks. Still runs fine with Windows (XP, it was that or Vista) but more often Solaris or Debian. It was bought as a combined home + work-from-home system, so I did go for a decent one. I suppose more workstation than home desktop.

            3. Michael Wojcik Silver badge

              Re: @thames

              My old Dell Precision 390, bought 8 years ago, has ECC RAM.

              ECC RAM was not uncommon in UNIX workstations circa 1990, for that matter. Commoditization of desktop systems largely pushed it out.

      2. Michael Wojcik Silver badge

        Re: It's not the fault of Linux

        For anyone with vulnerable hardware, you're pretty much out of luck.

        Yes, and it's broader than this particular attack - this attack is just a proof-of-concept based on the underlying vulnerability, which has various manifestations. The original paper, which was published last June, has more details.

        The paper also mentions, in a footnote:

        The industry has been aware of this problem since at least 2012, which is when a number of patent applications were filed by Intel regarding the problem of “row hammer” [6, 7, 8, 9, 23, 24]. Our paper was under review when the earliest of these patents was released to the public.

        It seems to me quite unlikely this isn't being used by APT teams run by various governments, on those occasions where they can't find an easier way to elevate.

        A related note: Some may remember a successful attack on the Java type system some years ago, which involved an application that filled available memory with objects of a particular type, then stressing the RAM to cause (with high probability) a bit flip in the type label for at least one object, which the application then exploited to escape the type protections. The authors of that paper used a heat lamp to cause random faults in the RAM chips of their target system; Row Hammer shows it's possible to do that in software.

    2. Gordan

      Re: It's not the fault of Linux

      While this is an exploit, it shows that modern hardware is actually unstable out of the box even without overclocking or other tuning that reduces the margins for error. Anything that causes memory corruption on hardware level is, IMO, a hardware fault, and therefore grounds for returning the hardware to the retailer as unfit for purpose.

      Given the descriptions of the methods, this is also mostly a RAM fabrication issue, rather than being largely related to the rest of the machine, as the leakage happens directly within the RAM chips. So using better RAM from a different manufacturer would almost certainly reduce the exposure to this bug, much more so than using the same RAM in a different laptop.

      But in any case, ECC is the way forward - if only it was more commonly available in laptop and desktop grade chipsets.

      1. phuzz Silver badge
        Facepalm

        Re: It's not the fault of Linux

        "Anything that causes memory corruption on hardware level is, IMO, a hardware fault, and therefore grounds for returning the hardware to the retailer as unfit for purpose."

        All electrical and mechanical devices will fail if you take them far enough out of their normal operating regime, this is no different.

        This is the equivalent of taking an ordinary road car, and driving it constantly up and down a road full of potholes, until parts fall off (the clever bit here is they've tuned the potholes just right to make the bit they want fall off). It's just not something that's going to happen in normal daily operation. If you have to drive over bumpy roads, you buy a vehicle with better suspension, if you're hammering your DDR then you buy ECC.

        1. DropBear

          Re: It's not the fault of Linux

          All electrical and mechanical devices will fail if you take them far enough out of their normal operating regime, this is no different.

          Bollocks. The point is that this IS WITHIN their normal operating regime. You should be able to flip bits at full tilt 24/7/365 without any of this shit happening, as long as you don't actually overclock that RAM - and I have not seen overclocking being mentioned.

    3. Robert E A Harvey

      Re: underlying hardware

      I used to design embedded systems between the 70s and 90s, and we used to use static ram exclusively because of effects like this - accidental, not malicious.

      If you are involved in volume design, then it is a lot easier, but in small volumes a cockup in the layout or timing of dram was very costly, in time and money. So we didn't use it at all.

  2. Scott Earle
    FAIL

    “On the software from” ?

    Have El Reg’s proofreaders all gone on holiday or something?

    1. Captain DaFt

      Re: “On the software from” ?

      "Have El Reg’s proofreaders all gone on holiday or something?"

      <PFFFFFT> Uh... yeah, might as well ask about their unicorn while you're at it.

      1. Scott Earle
        WTF?

        Re: “On the software from” ?

        Wait … they fired the unicorn?!?

        1. TRT Silver badge

          Re: “On the software from” ?

          No, it left of its own accord when there were no more virgins in the office.

          1. Alfred
            Coat

            Re: “On the software from” ?

            No virgins in a web-based technology magazine's office? A likely story.

    2. diodesign (Written by Reg staff) Silver badge

      Re: “On the software from” ?

      It's been fixed. Click on some ads and we'll hire more proofreaders :-P

      There's always corrections@thereg if you want to point out typos. We don't have time to read every comment, so those emails are appreciated.

      C.

      1. Anonymous Coward
        Anonymous Coward

        Re: “On the software from” ?

        Click on some ads and we'll hire more proofreaders :-P

        Ooh, very brave! IME saying (our even hinting at) that is just about the only thing that upsets Google enough for them to send you nasty emails threatening to cut off your ads. :-)

        Of course given El Reg's principled antipathetic editorial attitude to Google they may well use alternative ad networks that are less sensitive about click fraud...

      2. Dan 55 Silver badge
        Trollface

        Re: “On the software from” ?

        If we promise to click on some ads will you get the people involved in the website makeover to finish it off?

        1. Paul Kinsler

          Re: will you get the people involved in the website makeover to finish it off?

          Two bullets or three? :-)

      3. auburnman

        Re: “On the software from” ?

        What's your ratio of corrections mailed to corrections@thereg vs. posted in the forums? I really think it would be worth adding a button that lets you flag your own post as correcting the article. Limit it to badge holders and take it off anyone who abuses it.

  3. Anonymous Coward
    Anonymous Coward

    Desktops don't have ECC

    At least not Intel's. The reason it effects laptops moreso than desktops is that laptops use low power DRAM built with smaller processes. The smaller the process, the greater the likelihood rowhammer will work. ECC will put a stop to it (correcting single bit errors, and taking a machine check for uncorrectable double bit errors)

    Pretty much certain that all smartphones would be vulnerable to this attack, as they use low power DRAM without ECC.

    Since Apple designs their own SoC and therefore their own memory controller and the iPhone tends to ship with / require less RAM than high end Android phones, they could fix this by adopting ECC. Without source code access it might be a bit harder to develop an exploit for this against iOS, but it isn't impossible.

    1. thames

      Re: Desktops don't have ECC

      This is making use of an x86 hardware feature. We don't know if an equivalent exploit is possible with ARM. If it is, then any SoC manufacturer could add ECC support, if it doesn't have it already, and any phone manufacturer can add ECC.

      The real problem is going to be all the existing phones out there. If you bought a cheap phone, then it's not a big deal to throw it away and buy a new one. The people who bought high priced phones though will be completely stuffed. There's no such thing as a "patch" to fix this. They'll be stuck with vulnerable phones bought on multi-year contracts and whose resale value has sudden fallen to zero.

      So if you want to see which manufacturers will be most affected, look at the ones who sell the most expensive phones.

      1. joeldillon

        Re: Desktops don't have ECC

        If the 'hardware feature' is page tables then modern Android and iOS phones have those on ARM. They're pretty fundamental to any modern protected-memory OS.

      2. Gordan

        Re: Desktops don't have ECC

        "This is making use of an x86 hardware feature."

        It's not an x86 specific feature per se. The testing code uses an x86 assembly instruction that bypasses CPU caches for reads. It is quite likely that similar equivalents exist on many other if not most CPU architectures.

        1. thames

          Re: Desktops don't have ECC

          Gordan - "It is quite likely that similar equivalents exist on many other if not most CPU architectures."

          Perhaps, but as I said, this particular exploit is x86 specific. Nobody has demonstrated whether an ARM (or MIPS) equivalent is possible yet. Given though that this thread started off with someone talking about how he felt that Apple phones were going to be better than Android phones when it comes to dealing with this issue, I think we need to step back a bit and admit we don't know if it is a problem for ARM yet.

          1. Michael Wojcik Silver badge

            Re: Desktops don't have ECC

            Nobody has demonstrated whether an ARM (or MIPS) equivalent is possible yet.

            Perhaps not with those specific CPU families, but the general thrust of your argument is wrong. Read the original paper by Kim et al (DOI 10.1145/2678373.2665726). Part of their study involved testing a range of DRAM chips using an FPGA-based system - no x86 in sight.

            The Row Hammer vulnerability is a flaw in DRAM implementations. The particular attack developed at Google Project Zero is x86-specific, but the vulnerability is not, and there's no reason to believe it can't be extended to most systems that use a different CPU but the same DRAM.

    2. diodesign (Written by Reg staff) Silver badge

      Re: Desktops don't have ECC

      "laptops use low power DRAM"

      I've tossed that into the story. FWIW Intel does do desktop mobos with ECC support.

      C.

    3. Anonymous Coward
      Anonymous Coward

      Oh dear. Wrong in lots of ways.

      Not least affect/effect but to state 'Desktops don't have ECC, At least not Intel's'

      (I'm forgiving you for the apostrophe you used in 'Intel's' as it's not entirely incorrect but I suspect you didn't mean it that way)

      http://www.intel.com/support/motherboards/desktop/sb/cs-009023.htm

      ECC is already available on the chipsets used in some Android devices and has been since 2010 (possibly earlier)

      Apple may 'design their own' SoC but it's a rehash of someone else's IP (ARM license a lot of IP to Apple) so yeah, it's possible that it amy contain ECC already but to include it is considerably more difficult than picking a chip and OS that can cope with ECC by design.

      1. Anonymous Coward
        Anonymous Coward

        Re: Oh dear. Wrong in lots of ways.

        The only ARM IP in Apple's SoCs is the ISA itself (i.e. written documentation in English) the cores are entirely Apple's design since the A6. If you think Apple is using more than that, you'll have to explain how they managed to complete the design of their first 64 bit SoC before ARM did, and well before ARM released the RTL to their partners.

        I didn't mean to imply that Apple is any closer to Android today in being able to defeat this sort of issue, assuming someone proves it is exploitable on ARM (I see no reason why it wouldn't be as it supports uncached reads) I simply meant that if Apple decides they want to do ECC in future iPhones, they can do it a lot more quickly than Android OEMs because they control the design of their SoC. Aside from Samsung, Android OEMs rely on others to design their SoCs - and even Samsung still uses ARM designed cores, they have yet to release a SoC using a core they designed themselves.

        I wouldn't worry about phones having value "dropped to zero" due to such an exploit. You may not be able to patch a hardware flaw via software, but you can certainly prevent the code sequence that exploits that hardware flaw from running on your platform if it requires signed code.

  4. Nate Amsden

    I wonder

    How well something like HP's Advanced ECC or IBM's Chipkill which go well beyond basic ECC would hold up to this sort of attack. Myself I don't deploy any serious systems without this technology, as the systems tend to have dozens to hundreds of gigs of ram and ECC alone just doesn't cut it in my past experience anyway.

    Last I looked I could not find good info on IBM's ChipKill but HP has good info here on Advanced ECC:

    ftp://ftp.hp.com/pub/c-products/servers/options/c00256943.pdf

    some text from the pdf

    "To improve memory protection beyond standard ECC, HP introduced Advanced ECC technology in 1996. HP and most other server manufacturers continue to use this solution in industry-standard products. Advanced ECC can correct a multi-bit error that occurs within one DRAM chip; thus, it can correct a complete DRAM chip failure. In Advanced ECC with 4-bit (x4) memory devices, each chip contributes four bits of data to the data word. The four bits from each chip are distributed across four ECC devices (one bit per ECC device), so that an error in one chip could produce up to four separate single-bit errors.

    Since each ECC device can correct single-bit errors, Advanced ECC can actually correct a multi-bit error that occurs within one DRAM chip. As a result, Advanced ECC provides device failure protection

    Although Advanced ECC provides failure protection, it can reliably correct multi-bit errors only when they occur within a single DRAM chip."

  5. Hackbert

    Just checked the calendar. No, it's not April 1st.

  6. Conundrum1885

    Re. RowHammer

    Interestingly I suggested quite a while back something along these lines to implement a neural net using Flash memory, and actually have some schematics here for an AI that uses this exact technique to get nearly-quantum level speedup effects using a bootable pendrive that runs DSL and then uses the leakage between the memory cells (has to map out chips and look for correlations but that is doable) to run the NN.

    It should work on any old x86 laptop with 2GB RAM but obviously the faster CPUs are more efficient and for something like this a custom BIOS that overclocks the RAM chips just enough would be ideal with a thermal sensor for feedback to keep the chips in the desired temperature range.

    Not exactly on topic but still interesting, as chips are getting more and more dense it is entirely possible that something as simple as a RaspPi2 (1GB RAM) if kept at just the right low temperature in a strong magnetic field could implement a limited subset of the Turing test..

    Also relevant, if the memory manufacturers would get back to me and send me the NDA already I could make this work on a 64GB microSD as the densities on these are many times greater and with billions of potential artificial neurons between the adjacent cells that are currently ignored due to wear leveling.

    Had some success flash X-raying defective 32GB chips to bring them back to life and noticed effects suggesting this could work but ran into problems replicating the effect as the power supply conked out during testing.

    Anyone interested?

    1. Anonymous Coward
      Anonymous Coward

      Re: Re. RowHammer

      I'm interested. Now put on this lovely white jacket and we'll fasten the straps at the back for you.

    2. Jimmy2Cows Silver badge

      Re: Re. RowHammer

      Surely it'd be easier to replace the power supply than battling manufacturers for an NDA. For a few quid you could continue your glorious endeavour, rather than blaming your inability to proceed on lack of an NDA...

      1. Conundrum1885

        Re: Re. RowHammer

        As in the NDA to get the memory controller specs for microSD cards.

        Its pretty hard to reprogram these AFAICT but some of the older cards did have points under a removable label where the pins could be accessed.

    3. mevets

      Re: Re. RowHammer

      Very interested in the travesty generator you used to generate your comment.

      How much did you have to seed it with to get as close to this topic as you did?

  7. Anonymous Coward
    Meh

    "While this was a high cracking rate, the team reported almost no success on desktop machines. This is possibly because those computers use newer RAM with error-correcting memory (ECC)"

    Most desktops don't use ECC due to the fact it would of been about double the price when new, workstations may of shipped with them, but your run of the mill pc won't have done. Also as for "newer" RAM, if your comparing a 2010 pc with a 2015 pc of course it will.

    My guess is the fact one is likely to be DIMM and the other a SODIMM, but I'm just guessing.

    1. Bronek Kozicki

      I guess the difference is DRAM refresh rate. High refresh rate means higher power utilisation to keep RAM powered up. This is insignificant for a desktop PC with AC power attached, but significant for a laptop.

    2. Paul Crawford Silver badge

      Double the cost?

      Really? ECC memory costs more, but typically 20% and the RAM is often only a fraction of the machine cost.

      True, proper servers cost a lot more than desktops, but there are other factors in that cost such as dual PSU options, easier to change fans, hot swappable HDD, etc, (and probably a bit of profiteering as well).

      1. Paul Shirley

        Re: Double the cost?

        @paul Crawford

        We live in a world's where Lenovo were prepared to ship malware on PCs because margins are too slim on the hardware. 20% on the ram really is a significant overhead for most of the devices shipped.

  8. Dan 55 Silver badge
    Facepalm

    Meanwhile, in a parallel universe...

    Palo Alto, CA...

    Newly incorporated security outfit Project Zero claims it has found a serious design flaw in the HAL Laboratories' HAL 9000 computer. The claim that if the user were to gain access to the the interior of the computer and randomly remove memory modules from memory arrays, they could force a system shutdown even they don't have permission to do so.

    A spokesman at HAL Plant in Urbana, Illinois said, "No shit, Sherlock".

  9. RyokuMas
    Coat

    Start the clock...

    Hardware manufacturers, you have 90 days to fix this, starting NOW!

  10. Tromos

    Brands

    Is anybody looking into differences between RAM chip manufacturers as regards to susceptibility to this attack? I'm sure some brands will come out better than others as this essentially relies on a fundamental design flaw in the chip.

    1. Anonymous Coward
      Anonymous Coward

      Re: Brands

      Sounds like a marketing opportunity to me.

  11. naive

    It is just an elevation

    Although unsettling, it is "just" an elevation. Somebody already needs access on user level to the system before this hack can be deployed.

    If I understood well, the solution would be to allocate the sensitive CPU privilege bits further away from memory regions accessible by the user ?.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like