back to article Meltdown, Spectre bug patch slowdown gets real – and what you can do about it

Having shot itself in the foot by prioritizing processor speed over security, the chip industry's fix involves doing the same to customers. The patches being put in place to address the Meltdown and Spectre bugs that affect most modern CPUs were supposed be airy little things of no consequence. Instead, for some unlucky people …

Page:

  1. Nate Amsden

    hyperconverged

    Haven't noticed anyone talk about this yet, but given the hit is much harder on systems that do a lot of syscalls I am curious the impact to hyperconverged systems.. Standalone storage systems that primarily leverage CPUs for storage stuff could do without the fixes since they are generally tightly controlled running only trusted software. Hyperconverged of course doesn't quite have that luxury in a typical deployment scenario.

    Of course if your hyperconverged system isn't pushing much I/O then you probably won't see a big impact.

    The lustre results are interesting.

    1. kryptylomese

      Re: hyperconverged

      My test ProxMox cluster (Intel Xeon x5675 based) and the guest systems that are comprised of both Windows 10 and Linux under KVM as well as a bunch of Linux containers under LXC have not seen any noticeable impact.

      Workloads include databases, file servers, PXE boot servers, web servers and PCIe pass though of both Graphics and Sound Card testing with intensive applications including games.

  2. Anonymous Coward
    Anonymous Coward

    So how much will this throw Intels release schedule out by?

    If it takes Intel 12-18 months of engineering and testing before they can release replacement architecture does that mean no CPU based speed increases before 2020?

    If it does what will be the likely revenue hit up and down the channel?

    1. TReko
      Stop

      Don't buy a new Intel based system for a while?

      There may be a huge hit to PC sales.

      Our company has temporarily suspended all but essential new system purchases until fixed silicon is available or we switch to AMD or we figure out what the performance hit is.

      1. illiad

        Re: Don't buy a new Intel based system for a while?

        BUT!! AFAIK it affects AMD and apple too... : O : O

        1. Anonymous Coward
          Anonymous Coward

          Re: Don't buy a new Intel based system for a while?

          Hmmm

          Since AMD is not effected by one of the bugs and less so by the other AMD processors are far less effected. At this point, the only reason that AMD systems are recording any significant slow downs is because Microsoft is forcing AMD users to incorporate ALL the bug fixes onto their system even though they are not needed and this is causing the additional slow downs. A person would have to be truly daft not to believe that Intel and Microsoft, both of whom had recently hit a wall with consumers on their latest offerings and were beginning to lose significant market share to AMD/Linux are colluding to improve their lagging sales (Intel because their newest CPU's are wildly expensive, hot running, energy hogs. And Microsoft because Windows 10 is nothing more than spyware in drag and a huge resource hog)

      2. pogul

        Re: Don't buy a new Intel based system for a while?

        > Our company has temporarily suspended all but essential new system purchases until fixed silicon is available or we switch to AMD or we figure out what the performance hit is.

        Except, AMD and ARM designs are affected too -- which everyone seems to be ignoring. The initial reports that broke in the media said AMD and others were unaffected: this is not the case, as has been widely publicised.

        I can't help wondering (as I haven't dug deeply into the speculative code execution thing) - is this flaw inherent in this architecture or did certain companies reverse engineer another company's implementation?

        1. Alan J. Wylie

          Re: Don't buy a new Intel based system for a while?

          AMD is (mostly) not affected by Meltdown (userspace reading kernel memory). It is affected by Spectre (userspace reading userspace memory, either in the same process, e.g.3rd party Javascript reading cookies in a web browser or one process reading another processes memory).

          There is, however a case on AMD when eBPF JIT is turned on, which allows userspace to read kernel memory.

          AMD doesn't have PCID.

          IBM have announced that firmware and kernel patches for their POWER architecture are will be released soon.

          1. Paul Shirley

            Re: Don't buy a new Intel based system for a while?

            "AMD when eBPF JIT is turned on"

            While running interpreted out of bound access will be checked and speculative execution will be in the interpreter, not an attacker controlled address. Turning on JIT allows an attacker to craft code that will be compiled to machine code to run potentially without checks. It's a way of weaponising an otherwise unusable kernel exploit.

            If enabling JIT does make AMD vulnerable then AMD is vulnerable in that test and you read too much into this. I believe the only test they succeeded with was user to user snooping which is expected to work. The more frightening user to kernel Meltdown blocking claim hasn't been disproved yet.

          2. Nick Ryan Silver badge

            Re: Don't buy a new Intel based system for a while?

            Please explain how JavaScript, which is an interpreted language, can be used to trigger the exact cache violations required to access memory owned by other processes. The exploit code is relatively trivial assembler code, however without using an exploit allowing the arbitrary execution of pre-compiled code, how is JavaScript code going to replicate the same cache violations?

            /confused

            1. Warm Braw

              Re: Don't buy a new Intel based system for a while?

              The Spectre paper goes into great detail, but there's a summary here.

              JavaScript isn't necessarily interpreted - the example exploit takes advantage of the JIT compiler in Chrome whose output is predictable machine-language instructions.

              In the same way, the eBPF JIT compiler can be used to inject known code into the kernel, if eBPF is enabled.

              The temporary workround is to reduce the resolution of timers available to JavaScript (to make the cache differences harder to spot). A longer-term resolution will involve, amongst other things, changing the JIT compilers to emit code that includers speculative execution barriers (where relevant and available).

              1. Name3

                Re: Don't buy a new Intel based system for a while?

                Maybe we should interpret Javascript again, instead of doing JIT. Seems safer. And get rid of WebASM, no one is using it anyway, it's too new and as we see very insecure.

                1. PlinkerTind

                  SPARC immune to Meltdown

                  but susceptible to Spectre.

          3. RandSec

            Some AMD Clarity

            AMD is not affected by Meltdown, but Intel is affected and requires a sometimes crippling patch.

            AMD is affected by Spectre, in the same way as Intel, and both only have partial mitigation so far.

            AMD EPYC servers have hardware encryption for VM's, which means a Spectre attack can succeed and still get only encrypted data.

        2. Downside

          Re: Don't buy a new Intel based system for a while?

          No need to reverse engineer - speculative execution is not a secret, it's just hardware engineers don't have an appreciation of software and/or couldn't be bothered to clear down the registers on execution failure.

          For once you can't "fix it in software" - which, if you've ever worked for a co that designs hardware, is a big fat chicken coming home to roost.

        3. Giles Jones Gold badge

          Re: Don't buy a new Intel based system for a while?

          ARM and AMD are affected less. Plus you forget one thing about ARM, they don't make their own processors and therefore those affected can quickly fix the chips they use without waiting for a manufacturer to issue the new part.

        4. John Brown (no body) Silver badge

          Re: Don't buy a new Intel based system for a while?

          "Except, AMD and ARM designs are affected too -- which everyone seems to be ignoring. The initial reports that broke in the media said AMD and others were unaffected: this is not the case, as has been widely publicised."

          That's because everyone is conflating 3 issues. The first one is Intel specific, the other two affect other CPUs to varying degrees and likely more difficult to exploit.

        5. Anonymous Coward
          Anonymous Coward

          Re: Don't buy a new Intel based system for a while?

          What I heard was that Intel, and one type of ARM core, was hit by the Meltdown problem, Spectre hits everything, and there is a third AMD bug that requires physical access to the hardware to exploit.

          What I also heard was that Meltdown can be patched against at OS level, while Spectre needs everything patching.

          The stuff being published by the media is contradictory, and I am getting to the point where I don't trust anyone to know what they are talking about. It also looks as though labels such as Athlon and Phenom were used by AMD for products of the same line, depending on how many cores worked.

          Why should I trust anyone?

    2. Anonymous Coward
      Anonymous Coward

      Re: So how much will this throw Intels release schedule out by?

      Well, they released Coffee Lake despite knowing about it.

    3. Anonymous Coward
      Anonymous Coward

      Re: So how much will this throw Intels release schedule out by?

      Have there really been any CPU based speed increases since about 2000?

      1. d3vy

        Re: So how much will this throw Intels release schedule out by?

        "Have there really been any CPU based speed increases since about 2000?"

        Your asking if there have been improvements since 2000?

        The fastest CPU I could find released in 2000 was a 1.5Ghz AMD - Didnt have the model name but given the year I'd guess early Athlon... 32bit.

        I was working doing system builds in 2002 and remember some of the first 64bit CPUs coming in.

        ---

        So to answer your question... YES.

        You sound like you've been lucky enough never to have to borrow the "spare training laptop" at work.. you know, the one at the back of the cupboard... :)

    4. Giles Jones Gold badge

      Re: So how much will this throw Intels release schedule out by?

      This will motivate many to investigate AMD or ARM. You certainly shouldn't reward such major incompetence with repeat business.

      Pretty sure Intel will have devalued their offerings by ruining their reputation, they sure won't be changing a premium price for a while.

    5. d3vy

      Re: So how much will this throw Intels release schedule out by?

      "If it takes Intel 12-18 months of engineering and testing before they can release replacement architecture does that mean no CPU based speed increases before 2020?"

      Dont be daft theres a software fix now, this wont affect the schedule at all, If we assume that they are working now on the CPUs that we will be getting in early 2020 I dont think we will see fixed silicone until 2021/22.

      I'd put money on the new "FIXED" CPUs project not even being a dot on some project managers Gantt chart yet.

      1. Anonymous Coward
        Anonymous Coward

        Re: So how much will this throw Intels release schedule out by?

        ---> Dont be daft theres a software fix now, this wont affect the schedule at all, If we assume that they are working now on the CPUs that we will be getting in early 2020 I dont think we will see fixed silicone until 2021/22.

        I'd put money on the new "FIXED" CPUs project not even being a dot on some project managers Gantt chart yet.

        ----------------------

        There isn't a fix now, there's a performance degrading patch (in some instances seriously degrading).

        Until Intel release replacement Silicon that has a proper fix I wouldn't be surprised to see a 30-40 percent drop in sales.

        Corporate IT Managers will not order silicon with a known flaw (regardless of the patch) unless they absolutely have to, because people get fired over this kind of serious shit.

        1. Roo
          Windows

          Re: So how much will this throw Intels release schedule out by?

          "Corporate IT Managers will not order silicon with a known flaw (regardless of the patch) unless they absolutely have to, because people get fired over this kind of serious shit."

          Few of the folks making the purchasing decisions read the errata let alone wait long enough for the showstopper errata to be discovered. Errata such as ECC failures leading to undefined behaviour didn't stop or noticeably delay folks buying the last few gens of Xeon in for example...

    6. Hans 1

      Re: So how much will this throw Intels release schedule out by?

      does that mean no CPU based speed increases before 2020?

      Well, we have not really had a speed increase since 2014 on Intel anyway, in some cases lower TDP, yes, slightly higher clock speeds and 6% performance increase ... negligible ....

      I have a 5820K and will not get the 6800K, nor 7800X when the 5820K kicks the bucket ... Ryzen all way

  3. MysteryGuy

    Windows 7 may not use PCID capability.

    On the few Windows 7 64-bit systems I updated with the meltdown patch, none seem to show PCID optimization enabled. At least according to the MS Powershell Get-SpeculationControlSettings script.

    This is on systems which do have PCID and INVPCID capabilities. So, this may not be an option under Windows 7.

    1. Anonymous Coward
      Anonymous Coward

      Re: Windows 7 may not use PCID capability.

      That's right, it's only supported on Windows 8.1+. The only Linux distro I know of that supports it is OpenSUSE Tumbleweed, though I'm sure there are probably others.

      1. Paul Greavy
        Unhappy

        Re: Windows 7 may not use PCID capability.

        "That's right, it's only supported on Windows 8.1+. The only Linux distro I know of that supports it is OpenSUSE Tumbleweed..."

        Hmmm... I installed the patched kernel, rebooted and the workstation crashed while loading the desktop. Repeated three times with the same results, before rolling back to the previous kernel. So I'm not sure that Tumbleweed supports this, yet.

        1. Chemist

          Re: Windows 7 may not use PCID capability.

          "So I'm not sure that Tumbleweed supports this, yet."

          Well I get :

          dmesg | grep isolation

          [ 0.000000] Kernel/User page tables isolation: enabled

          and CONFIG_PAGE_TABLE_ISOLATION=y in /boot/config-4.14.11-1-default.

          Booting fine but I am in a VM for the moment

        2. Anonymous Coward
          Anonymous Coward

          Re: Windows 7 may not use PCID capability.

          Are you sure you don't have OpenSUSE Leap? Tumbleweed is running on the latest version of Linux kernel, 4.14 while Leap is still under 4.4 which doesn't include PCID support. Any distro that's running Linux 4.14 has support for PCID.

  4. Anonymous Coward
    Anonymous Coward

    broadcastify.com

    I wonder if that explains the outage at Broadcastify.com. They had some "system maintenance" the other day, and seem to be having sporadic outages ever since.

    So if I were to run a reasonably secure system, could I bypass the update? reasonably secure as in I (ostensibly) control all the software on it, as opposed to a AWS instance where the provider needs to protect the customers from each other?

  5. AdamWill

    so, er...

    So, uh. Intel said:

    "Intel said as much in its statement, claiming "any performance impacts are workload-dependent, and, for the average computer user, should not be significant and will be mitigated over time.""

    and then you said:

    "While most casual desktop users and gamers won't notice any prolonged slowdown, or any performance hit at all, people running IO or system-call intensive software, such as databases on backend servers, may notice the difference."

    So...where exactly are you claiming that Intel was wrong, or misleading? Or are you suggesting that "average computer users" are running "databases on backend servers"?

    1. AdamWill

      Re: so, er...

      Three thumbs down, yet no counterpoints. Interesting.

      I'm not saying Intel's exactly being *up front*, here. But the story is more or less accusing them of *outright lying*, without backing it up. Intel says the impacts are "workload-dependent" and won't be significant to "the average computer user". Which, sure, is a very spin-ny way of saying they *will* be significant to workloads which *aren't* those of "the average computer user". But it's not actually a lie, and nothing the article presents as evidence actually contradicts it. In fact, the article agrees with it, explicitly, in the quote I pulled.

      1. Dan 55 Silver badge

        Re: so, er...

        I humbly submit that the average computer user will be impacted (ugh) now by cloud and server outages and soon by the consequent rise in prices in cloud services.

        If/when there are more kernel updates with optimisations, prices won't drop.

        1. AdamWill

          Re: so, er...

          I'm not disagreeing with you. But what I was really commenting on here was the *journalism*, not Intel's quote.

          The Intel quote isn't new - it's been quoted and referenced tons of times, including in at least three Reg articles. So anyone who's paying attention has already seen it. El Reg has also already explained, more than once, how it's more than a tad disingenuous. So when El Reg prints a *new* article, includes the quote *again*, and surrounds it with text like:

          "The patches being put in place to address the Meltdown and Spectre bugs that affect most modern CPUs were supposed be airy little things of no consequence. Instead, for some unlucky people, they're anchors."

          that reads to me like El Reg is suggesting something new - either that they can now somehow show that far more cases are going to affected than we previously thought (i.e. there are, for some reason, more syscall-dependent workloads out there than had previously been realized), or that Intel had claimed that the fixes wouldn't significantly affect performance for *anyone*.

          Yet in the end the article does neither - it just adds some random field reports of what we essentially already knew, i.e. that there *is* a significant performance impact on syscall-dependent workloads.

          That's not valueless, but it doesn't really pay off such a dramatic introduction, or justify the "airy little things of no value" line. That's what I was questioning.

    2. Anonymous Coward
      Anonymous Coward

      Re: so, er...

      I think the issue was using a comma where a semi-colon may of been more appropriate. I certainly read it as most people are not affected, but people running intensive work may be,. I know often we are not treated as such, but I do think those working in DC environments are still "people".

      Also I believe Intel were trying to say it was negligible for home users and up to 30% (max) for others, whereas home users are showing some impact and DC users are showing even higher than expected problems.

    3. John Brown (no body) Silver badge

      Re: so, er...

      "will be mitigated over time"

      I wonder if its this bit? How can hardware faults be "mitigated over time"? Surely only software can mitigate the hardware fault by working around it. Sure, patches with more time spent on them might get more efficient, but that is still never going to mitigate the slowdowns caused by the hardware fault workaround. Maybe Intels "over time" reference is about people replacing hardware on a time scale?

      In other words, they can't get rid of the slow downs until the users buy new kit.

  6. AdamWill

    also weird

    Also, this is clearly silly:

    "Via Twitter, Francis Wolinski, a data scientist with Paris-based Blueprint Strategy, noted that Python slowed significantly (about 37 per cent) after applying the Meltdown patch for Windows 7."

    Python is a *programming language*. (OK, it can also mean a particular interpreter for that language, but it makes no practical difference; the performance characteristics of the interpreter depend on the code it's interpreting). You can write something in it that does a lot of syscalls, or something that does very few syscalls. It makes no sense to suggest that "Python", as a single thing, can be slowed down by a single amount by these changes.

    Also weird, the graph claimed to be "An example AWS CPU utilization spike after installing CPU flaw" - the change described as a 'spike' (which, uh, isn't a spike) appears to be on 2017-12-22, which is two weeks before all this stuff came out. I haven't seen any suggestion that Amazon was patching AWS in December. Also, the change doesn't look at all like something you'd expect to be caused by the Meltdown patch: it seems like the instance suddenly started "idling" at about 59% CPU usage for some reason, instead of idling close to 0%. I've no idea what'd cause that, but it doesn't seem to match the characteristics ascribed to the Meltdown fix at all.

    1. MacroRodent

      Re: also weird

      It makes no sense to suggest that "Python", as a single thing, can be slowed down by a single amount by these changes.

      Depends on its implementation. Python is an high-level interpreted language that may be doing a lot of things not explicitly written into the program code.. A wild guess: maybe the interpreter loop has code that occasionally queries the system time, which needs a syscall. Or polls some file descriptor state. I don't know if any of these is the case, but they are plausible. I guess I right now need to stop talking out of my ass, and go look at the actual (open source) code to see if I can find anything like that.

      1. MacroRodent

        Re: also weird

        Answering myself: had a look with a test program and strace on Linux. I did NOT find any extra system calls in the interpreter loop. So there is NO intrinsic reason why Python should slow down more than similar code written in other programming languages.

        1. Richard 12 Silver badge

          Re: also weird

          My wild guess is that python is interpreted and so always loads a lot of files (source dependencies) during startup and possibly during runtime.

          A compiled application tends not to load as many files or dynamic libraries (other than the system dynamic libraries, which presumably python also needs anyway)

          1. Anonymous Coward
            Anonymous Coward

            Re: also weird

            >My wild guess is that python is interpreted<

            Python is always compiled. The rest of your comment was correct: python applications often consist of a number of library files.

            1. MacroRodent
              Boffin

              Re: also weird

              Python is always compiled.

              Depends on what you mean by compiled. There are actually multiple Python implementations, but the most commonly used (the one from www.python.org) compiles into an intermediate "bytecode", and then interprets that.

    2. martinusher Silver badge

      Re: also weird

      I was under the impression that these bugs aren't concerned with syscalls but turn on the processor speculatively executing instructions in advance of a jump. Instruction execution in a high performance processor is a complex activity, the processor doesn't execute one instruction at a time but rather is in the process of processing several instructions at any instant. Since a branch will require the processor to dump the work in progress the designers add in duplicate register sets so that they can keep the instruction pipeline going regardless of whether a branch is taken or not. The design flaw centers around the access checks which are made only the instruction being executed -- completed, as it were -- rather than when it starts collecting the data the instruction needs to use. If the data being accessed by a particular instruction is in a protected area it won't trigger an exception unless that execution path is taken so this leaves a neat worm hole where you can get the processor to access areas of memory that it shouldn't without anyone noticing. The rest of the exploit is coming up with ways to deduce what's in that memory location. This is all a bit esoteric for most people -- they've got enough on their plate getting their code to work without having to go into the minutiae of exactly how the processor goes about doing its work.

      I personally think that we've asking for trouble for years by using highly optimizing compilers and complex processors to cover for writing inefficient code. This is why I'm not surprised by the variability of impact of the patches -- I'd expect purpose built, well coded applications to suffer little to no impact while I'd expect a lot of interpreted code, especially of the "hose it at a barn wall and see what sticks" type, to suffer quite a lot.

  7. Anonymous South African Coward Bronze badge

    What will it take for Amazon et al to create their own, secure CPU?

    Or chuck out their current CISC infrastructure and switch over to a RISC-based one? (unlikely).

    1. Richard 12 Silver badge

      AWS customers want x86-64, so moving to ARM is unlikely.

      Unless AWS customers start demanding ARM of course.

    2. Lysenko

      Or chuck out their current CISC infrastructure and switch over to a RISC-based one? (unlikely).

      Given that we know ARM is impacted by these security flaws as well, the CISC/RISC calculation remains unchanged - unless you're suggesting they port the whole of AWS to MIPS.

      1. dbannon

        "Or chuck out their current CISC infrastructure and switch over to a RISC-based one? (unlikely).

        Given that we know ARM is impacted by these security flaws as well, ..."

        Ah, but apparently Itanium is not affected.

        (I remember the HP rep telling me it was the future of (HPC) computing, he'd know ....)

    3. Anonymous Coward
      Anonymous Coward

      "What will it take for Amazon et al to create their own, secure CPU?"

      A lot of time and money?

      Given all the main CPU supplier failed one way or the other - with Intel failing any of them - it's not exactly a simple and cheap task to build a high-performance high-security CPU.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like