back to article CPU bug patch saga: Antivirus tools caught with their hands in the Windows cookie jar

Microsoft's workaround to protect Windows computers from the Intel processor security flaw dubbed Meltdown has revealed the rootkit-like nature of modern security tools. Some anti-malware packages are incompatible with Redmond's Meltdown patch, released last week, because the tools make, according to Microsoft, “unsupported …

  1. redpawn

    SOP

    The only way to keep your computer safe if you are using competing products is to break your computer. You knew from past history this could happen yet you use non MS products.

    1. Steve Davies 3 Silver badge
      Holmes

      Re: SOP

      to break your computer. You knew from past history this could happen yet you use non MS products.

      Applies to MS products as well. How many people have had their Windows 10 system refuse to boot after the application of [cough-cough] secutiry updates supplied by MS themselves?

      None of us are safe from having 'borked' systems. Sad fact of life today.

    2. Anonymous Coward
      Anonymous Coward

      Re: SOP

      I've just put my computer through a company-wide domain migration. Initially it looked okay, but as time goes on more and more things are crawling out the woodwork.

      The domain migration portal/ActiveX control is an MS product, by the way.

  2. Valeyard

    Logic

    "if you're not using an antivirus" (presumably as then the registry isn't set by it) "then you won't get security updates!"

    That's just setting systems up to fail in ways worse than meltdown and spectre, I just can't get my head around that logic at all

    1. Charles 9

      Re: Logic

      Simple. It's a CYA move. If they force the issue and business-critical computers get bricked as a result, companies lose money and Microsoft can face a lawsuit as a result. At least an un-updated system can still run, and if they're not in a position to update when they get pwned, then that's Intel's fault, not Microsoft's.

      1. streaky

        Re: Logic

        You'd have to actively disable things to get into this state, namely defender. Sure defender is garb, it wasn't I don't think ever supposed to be that great but I'm sure defender will set the key (edit: Microsoft do explicitly list it on their spreadsheet). Defender will only disable itself, and will always disable itself if you're running other AV.

        If you've specifically decided to go naked as it were, it's kinda your own fault and you should be paying attention to things like this. As for lawsuits, you're the one making the positive step not Microsoft, there's no liability here.

        1. Danny 14

          Re: Logic

          sophos enterprise manager had a message for me the same day saying they have set the registry key. not all AV companies were dodging things.

          no blue screens here on 1709 w10. we use a mix of i3 and i5s

        2. mr.K
          WTF?

          Re: Logic

          "If you've specifically decided to go naked as it were, it's kinda your own fault and you should be paying attention to things like this."

          Come on! How is it my fault that line of products that I do not use are faulty? And how exactly show I be paying attention to things like this.

          I am a fairly simple man running a fairly simple system. I do simple things and manage to keep it tidy. I run no AV because they do more harm than good and I have got infect two times. Once around 1993 from 1.44 floppy disks with pirated games, and once in 2004 when I foolishly tried to patch a fresh win2k install on an open network. I manually apply all updates because I want to know my system and I need it to behave in a predictable way. There is no superfetch running and all activity on the hard drive, i.e. the blinking light, I can account for. I do however keep software updated and even though I have come to distrust windows update since it is used for advertisement and spyware, I have yet to experience it holding back on updates.

          So I am supposed to seek out the local Microsoft office or what to get the news? Displayed in the basement perhaps? The basement without stairs or lights. Where there might be a notice on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying "Beware of the Leopard".

          1. streaky
            Alert

            Re: Logic

            I can't tell if this is serious or not ^

    2. Anonymous Coward
      Anonymous Coward

      Re: Logic

      Well Windows comes with Defender, presumably that sets it to 1. It would only get set to 0 when installing a 3rd party AV, so that the check can be made to re-set it.

    3. big_D Silver badge

      Re: Logic

      If you are running a modern system without AV, you have taken a conscious choice to remove it - either MS AV or the crud that was supplied by the PC supplier. Therefore you should be taking a care of the system through other methods and therefore should be keeping an eye out for such security problems.

  3. Duncan Macdonald

    What is difficult about setting a registry key ?

    If the AV supplier does not want to add registry key setting to the main product, just spawn an administrator level command prompt that runs regedit to set the key.

    As the key basically says "AV is ok for Meltdown and Spectre patches" - what concern is it of the AV supplier if the system has a third party incompatibility.

    1. Mark 85

      Re: What is difficult about setting a registry key ?

      As the key basically says "AV is ok for Meltdown and Spectre patches" - what concern is it of the AV supplier if the system has a third party incompatibility.

      Lawsuits... the ambulance chasers are already afoot. If you're big user of Windows (say... FMC or GM) and all your computers BSOD, or they decide not to patch and get hit by the bug, I suspect their lawyers will be looking for blood and lots of it.

    2. a_yank_lurker

      Re: What is difficult about setting a registry key ?

      By the time some heavy hitters with massive numbers of Bloat boxes get hammered by various failures they will sue everyone they can. Touch the registry and you might be tossed into the mix. Chipzilla and Littlezilla will definitely be named, Slurp and other OS suppliers will probably be named, security software peddlers will probably be named, etc. The further down the food chain the easier time you will have but you will still have to defend yourself initially. Many initially named will wiggle off the hook.

      1. DailyLlama

        Re: What is difficult about setting a registry key ?

        "Chipzilla and Littlezilla will definitely be named"

        Chipzilla and Chipzooky, shurely?

        1. Jay 2

          Re: What is difficult about setting a registry key ?

          Ha! Haven't thought of that for a while, for good reason! I always hated the mid-80s requirement for cartoons to have some sort of younger, "edgy", annoying-as-fuck character such as Godzooky or (the worst offender) Scrappy Doo.

          I'd like to think that The Simpson's use of Poochie in Itchy & Scratchy has hopefully stopped this sort of thing forever.

          1. Teiwaz

            Re: What is difficult about setting a registry key ?

            the mid-80s requirement

            I think mid-80's is a little late - pretty sure they were all earlier than that - yup, Godzilla was 1978-1980, the droids added to the Westernised Battle of the Planets (Gatchaman). Scrappy Doo was 1979.

            By the time we got to the 90's there was a return to form with the 'Steven Speilberg' produced series.

            I doubt some brain dead TV exec will commission some dross again, or some 'concerned mothers' pogrom will kick up a fuss again somewhere down the line.

            1. Asylum_visitor

              Re: What is difficult about setting a registry key ?

              They don't have a 90's, 00's or 10's iteration of Scrappy/Godzuki. It is much, much worse...

              https://youtu.be/K05N2jqFHc8

    3. Anonymous Coward
      Anonymous Coward

      Re: What is difficult about setting a registry key ?

      "they'd have to modify the product, maybe substantially"... please, tell me how s***y your installers are?

      Well, I've seen applications that are unable to update their installations, and to be "safe", they uninstall the previous version and then install the new one. Hundred of megabytes.

      Probably because they outsourced setups to people without a clue about them.

      1. Anonymous Coward
        Anonymous Coward

        Re: What is difficult about setting a registry key ?

        Not defending all installers here, but I actually *like* having this option. It reassures me that I'm not building layer upon layer of crud. For example, which would you rather have, a fresh install of Office 2016 or one that started with Office 2010, got upgraded to 2013 and then landed on 2016? The amount of cruft left on such a system would be terrifying.

        Wireshark is one well-respected tool that does exactly this, and I've never had problems with it.

  4. pdh

    Something I've been wondering about... There are people who run multiple AV products. Suppose AV #1 is OK with the patches but AV #2 is not. If AV #1 sets the key, will AV #2 proceed to brick the system?

    1. Anonymous Coward
      Anonymous Coward

      Something I've been wondering about... There are people who run multiple AV products. Suppose AV #1 is OK with the patches but AV #2 is not. If AV #1 sets the key, will AV #2 proceed to brick the system?

      AV #2 will brick the system. I mean it's not like AV #1 can cancel out what AV #2 going to brick.

      Also color me impress if you run multiple AV and managed not to brick either AV and your system at the start.

      1. Anonymous Coward
        Terminator

        I've seen...

        Mcaffee frigates off of Vista. Quarantine processes fired from Bulldog. I've seen AVG self destruct when it detected Kaspersky Labs. It was beautiful. Time to die now I think?

    2. eldakka
      Holmes

      That was exactly one of the points covered in the article:

      SentinelOne is upset that "the responsibility of setting the registry key" is shifted to the AV vendor. "While our testing revealed no incompatibilities, we are unwilling to take on the risk of setting this registry key,” the security software house said.

      “This is because our customers may have other software products that use unsupported/undocumented APIs that are incompatible with Microsoft’s latest patches. In such a case, our customers may experience stop errors/system instabilities caused by other products that are not compatible with Microsoft fixes,”

    3. streaky

      There are people who run multiple AV products

      This kind of thing is extremely bad practice, most people who work in security and AV vendors have been telling people to not do it for at least a decade, at least as far as active protection goes. Race conditions playing around in kernel memory space is bad juju.

      Just don't do it.

      Now having AV soft where one does your active protection and another that can scan but actively protecting the system is kinda viable, the answer in that case would be yeah, you better hope that the one you have doing the protection is the one that is compatible. If it isn't..

      1. Charles 9

        "This kind of thing is extremely bad practice, most people who work in security and AV vendors have been telling people to not do it for at least a decade, at least as far as active protection goes. Race conditions playing around in kernel memory space is bad juju."

        Isn't placing your trust in ONE vendor who by nature can't catch everything ALSO bad juju? This sounds like a Catch-22. You either choose one and lose when something slips through or try to avoid monoculture and get bricked when they clash.

        1. streaky

          Isn't placing your trust in ONE vendor who by nature can't catch everything ALSO bad juju?

          The chances of you getting hit by a virus or malware that a reasonably competent AV vendor hasn't accounted for and another has and you happen to have picked the right AV vendor combination to cover that venn diagram is almost nil - in fact it is nil. If you're a target for the CIA, usually you'd know and frankly you should be taking precautions like, I don't know, maybe not so much with the running of the Windows. Also, yeah, therein lies the trick when choosing AV.

          Each to their own, but running two AV products at the same time isn't really viable, potentially it could do more damage than malware could.

  5. sloshnmosh

    Win 7/Microsoft Security Essentials

    I have several computers with different configurations, I have a Windows 7 machine running with Microsoft's Security essentials on it that already had this registry key set before the infamous update. Another computer with Windows 7 that has Trend Micro installed needed this registry key added.

    (Trend Micro created a .reg file for users to download to add the key)

    I don't know when the registry key on the Security Essentials machine was added but I know it existed before I installed the Meltdown update.

    1. Naselus

      Re: Win 7/Microsoft Security Essentials

      "I don't know when the registry key on the Security Essentials machine was added but I know it existed before I installed the Meltdown update."

      I doubt it; this is a brand-new, never-previously-existed reg key. MSE may have had a recent update that told it to add the key if no other AV was present on the machine?

  6. Anonymous Coward
    Anonymous Coward

    Only M$

    is allowed to make “unsupported calls into Windows kernel memory,”

    1. phuzz Silver badge

      Re: Only M$

      Surely if MS do it then it's a supported call into kernel memory?

      1. Naselus

        Re: Only M$

        "Surely if MS do it then it's a supported call into kernel memory?"

        I suspect that actually depends which bit of Microsoft makes the call. The Office team probably shouldn't be writing software that reaches into the kernel all the time (which they did quite a lot in the 90s and early 2000s).

      2. Martin an gof Silver badge

        Re: Only M$

        Surely if MS do it then it's a supported call into kernel memory?

        Wikipedia

        Starting in the 1990s, Microsoft was accused of maintaining "hidden" or "secret" APIs: interfaces to its operating system software that it deliberately keeps undocumented to gain a competitive advantage in its application software products

        M.

        1. Anonymous Coward
          Anonymous Coward

          Re: Only M$

          @martin an gof, at last someone who wasn't born yesterday, I too remember this little plum

          The obvious answer is to download the patch as an exe disconnect the network, uninstall the AV install the patch then reboot and see if you can put the AV back on again. If you already have infections on your machine then nothing has changed and if it bombs then you can legitimately say I didnt have a AV installed it's all your fault Bill

          It is nice to see MS finally recognising the value of covering your own arse after years of blase "just keep restarting it until it works"

  7. Anonymous Coward
    Anonymous Coward

    As annoyed as I am that they dropped Linux support, Avira was one of the first to be listed as compatible with the fixes. Though being more than 10 years since I last had a virus, I'm wondering how necessary they are beyond mitigating user screwups.

    1. Naselus

      "Though being more than 10 years since I last had a virus, I'm wondering how necessary they are beyond mitigating user screwups."

      Yes, it's been more than ten years since I had flu, so I'm sure I don't need to keep going for my annual flu shot anymore....

      1. Anonymous Coward
        Anonymous Coward

        Yeah, cos that's the same. Do you get regular tetanus shots? How about rabies? You can get flu even without coming into contact with someone who has it, you can't get tetanus without coming into contact with infected soil or rust, you can't get rabies without being bitten by an infected animal.

  8. Anonymous Coward
    Anonymous Coward

    More proof of how poorly Windows is and how inept Microsucks is at fixing their defective crapware

    It should be obvious that Microsucks is the one who has erred with their rushed-to-market patch top try and mitigate the financial disaster that is Intel CPUs. The WinTel Cabal has finally succumbed to their own incompetence and greed. Now consumers are faced with replacing all Intel PCs or suffering the consequences. Everyone using Windows is going to be penalized for Intel's serious violations of good security practices with their defective CPUs sold over the past decade plus.

    1. Charles 9

      Re: More proof...

      Um, given that Linux needs to be patched, too, why does Microsoft get the blame for problems of others' making?

  9. Anonymous Coward
    Boffin

    Useful

    This has provided us with a handy list of badly written AV tools you don't want anywhere near your PC.

    1. Ken Hagan Gold badge

      Re: Useful

      They are all "badly written", by design. This is just a heads up at the sort of shenanigans they have been getting up to all these years. AV tools are an invasion of your kernel internals by someone who doesn't know enough about your kernel and cannot respond to implementation changes in a timely fashion and if they get it wrong then your entire system is tanked and you might as well not own a PC.

      1. Jo_seph_B

        Re: Useful

        Agreed. How many times has an AV vendor caused a blue screen, plenty. I'm sure Sophos have had a couple of issues before. Once it even identified its own update engine as a virus and deleted/quarantined its own update files thus rendering it useless.

  10. Anonymous Coward
    Anonymous Coward

    Could someone explain how anti-virus is accessing the kernel if it's in a place you couldn't access it until these problems with the CPU happened? If the issue is calls now the kernel has moved then those calls shouldn't have been possible in the first place.

    1. Ken Hagan Gold badge

      You have it the wrong way round. The kernel *was* previously visible to these tools (though undocumented) and so the AV folks reverse engineered enough to learn how to hack into it. Now it is no longer visible and the same hacks fall into a black hole and bring down the system.

    2. Anonymous Coward
      Anonymous Coward

      AV do install drivers which give them access to the kernel - intercepting processes and I/O calls needs to be done at this level. These communicates with userland code, and because coding everything into a kernel module is difficult, if not impossible, they usually make data available to the userland part for processing. It looks they've been caught doing it in ways not compatible with these patches.

      1. Anonymous Coward
        Anonymous Coward

        Makes sense, thanks.

  11. Anonymous Coward
    Anonymous Coward

    Running AVG

    Was fixed back in December.

    McAfee however - still broke.

    1. Anonymous Coward
      Anonymous Coward

      Re: Running AVG

      I think I spotted a typo. Should it be "McAfee however - broken since forever"?

    2. TheVogon
      Headmaster

      Re: Running AVG

      "McAfee however - still broke."

      I'm pretty sure McAfee is solvent. Or did you mean broken?

  12. Anonymous Coward
    Anonymous Coward

    Isn't is it bad practice to run more than one AV at once SINCE ALMOST 3 DECADES? Who the f... does that???

    1. Anonymous Coward
      Anonymous Coward

      Depends - if you just run the file/processes inspection part on demand it's OK. If you let them "infect" your system with all their drivers and services to hook whatever they like automatically, there's a good chance they start fighting against each other.

      On a file server or a mail server, for example, you may want to inspect file/attachments using more than one AV engine, the way VirusTotal works. Relying on a single one may be dangerous.

      1. Ace Solo McCloud

        For email (before the days of the cloud), you would get a email filtering product that could include multiple engines, BitDefender, Kaspersky, etc. however it's one main product installed to the system, utilising the multiple engines it has licensed from the vendors.

        It's also possible... to deliver email/web through multiple servers. Each one with a different AV / Filtering product that checks/cleans before delivery to the final mailbox server.

        File servers, you'd have AV on the server and on the client, you should be concerned about entry points, email, web, USB stick / CD ROM and ensure each has it's own protection, in combination should have as well protected a network as possible.

        Running multiple products on a single server/PC, not a good idea, unless most are passive/manual and only one "active"/installed/running in the background otherwise it will slow down/crash the system running two or more full AV "installed/active" products.

      2. oldenoughtoknowbetter

        And those making use of some "next gen" AV.

        Cylance, Crowdstrike, etc. can often be used along with other AV, or even together.

        Not uncommon, and can add complementing functionality.

    2. Blotto Silver badge

      @AC

      the people who run multiple AV are likely the lot who installed AVG free and then someone sold them another AV / they got one from their bank / Which told them about some other recommended product / some random bloke down the pub who cleans at a tech company recommended 1 etc but have no clue how to install the old one. They are likely the same people who have malware.

      it happens more often than you think on consumer and small business pc's.

      1. Camilla Smythe

        The people who run multiple AV are likely the lot who have Google Chrome as their default browser.

        1. Anonymous Coward
          Anonymous Coward

          "are likely the lot who have Google Chrome as their default browser."

          Yes, if they wanted fast they would be using Edge.

    3. DropBear

      Ummm... people who click "yes", I suppose? To something as innocuous as a Flash plugin update that used to be essentially forced upon them by browsers and such until not so long ago (assuming they already had a different AV installed)...?

  13. jigr1969

    I've regularly seen computers running more than one instance of anti-virus software, the most I've come across running on a laptop, was four!!

  14. mark l 2 Silver badge

    part of the problem is that you must have AV on a Windows box if you want to use it anywhere near the internet, but because it is closed source the AV vendors have no choice than to try to fudge things to get them working.

  15. JeffyPoooh
    Pint

    "...QA testing...", Symantec...

    No such thing.

  16. MrBoring

    A reg key question.

    Is the detection built into the patch executable (KBxxxx.exe) so does it fail to run, or is it just WIndows Update that detects it?

    So if you approved the patch in WSUS or roll it out via SCCM do the clients install it anyway even if the reg flag isn't set?

    1. Naselus

      The clients don't install unless the reg key is set, regardless of whether it's coming from WSUS, SCCM or Windows Update.

    2. Duke`

      You can safely approve it in WSUS and it just won't be made available to the machines until they have the correct registry entry. As soon as it sees that (we just pushed the reg change via group policy as we know our AV is fine), the update will be available to install as per normal. Its the same for Windows Update - it just won't appear until the reg change. You can however download the patch manually and install it but at the risk of it causing a BSOD if your AV isn't compatible.

  17. David Austin

    Anyone Else Surprised...

    Symantec wasn't ready from the Get-Go?

    Anyone?

  18. Gnoitall

    What the AV tools have been doing

    An AV tool that can't accept the Meltdown patch is an AV tool which has been silently exploiting the Meltdown vulnerability for as long as it's been "drill[ing] deep into the kernel's internals in order to keep tabs on the system."

    In other words, the AV software have been the exploiting malware we've all been worrying about, and have been for years.

    1. Naselus

      Re: What the AV tools have been doing

      Not really, no.

      Meltdown and Spectre are bugs which use out-of-order execution to gain unauthorized access to kernel data in userland. AVs do not do that. AVs do rely on being authorized to read elements of the kernel in userland, but having elements of the kernel in userland is not actually a problem - in fact, it's highly desireable, as you get speed increases of up to 4000%. Unless you can use out of order execution to read code without authorization, of course.

      Basically, because Meltdown allows unauthorized access to the kernel in user space, and the problem is fundamental to processor design, the solution has been to remove kernel data from user space altogether. This means that programs which rely on AUTHORIZED access to kernel data in userland now do not work. The 'fix' for Meltdown is not actually a fix - it's a kludgy workaround to try and remove the circumstances that allow the bug to happen, rather than actually resolving the bug itself. Its also why we're getting massive performance reductions - not because every program ever has been exploiting the bug, but because most of them used user-side kernel data to speed them up.

  19. Stu 18

    ooo Pretty big job, gonna cost ya.

    Two thoughts, sorry if I'm repeating someone....

    1. Re one of the paragraphs, writing a registry key is going to require significant code work - yes that is what the text suggested, give me a break, a lot of software that is written by what I consider poor practise writes hundreds if not thousands of registry entries apparently for random fun and leave it behind when it is removed. Writing a single key is a very basic call. (I do realise re-writing your AV kernel driver thingie - now that might be hard).

    2. The alternative to a single key is a folder where each vendor software has to put in a "i'm good" notification and if everyone is good, updates continue.

    Having said that, with this registry thing, malware has a very simple way to switch off updates at will, simpler than borking windows updates - which lets face it Microsoft bork their own update engine regularly enough.

    So perhaps what we should have is a better way to register applications existence on the machine in a protected place such that they can be checked against an online 'master' record of goodness maintained by vendors / Microsoft. Heap of missed opportunities there with the whole 'certified' application certificates etc.

    This is turning into a bigger poo pile every day as we better understand the consequences!

  20. Stu 18

    Why can't...

    Clearly my basic reading of the issue needs some assistance;

    What I understood is that due to processor running future commands making the assumption that they will run before they are officially called, and the fact that the results of those commands can be read, we have the meltdown issue.

    Why isn't the answer to clear that pipeline and results whenever there is a kernel exception of the type that asks for privileged access? Or do kernel programmers use exceptions as their normal execution logic?

    I expect I'm being too simplistic and it will need a pipeline to track the pipeline...

    1. Anonymous Coward
      Anonymous Coward

      Re: Why can't...

      My understanding is that the problem has two parts. It's started by exploiting the speculative execution, true. But the processor does flush the invalid CPU instructions after it notices the exception. However, the processor doesn't revert any changes made to the caches, and for some reason the userland code is able to access the data in the cache directly.

      1. David Nash Silver badge

        Re: Why can't...

        Eben Upton posted a good explanation of how it all works on his blog while explaining that the RPi is not vulnerable:

        https://www.raspberrypi.org/blog/why-raspberry-pi-isnt-vulnerable-to-spectre-or-meltdown/

  21. JohnG

    On this issue, I'm with Microsoft: it isn't Microsoft's responsibility to check if every third party AV product has not done silly things to the OS that would make systems fail when the Meltdown patch is applied. MS have simply said to the AV companies "This is what the Meltdown patch does. You know what you products do, so you are the ones to decide whether your products are compatible with the Meltdown patch". Short of not issuing any patch, I don't see what other choice MS have.

    1. mr.K

      Us without AV

      They could have at least given the end user information about this through windows update. I was wondering why there wasn't any update yesterday and whether I happen to read an article in The Register or not isn't something I should have to rely on to keep my system updated.

      1. Charles 9

        Re: Us without AV

        You're assuming the Stupid User is in a position to understand this stuff.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like