back to article Sophos Windows users face black screens after false positive snafu

Users of Sophos’s security software were confronted with a black screen on starting up their Windows PC over the weekend as the resulted of a borked antivirus update. The botched update meant that the Windows 7 version of winlogon.exe was incorrectly labelled as potentially malicious, resulting in chaos and confusion all …

  1. Doctor Syntax Silver badge

    From the Graham Cluley blog: "To its credit, Sophos issued an update at 9am UTC on Sunday, fixing the false alarm."

    Which isn't a great deal of use if you can't log on. I hope none of my friends and family are using Sophos.

    1. Tom Paine

      Presumably the auto-updater runs as SYSTEM rather than under one specific user account -- it'd be pretty useless otherwise. Therefore you don't need to logon to fix the problem, or shouldn't need to. (Note I don't run Sophos, I'm just assuming their design is approximately sane.)

    2. energystar
      IT Angle

      AV issues are not going to stop...

      Bottom of the stack is reclaimed territory, by OS Houses. On behalf of a healthy ecosystem this issue should be - finally (after 30 years) - negotiated.

    3. Anonymous IV

      From the blurb about Graham Cluley in the linked article:

      "Graham Cluley is a veteran of the anti-virus industry having worked for a number of security companies since the early 1990s."

      No mention of the fact that those security companies included, for several years, Sophos itself!

  2. James O'Shea
    Black Helicopters

    Tinfoil hats on

    How much did Microsoft pay Sophos to cause problems for Win 7 holdouts? And this will surely not be the last 'accident'...

    1. James O'Shea
      Trollface

      Re: Tinfoil hats on

      I see that the MS fanbois have no sense of humor.

      1. Steve Davies 3 Silver badge

        Re: Tinfoil hats on

        re Sense of humour

        It was surgically removed when they joined the MS Borg.

        Along with the last remnants of their common sense.

        1. Dwarf

          Re: Tinfoil hats on

          re Sense of humour,

          They probably lost it due to all the issues they have had to endure by their favourite supplier doing nasty things to their chosen OS version.

          The rest of us were just smart enough to say enough is enough and do something about it, now we're smiling again.

          I'm sure they will catch up when they hit their pain threshold too.

      2. hplasm
        Coat

        Re: Tinfoil hats on

        No need- the joke is in the OS!

  3. frank ly

    Silver Lining

    "The problem was limited to users running a specific version of 32-bit Windows 7 SP1, according to Sophos."

    Microsoft can now target them with exhortations to upgrade to Win 10. Every little bit helps.

    1. Anonymous Coward
      Anonymous Coward

      Re: Silver Lining

      Seriously? What drugs are you on? Why is every bug and mistake out there always part of some hidden conspiracy?

      1. Anonymous Coward
        Anonymous Coward

        Re: Silver Lining

        Maybe something to do with the whole Windows 10 upgrade shenanigans? Or are you just asking why it would be hidden as Microsoft are normally a lot more blatant?

      2. James O'Shea
        Devil

        Re: Silver Lining

        "Why is every bug and mistake out there always part of some hidden conspiracy?"

        'cause it is, dammit.

      3. Doctor Syntax Silver badge

        Re: Silver Lining

        "Seriously?"

        See James's second post above.

      4. frank ly

        @AC(1)Re: Silver Lining

        Seriously?: No.

        drugs?: Caffeine, nicotine and pork pies.

        conspiracy?: No, marketing opportunity - serendipity.

        silver lining: sly reference to the previous 'tinfoil hat' posts as well as the old saying.

        summary: Whoosh.

      5. not.known@this.address
        Trollface

        Re: Silver Lining

        {sad attempt at a joke}

        You are either a new Windows user or a Micro$haft shill and I claim my free downgrade to Windoze 10

        {/sad attempt at a joke}

        (Troll, cos I prefer it to the 'Joke' icons...)

  4. Anonymous Coward
    Anonymous Coward

    I still remember the great Sophos cock-up of September 2012

    It borked almost every updater of hundreds, if not thousands of programs including itself.

    Therefore once infected with the Sophos update you couldn't re-update to a safe version. You also couldn't remotely uninstall as it wasn't working, or re-install a new version. Most of the programs on individual PCs were borked, some proved almost impossible to repair without a re-image - very inconvenient for laptop users and non-standard image users.

    Overall it took 3 days of overnight work to get many PCs working again, a week to get slightly stable and months before all PCs were back to how they should be - although there was still the odd issue that was never fixed.

    The best part - how could an AV company have such a serious upgrade issue that affected even it's own upgrade service. Apparently it went through 5 different checks which all should have picked up the issue - but they all failed! There was no false-positive testing on an actual Windows machine, the false positive testing was done on a linux box.

    https://www.sophos.com/en-us/support/knowledgebase/shh-root-cause-analysis.aspx

    It was this, that made me never use Sophos again, and since then I have been using Bitdefender with no major issues.

    Hearing this news makes me glad that I chose to move away.

    1. Jo_seph_B

      Re: I still remember the great Sophos cock-up of September 2012

      I remember this too. Mostly because at the time I managed an estate of 5000+ PC's and ran Sophos on all of them. Took months to sort it all out.

    2. anothercynic Silver badge

      Re: I still remember the great Sophos cock-up of September 2012

      *cues hysterical laughter* What, and Bitdefender is better? The law of averages will get Bitdefender too at some point.

      *Every* AV vendor goes through this one, although I'm surprised that Sophos has suffered this *twice* in the last 4 years, given the massive cluster their 'update' SNAFU caused last time and they promised to fix their procedures. Clearly the procedures are *not* fixed.

      Shame, shame, shame...

      1. Anonymous Coward
        Anonymous Coward

        Re: I still remember the great Sophos cock-up of September 2012

        Whether an AV provider will have false positives is not in question. The difference is their Q&A. Any update should not bork their own software or any standard set of software including the operating system itself.

        I don't mind (too much) if a piece of niche software, or an ancient version of well known software gets quarantined - that's to be expected. However for Sophos to kill about 50% of software on the system including. catastrophically, its own executables is unforgivable. So is killing a core Windows executable.

        This leads me to believe their fundamental management and systems are not working properly. No-one knows exactly what the processes and flaws in other providers are until they have a similar issue - Bitdefender hasn't, so far, while I've been using it. If they have something similar then I will also be leaving them - it's a matter of trust.

        1. anothercynic Silver badge

          Re: I still remember the great Sophos cock-up of September 2012

          Hence the 'shame, shame, shame' because it's clear that management has *not* learned from the last catastrophic cockup. Or they learned, forgot, and well, you know what they say about forgetting the past, right?

          But yes, *anyone* who buys into a software package in the belief that 'it'll *never* happen to them, surely' is naive at best. Sorry, I'm extremely cynical when it comes to sales pitches. :-)

    3. David Austin

      Re: I still remember the great Sophos cock-up of September 2012

      Oh God: That was about six weeks of cleanup across the customer base. On the plus side, after a few (Strong and loud) words with my account manager, most of them didn't pay for Antivirus for three years after that, which is precisely how much most of them thought it was worth...

    4. Halfmad

      Re: I still remember the great Sophos cock-up of September 2012

      I remember it as we'd just switch over to a new, crappier AV two weeks prior. Never in my life did I think I'd be glad we were using Mcafee!

  5. alain williams Silver badge

    ''All vendors suffer them from time to time''

    Really ? I have not noticed it with RedHat, Mint and other versions of Linux. Am I just lucky ?

    1. lglethal Silver badge
      FAIL

      Re: ''All vendors suffer them from time to time''

      The author quite obviously means AV vendors. I don't think RedHat, Mint or Linux are AV Software are they?

      1. Pascal Monett Silver badge
        Trollface

        Re: I don't think RedHat, Mint or Linux are AV Software are they?

        Well, given that 99.999% of all virii target Windows, in a sense, they are.

        1. Glenturret Single Malt

          Re: I don't think RedHat, Mint or Linux are AV Software are they?

          Viruses is the plural of virus, like omnibus (bus), callus, grampus etc. etc.

      2. Anonymous Coward
        Anonymous Coward

        @Iglethal - Re: ''All vendors suffer them from time to time''

        In case you didn't get it, actually all Linux distros ARE anti-virus software.

      3. OtotheJ

        Re: ''All vendors suffer them from time to time''

        Thanks for reminding me:

        Linux = Great

        Windows = Not So Great

        For a moment there, I thought I was reading an article and comments about an Anti-Virus vendor.

    2. shanen

      Did you forget about this recently?

      http://www.theregister.co.uk/2016/02/21/linux_mint_hacked_malwareinfected_isos_linked_from_official_site/

  6. jms222

    winlogon.exe is presumably signed by Microsoft so why exactly doesn't AV software respect this ?

    1. James O'Shea
      Mushroom

      Son, you must be new here. I remember when [name of AV app redacted, they have much better lawyers than coders] ate SVCHOST. That was a fun fix.

    2. Steve Knox
      Holmes

      winlogon.exe is presumably signed by Microsoft so why exactly doesn't AV software respect this ?

      Might have something to do with the fact that signing keys have been stolen in the past. Might have something to do with the fact that signing keys don't even have to be stolen to infect signed files:

      http://www.theregister.co.uk/2016/08/08/stealthy_malware_infects_digitallysigned_files_without_altering_hashes/

      Basic defense-in-depth principles require that the design of each layer must assume that the other layers may have been compromised.

  7. splodge

    It's about time AV systems hit the root cause...

    1. Alumoi Silver badge
      Trollface

      It's about time AV systems hit the root cause...and nuke any and all Windows partitions from orbit.

  8. Tom Paine

    Early this century I had a gig as a QA engineer at a well known AV vendor. They had just built out a big rig to check for false positives, as they'd recently suffered a similarly embarrassing cock-up leading to a core Windows DLL being quarantined as infected "with hilarious consequences". The very first test apps to be loaded onto the FP environment were every version of Windows then around, including the shiny new Itanic version of XP. I find it really hard to understand why a reasonably large vendor like Sophos wouldn't have something similar in their update build / test / release chain, in 2016. It's not as if it's very expensive, in the grand scheme of things, you could pick up a copy of probably 80% of code in use for, oh, well under £50k I should think.

  9. Joseph Haig
    Coat

    What does it do?

    I guess that winlogon.exe is related to logging a user into Windows. Yes, I can see how that would be considered malicious.

  10. a_yank_lurker

    Sophos may be right

    Has Sophos finally figured out the be biggest malware supplier is Slurp? (Ducks from incoming barrage of fanbois hate)

    1. Stephen 11
      Big Brother

      Re: Sophos may be right

      Right idea, wrong OS... Surely they should be targeting winlogon.exe in Windows 10 as malware, rather than Windows 7...

  11. Mikel

    Solving the wrong problem

    All antivirus is solving the wrong problem, attempting to recognize an infinite variety of patterns - which is, of course, impossible. The need for this arises because the operating system itself is laughably insecure and attempts to run everything from everywhere by default.

    The correct solution to this problem is to use an operating system correctly designed and configured to not take candy from strangers. Then antivirus is not needed.

    1. P. Lee

      Re: Solving the wrong problem

      Have an upvote to couinter-act the presumably robo-downvote.

      OS design is the problem. Windows is also not the only culprit. Until people are willing to swap performance/features for solid security, things won't change.

      The OS needs to mediate all resource access, not just disk, ram allocation and CPU. We need finer-grained control over applications that simply, "which user is it running under." That control needs to include declared network/url access, disk access, it needs to be set at install time and vendors need to take the lead in providing secure, not just functional permissions. There is no good reason for flash running in a browser to have access to every file the user owns. Rights should be inherited. Running a PHP interpreter from a browser should mean it gets the browser's rights which may be different from running the same thing from a desktop shell. Along with virtual memory, how about providing each application with virtual disk and a virtual network proxy? The OS filesystem should be read-only. OS and application binaries and configuration should not be co-mingled. Binaries and config should not be co-mingled. Applications should not be running random executables to update themselves - the OS should be checking known repositories for updates.

      All this AV stuff is a fudge and a massive performance drain. Has anyone tried measuring the performance hit of AV vs doing OS design right?

      1. Stephen 11
        Meh

        Re: Solving the wrong problem

        While there is always room to make OSs more secure, it is possible to configure Windows to be reasonably secure in pro/enterprise editions with group policy... it is just not convenient, so the majority of users never do. Software Restriction Policies is a powerful feature, but sadly is under utilised.

        A large part of this falls into security versus convenience problem...

    2. David Hicklin Bronze badge

      Re: Solving the wrong problem

      AV currently works on the basis of "blacklist" the bad stuff - allow anything else.

      To get your solution means turning it on its head and whitelist what is allowed and block everything else. It can be done as long as everything executable on the computer is stable and does not change very often - and that needs planned and tested updates.

      Sadly not exactly practicable now unless you wall the OS off from all other software creators/vendors and only allow certified software from a "store"....oh wait..

  12. hmarc

    Any publicity is good publicity right?

    I don't really believe that title is true although, I do still highly recommend Sophos Home, I run it on all my Macs and PCs (never used the Linux or Android offerings). In exchange for a couple of details they give home users some pretty impressive security tools and software - including their cloud based AV, and it's all free of charge.

    I can't imagine this 'outbreak' being a very significant problem - a business or user home user who is running Sophos AND a 32-bit version of Windows 7 SP 1. That seems like a very niche and shrinking market, further deluding these wild conspiracy theories.

    Unsurprisingly maybe, I didn't immediately see anything mentioned on their security blog/website (that I won't mention by name here).

  13. Duffaboy
    Trollface

    Are you sure

    It was a False postive

  14. Anonymous Coward
    Anonymous Coward

    Yep, hit our company this weekend...

    Ended up having to drive in in order to confirm it was a dodgy update and then to put an specific exclusion in the enterprise console in order to fix.

    Anon because I rather like my paycheck.

  15. Tommyinoz

    Antivirus is worse than viruses

    Does anybody remember the big virus outbreak that affected Linux desktop and server machines?

    Me neither.

  16. Medixstiff

    Trend Application Control is worse.

    Half our fleet needs replacing with freshly SOE'd machines, it doesn't uninstall itself properly - about 25% of installs keep the service running which means the problem of it slowing systems to a crawl whilst scanning the partitions (this is on Core i5's with 8GB's of RAM and 250GB SSD's for god's sake) is still occurring - we've had to replace multiple machines with freshly imaged ones and all we hear back is it's with the developers.

    Although if it came down to using Trend versus any Symantec product, we'd stick with Trend any day.

  17. Anonymous C0ward
    FAIL

    Testing, motherf*cker

    Do you do it?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like