@The Original Steve
Well, you asked for details, and here they are.
The e-mail is protected by a forcefield consisting entirely of a CentOS VM. This VM runs ClamAV, spamassassin, Pyzor what-have-you. It filters and scans all inbound mail, and then passes it along to the exchange server. It drops any mail it finds contains a virus, and marks spam with [SPAM ASSASSIN DETECTED SPAM] in the subject. The exchange server (through some powershell jiggery-pokery) dumps every e-mail it sees with that tag in the subject into the user’s Junk E-mail folder. The Junk e-mail folder cleans itself every night, deleting anything older than some preset age whose number I cannot remember off the top of my head.
When it is working, this works like a HOT DAMN, letmetellyou. It’s free, stupendously simple to set up, and catches bloody-near everything. So how in the nether fnord did this little gem squeak by?
Well that would be the fact that CentOS is complete and utter PANTS at keeping a halfway up-to-date install of clamAV in it’s repositories. Of course my system is set to auto-yum-update certain packages every night, and run freshclam twice a day…but eventually the good folks who run ClamAV decide your version is too old, and they stop supplying new defs. Since CentOS updates ClamAV with a frequency similar to that of planetary glaciation events, this means that eventually the time will come when my ClamAV is too old, and it Just Doesn’t Work Anymore. This spamserver has since been replaced.
To cope with this, my latest spam server is based not on CentOS but on the latest Fedora, (at the time of last build, 12,) with the addition of the RPMforge repository. (http://dag.wieers.com/rpm/FAQ.php#B). This is because RPMForge do me the favour of almost keeping up with ClamAVs spectacular update pace. We will see if this new one is any better than the old one.
What was running on that user’s desktop that allowed the little zipped ball of yuck to walk right into his system and pwn the living crap out of it? Both NOD32 and Microsoft Security Essentials. Don’t get your hopes up that any other scanner is any better; I have similar horror stories for AVG, Trend Micro, McAfee F-Secure, ClamWin, Kaspersky BitDefender, Avira, G-Data and of course Symantec. Useless, the lot of them. You have to keep them around in the vain hope theywill at least let you know that you’ve been pwned, but frankly it’s safe to just rebuild every six months on general principal.
As to migrating to Windows 7; that project is currently underway.
Now, understand that these are all preventable issues. I should have been all over that spam server the instant the clamav defs refused to update. (Naughty me for not configuring the cron job to e-mail me the results. I get a slap on the wrist for that.) I also should have run around with a cluebat telling people DO NOT OPEN E-MAILS WITH ZIPPED ATTACHMENTS CLAIMING TO BE FROM COURIER COMPANIES. (Okay, actually I did that several times, but really you have to do it on a regular basis or they forget.)
There are other things I should have done; regular threat sweeps, actually checking my damned IDS software…you know the drill.
The truth is…we missed it. There are three of us, (myself, a senior sysadmin and a junior sysadmin.) We’re looking after a network with 130+ server VMs, 60+ Desktop VMs, 60+ physical (non thin-client) desktops and 45ish physical servers. We deal in data volumes measured in Terabytes of network traffic a day across the internal network. Peak external traffic input to the network topped 100GB a day in 2009. There are scads and scads of industry-specific software (and even hardware) to deal with, as well as some pretty demanding functionality requirements handed down from They Who Sign The Cheques.
I am the head of IT, but the company has a CTO and a CEO who both have some reasonable say in IT policy. I have to design the network, make sure it’s implemented, maintained, secured and all the other things. The other sysadmin I work with is fantastic, and our bench tech knows his stuff. (He’s learning the trade, and even taking on junior sysadmin tasks.)
You get one project “working,” but not “fully complete and polished,” and you toss it on the line because it needs to be in service *EFFING NOW* and then move on to the next pile of combusting feces on your desk. By the time you even think about the first project thrown into service at 80% completion, it’s been MONTHS, and there are not only configuration changes that need to be made, but it’s now a critical service item, and you have to schedule your downtime as well. (That gets interesting when you have 5 networks in 4 cities stretched across 3 provinces covering 1900km. Oh, and there are roaming users based in cities all over Canada.
Somewhere in there I also have to deal with vendors, prototype new systems, vet patches and system updates, design and maintain websites and intranet services, keep the printers and the phones and the blackberries and the gods-only-know what else running…
So the really short answer is; In all the cloud of things to do, I simply missed the warning signs, and neglected parts of my network. I could sit here and list for you eleventeen squillion things wrong with my network. They are like an infinite number of needles in my eye; I know my network very well, and that includes knowing what’s wrong with it as well. I could spend three months just locking down the desktops, or cleaning out the AD. I could spend a year cleaning up the Linux estate, and doing All The Things That Should Be Done.
Truth is though, I’d never ever in a million years get caught up. SO at the end of the last year, I decided enough was enough, I’m burning the entire network down and replacing it. Every server, every desktop. The research I have put into this, the testing, the vetting, the experiences of it all are beg documented, so that I can share my experiences with those who might benefit from what I have learned. Some of it I have (and will be) documenting on my personal Blog, (http://www.trevorpott.com) but the majority of it ends up here. Somehow someone decided that they had had enough of me running around the comments section of this website shooting my mouth off at everyone and everything, and I should instead be putting my experiences to use for the benefit of Vulture Central.
And that’s how you get El Reg’s desktop management blog.