back to article How I poured a client's emails straight into the spam bin – with one Friday evening change

By misunderstanding how a single word was being used, I caused a boo-boo that counts as "really stepped in it this time". After a lot of research and testing, I thought that months of "the spam filter is crap, make all the spam go away" warring with "the spam filter is too restrictive because $client can't send me his …

COMMENTS

This topic is closed for new posts.

Page:

  1. Disgruntled of TW
    Pint

    Friday ...

    ... I mean really? You did this on a Friday?

    RFC1925 should have an extension to outlaw all PROD changes on a Friday.

    Lol.

    1. Trevor_Pott Gold badge

      Re: Friday ...

      Do it on a weekday and they'll have your hide if anything goes wrong. Do it on a weekend and there's not enough traffic to make it go 'ping'. Do it on a Friday, right after EOB and you have a few good hours of decent incoming traffic flow, a handful of folks who work late and are used to minor changes and an entire weekend to fix things if you bork them really badly.

      1. petur
        FAIL

        Re: Friday ...

        "a handful of folks who work late and are used to minor changes"

        Over here, people working late on Friday are people on a tight time budget or in enough sh*t to bother to work late, and who will be double pissed if the infrastructure goes titsup...

        1. Trevor_Pott Gold badge

          @peteur

          I try to discourage people working late, or on weekends. I have few enough maintenance windows as it is. If you work during off hours, well, I have no sympathy. There isn't a 24/7 global team of nerds to implement changes and patch things. So we have to sleep some time. If I have to be up for the 9-5 grind, then I'm not waiting until 3am to patch.

          Besides, some folks start getting in a 4am...

          1. Anonymous Coward
            Anonymous Coward

            Good time to work on live

            Lucky for some, all of the companies I have worked for the last 15 years were 7 days a week. The current one is 24x7 but without the budget for fully online redundant systems, which makes patching, updating and reconfiguring more than a little bit difficult.

            We have a comms room that really needs taking down, re-wiring and racks re-stacking, but there is never going to be a good time to do it as it will take about 8 hours.

          2. petur

            Re: @peteur

            "I try to discourage people working late, or on weekends"

            You make it sound as if people have a choice... The times I have been at work on a Friday night beating a deadline were not my choice, and I think this counts for most. Yes, we all want to get home on a reasonable time and get some sleep, so there's no good solution here, but some respect please.

      2. Vic

        Re: Friday ...

        > an entire weekend to fix things if you bork them really badly.

        And a skinful of beer and frustration to ensure you do...

        Vic.

    2. AndrueC Silver badge
      Stop

      Re: Friday ...

      RFC1925 should have an extension to outlaw all PROD changes on a Friday.

      As a programmer I have a general rule not to make anything 'live' after mid-afternoon and by preference do it at start of day. I sometimes work with colleagues in the USA and it's not nice to go home and inadvertently leave them with a pile of poo to resolve. It pisses them off and embarrasses me.

      It's much nicer to push things at start of day so you know it's been exercised before you quit for the day. I hate that feeling of impending doom when something has gone live and no-one else has tried to use it yet.

      1. Trevor_Pott Gold badge

        Re: Friday ...

        For a full "this is live and will stay that way", I agree. For a pre-permenant, data-gathering exercise that needs to run on live...this I prefer on the Friday EOB. Remeber, the goal here was not a permenant run, just a very brief test on live with just enough traffic to find bugs.

        Found one.

    3. tony2heads

      To quote that wise sage Harry Nilsson

      Duit On Mon Dei

    4. JeffyPoooh
      Pint

      What is this "Email Spam" of which you write?

      Never see any such thing. Never. Never ever. Not one. None, Zip. Zilch. Nada. Less than a quanta.

      Signed, a Gmail user.

      1. Gordon 10

        Re: What is this "Email Spam" of which you write?

        Dont you mean

        "Signed a stupidly trusting of Google user"

      2. This post has been deleted by its author

      3. Irongut

        Re: What is this "Email Spam" of which you write?

        What do you call all those emails from Google telling you what's new on YouTube, Play Store, Play Music, Play Video, etc, etc, etc. There is plenty of spam on GMail and most of it comes from Google.

        1. Gene Cash Silver badge
          WTF?

          Re: What is this "Email Spam" of which you write?

          What "emails from Google"?? I have had a gmail account since the year after they came out, and an Android phone since the original Droid and never had any such.

          1. James O'Brien
            Thumb Up

            Agree with you Cash

            Unless he is using his gmail account to sign up for anything and everything I would like to know how he gets them as I have had Gmail for slightly longer than you have and never see those emails.

            One way I avoid that shit is to have a burner account like Hotmail. That is all I use on sites which want my email information.

      4. Tim Ryan
        FAIL

        Re: What is this "Email Spam" of which you write?

        Then again you are using the mail service that simply doesn't do NDR's so you never know what you didn't get and your senders never know that it didn't get there. FAIL bigtime

    5. ElectricRook

      Re: Friday ...

      In my organization making even a minor change on Friday would get you canned outright. Thursday was highly despised, Wednesday not so hot, Monday a poor choice because that is new hire day and the affected customer might not be too sure of what caused the changes.

      Tuesday right after lunch . . . most preferential.

  2. lansalot

    actually..

    Your biggest mistake was flipping the switch and walking out the door leaving it unattended all weekend. For something so critical, that was a huge fail.

    "A bit of testing" never works out the same once the users start clattering it. For something as mission-critical as email, it's something to do and stick around to watch for a good while.

    1. Trevor_Pott Gold badge

      Re: actually..

      I watched it for about two hours. Nothing bizarre jumped out at me. I figured if something was going to go splonk, it would do so in a two hour timeframe. Guess I was wrong.

      1. big_D Silver badge

        Re: actually..

        As they say... Never assume, because it makes an ass out of U and me...

        1. James O'Brien
          Devil

          Re: actually..

          Or if you go by BOFH definition of "Assume' it makes an ass out of you and you

      2. NogginTheNog

        Re: actually..

        Of course I'm preaching to the choir, but your big fail was in the testing phase: you simply fed it some 'live' data when you really should have taken some time to look at the different types of data the system would encounter (ie. emails from multiple sources, destinations, mailing lists, attachments, hyperlinks, the works!), and then put together some suitable tests that simulated as much of those varied cases as possible. And that's before you start considering any form of peak load testing.

        1. Trevor_Pott Gold badge

          Re: actually..

          I did. I fed it simulated data for days. Of course, the one thing I hadn't thought of was that the X-SPAM-HEADER info would be a problem, so the simulated data all had X-SPAM-HEADER data of either "yes" or "no".

  3. Nick Ryan Silver badge

    There is an attachment to the idea of Outlook + Exchange + Public Folders that no force in the universe is ever going to dislodge.

    Microsoft is working very hard on this.

    1) They've been steadily depracating Public Folders with every release of Exchange and Outlook (including refusing to fix decade old bugs) in favour of... sharepoint.

    2) It's cloud, cloud, cloud, cloud all the way. Or, more accurately, subscription services under Microsoft's control.

    Luckily, Microsoft hasn't been entirely successful in killing off Public Folders yet.

    1. PaulCoote

      We have plenty of clients which are very attached to public folders, and many of them have been successfully moved to 365. Now you can convert your public folders to public folders in 365. There are a couple of caveats in sizing, which can mean they get split across multiple shared mailboxes. If the client is used to Exchange why move them to largely unfamiliar Google Apps, just move them 365 and have unlimited mailbox and public folders. The users are largely unaware of the change if the project is done properly.

      I am yet to find a client that can run Exchange onpremise cheaper than the 5.25 monthly fee for unlimited mailboxes and full retained backups (legalhold) which the 365 Exchange 2 plan costs.

      1. Kane
        Windows

        @PaulCoote

        You are a MS shill, and I claim my £10 gift voucher.

      2. Trevor_Pott Gold badge

        You get an exchange licence and 10 free CALs with your Action Pack. That's about $400/year. When you already have to have a virtual infrastructure to deal with all the other stuff you do....yeah, it's cheaper.

        Internal IT doesn't exist just to support one app.

  4. Anonymous Coward
    Anonymous Coward

    Quote to long to put int title

    My personal preference would be to punt the entire kit and caboodle into Google Apps and be done with it

    And I stopped reading there !

    1. Anonymous Coward
      Thumb Down

      Re: Quote to long to put int title

      "And I stopped reading there !"

      Because of an expressed preference and opinion, that doesn't even relate the rest of the article? Must be too much effort to use that crowbar to open your mind in the morning.

      1. Seanie Ryan

        Re: Quote to long to put int title

        didn't stop reading at that point, but my opinion of the author shot through the floor. Hope he has good insurance cover.

        Presuming he is in the UK/EU, you could land in hot water if you recommend that. Depending on your clients business, you could be contravening EU Data Protection Laws as Google App, Office 365 etc are owned by US companies, so the data, even if its on an EU hosted server, is covered by the Patriot Act.

        Bit of leaked data, and be sure someone will point the finger and say " He said to do it"

        I think there are some negotiations going on to circumvent the issue, if anyone in the know could provide more info…..?

        1. Anonymous Coward
          Anonymous Coward

          Re: Quote to long to put int title

          Must confess that Mr. Pott took a bit of a credibility hit there with me as well. Gmail may be easy; but putting business comms through a foreign advertising company doesn't strike me as particularly bright; even before you get to any legal and data-belonging-to-clients-of-the-company issues.

          Some of my clients do do it; and that's down to them...but I would never recommend it as a course of action. Quite the reverse, in fact.

        2. Trevor_Pott Gold badge

          Re: Quote to long to put int title

          I'm not in the UK/EU. I'm in Canada. And for this class of customer you cheerily can put them in Google Apps without consequence.

          1. Anonymous Coward
            Anonymous Coward

            Re: Quote to long to put int title

            Google is also a foreign advertising company from Canada's viewpoint. My personal view is that there is no good reason for using Google for email in any circumstances; and that goes triple for business use. I'll freely admit that I'm a bit religious on the subject; but you have no idea what they're doing with the data now; let alone what will happen in the future. The facts of the case are that Google is a business and they are there to make money...not to be nice to people.

            You may be happy exchanging a loss of privacy for ease and convenience; but I am not. Everybody's mileage varies. I wouldn't do it for my own email -trivial or not- and I absolutely would not recommend it for any class of business user.

        3. Anonymous Coward
          Anonymous Coward

          Re: Quote to long to put int title

          Re: Data from Europe held in the US, for all apart from very specific data sets, there's no issue holding personal data on US servers if the company has signed up to Safe Harbor.

          See point 6 on http://ico.org.uk/for_organisations/data_protection/the_guide/principle_8

      2. Anonymous Coward
        Anonymous Coward

        Re: Quote to long to put int title

        Because of an expressed preference and opinion, that doesn't even relate the rest of the article? Must be too much effort to use that crowbar to open your mind in the morning.

        No Not at all. It was Ahh Fuck it it would be easier if i could just dump it on somebody else.

        Anything goes wrong its not my fault.

    2. Irongut

      Re: Quote to long to put int title

      Should have carried on AC, laughing at an idiot explaining the depth of his idiocy is always fun.

  5. Alex Rose

    Hosted AS?

    I'm a bit confused as you seem to imply that the client should really be looking to host their email system in the "cloud" yet seem to insist on trying to homebrew your own anti-spam system.

    There are any number of really decent hosted AS systems out there that cost no more than a few dollars per mailbox per year. We currently run about 2000 mailboxes (across a number of clients) on a couple of different providers and I would no more think of trying to homebrew AS than I would AV to be honest, life's too short :)

    Unless your client insists on AS in-house? In which case you need to get your sales person's hat on!

    1. Trevor_Pott Gold badge

      Re: Hosted AS?

      Hosted AS is ultimately where I want to go. The history is as follows:

      1) Until recently, hosted AS was along the lines of "a few dollars per user per month" not "a few dollars per user per year." Which is more than the client would pay.

      2) Until recently, relatively simple in-house open source AS systems worked just fine.

      3) Having used the simple open source AS systems for so long transitioning away from them takes time. The existing system, for example, injects [SPAM ASSASSIN DETECTED SPAM] into the subject, rather than adding X-SPAM-STATUS

      My goal is to get them using an in-house AS system that uses X-SPAM-STATUS for the rest of the year and then have them transition to a hosted AS system at the end of the year. This will be possible because both the system I'm trying to deploy for the in-house option and virtually all hosted AS systems use X-SPAM-STATUS.

      Now, getting them to accept hosted AS will require getting them accept paying a subscription for an AS service when they're used to using free in-house stuff AND getting them to overcome their innate paranoia regarding having their e-mail hit servers in the states. I honestly don't know if I can "sell" that...and I'm pretty sure I don't care enough to try.

      What I can do is get them migrated to a solution that uses X-SPAM-STATUS instead of subject injection which will make the transition to a proper hosted AS a heckofalot easier in that mythical future when the decide to just pay the tithe like everyone else.

      That's the goal, anyways...

      1. Sampler

        Re: Hosted AS?

        How much your time in hours per annum spent on 'homebrewed as' vs how much outsource costs - that should make your argument pretty easy, as in why didn't we do this yesterday.

        Also, make sure not to underestimate how much time you actually spend on this, generally I find when you sit down and look at it what you might write of as a couple of hours quickly adds up over the year (with unexpected work like outages included).

        1. Trevor_Pott Gold badge

          @Sampler

          Oh, I tried that argument. I believe the response was "so we can take the cost of the hosted AS out of your salary?"

      2. Vic

        Re: Hosted AS?

        The existing system, for example, injects [SPAM ASSASSIN DETECTED SPAM] into the subject, rather than adding X-SPAM-STATUS

        That's trivially implemented by using clear_headers to remove the X-Spam-Status header and rewrite_header to inject a tag into the subject.

        Whether or not that is a good idea[1] is up to you...

        Vic.

        [1] It isn't.

        1. Trevor_Pott Gold badge

          Re: Hosted AS?

          @Vic; I'm on the long path to getting rid of a decade's worth of bandaids and nudging the client along towards a proper (though significantly more expensive/year) IT setup ahead of leaving. It's a long fight.

    2. Anonymous Coward
      Anonymous Coward

      Re: Hosted AS?

      They have the same issue of an outsourced mail server - all your company emails are routed through an external system you have to trust more or less blindly. An AS appliance on-premises could be better, if you have not the resources and expertise to setup up your own AS pipeline.

      At least what is good with SMTP is it is an end-to-end (from a server perspective) protocol, it doesn't need routers or relays, unless you want them.

      A homebrew solution is not that difficult to setup, and if you really hate Exchange you can simply setup another MTA of your choice in front of it, configure spam filters there, and then forward to the Exchange system. Just you could lose the Exchange-to-Exchange capabilities.

  6. Anonymous Coward
    Headmaster

    Speaking of language........

    Tut tut Trevor

    I could cheerfully sent email from my home address after reverting email from the new anti-spam

    Correct conjugation of the verb send please !

    1. Trevor_Pott Gold badge

      Re: Speaking of language........

      >_>

      <_<

      :(

      //sads

  7. Olius

    DailyWTF?

    Did you mean to post this on TheDailyWTF.com?

    Two major WTFs and one minor:

    1. Trying to reinvent the spam filtering wheel

    2. Putting something live on a Friday - let alone last thing in the day (Live = in the morning, near the start of the week)

    3. (minor) Trying to make MS do something complex. Next time try Exim ;-)

    1. dan1980

      Re: DailyWTF?

      Gotta be honest here - I find Fridays are very suitable for some changes. It all depends on the client, the work to be done and the circumstances but in some instances, it is the best option. Failing to even consider an option because of some personal rule is surely the bigger crime?

      Indeed, "never makes changes on a Friday" sounds very much like those who admonish all and sundry that they should stick to white paper solutions (to the letter) or not bother. Without knowing the client, their usage patterns, 'risk appetite' or indeed a whole host of variables, such proclamations are on the same level as "migrate to the Cloud" - i.e. blanket assertions of suitability that, when blindly followed, can lead companies and their IT departments down some very steep roads.

      IT is so varied and we are all at risk of falling prey to our own biases. That your rule has worked for you so far in the situations you are used to is great but really not an indication of if it is suitable for another situation - even one that looks, on the surface, to be similar.

      The one thing I would say, however, is that changes to live systems generally require monitoring real-world use (traffic/transactions/disk access/etc...) for a period after the changeover. If that is not possible on a Friday night, or you are not able/willing to stay up late to do so then you need to find another way. If you are willing/able/paid enough to do so, then there is no inherent reason why a Friday change-over should be off the table.

      Pairing with that, I would implore people to not underestimate the variability of real-world inputs and to make sure they have a solid understanding of the true scope, volume and variability of what the system is asked to deal with on a daily basis. But then, that's important whether you do your work on a Friday afternoon or a Tuesday morning.

      1. Anonymous Coward
        Anonymous Coward

        Re: DailyWTF?

        Indeed, "never makes changes on a Friday" sounds very much like those who admonish all and sundry that they should stick to white paper solutions (to the letter) or not bother.

        I first picked up the "No Production changes on Fridays" mantra from fellow techies - it's us who have to spend our weekend fixing stuff if it goes tits up!

        1. dan1980

          Re: DailyWTF?

          @AC

          Again, it depends on the situation but, for some scenarios, I would much rather have a weekend to sit, relatively un-molested, and fix a SNAFU than deal with clients breathing down my neck.

          Again, it's all very much dependent on you, your clients and exactly what you are trying to achieve and I can only speak for myself but having to spend your Saturday fixing a problem before anyone notices can be preferable to spending a Tuesday trying to make up new and interesting variants on "it'll be done when it's done!"

          d.

Page:

This topic is closed for new posts.

Other stories you might like