Posts by Trevor_Pott

Re: What's with the Google fascination?

I'd agree with you, for organizations willing to invest in the full stack. Exchange needs more than just exchange to get the benefits you speak of...and that stack needs a dedicated full time admin. Not an admin who is also doing storage, networking, applications, desktop support, websites, Linux, etc.

It was one thing to be the gneeralist who lumped in "and exchange" back in the Exchange 2000 or 2003 days. It's another thing entirely to try to keep up with e-mail today. Even for "basic" MTAs, there is so much to configure, and so many "conventions" on configuration you have to abide by to stay off greylists that it's crazy.

I agree exchange is amazing. I rather like it for many things...but only in cases where you're willing to pay the tithe. That means proper hosted AS. It also means keeping up to date on clients and all ancillary applications that tie into it.

As a unified communications stack, Exchange/Lync/Sharepoint/etc can be very powerful. But they aren't wrapper-ware and they aren't particularly good past their "best before" dates.

Where exchange truly shines is in things like retention rules, archiving, and all related stuff. If you need to do things like legal holds, in-depth content scanning, Exchange is pretty goddamned hard to beat.

The problem is that most companies absolutely don't need that stuff. They never use it, but they're sold on the idea that they "need" etiher the top-end collaboration stuff or the in-depth retention/legal policy framework, despite never actually wanting to engage any of it.

Worse, you sometimes get a CIO who thinks it's all really, really cool and wants everyone to use it, but simply can't get buy-in from the staff. Usually they'll try everything, including outright threats and bullying, but the staff have non-technological ways of communicating and getting things done that are simply faster and far more efficient for them.

The biggest thing I see with my SMBs is people wanting to use the full Microsoft stack to be "more efficient" at communications because one or two people (who typically telework for some or all of their day) feel "out of the loop." They try to impose a technological solution on a human problem and it fails every single time. The problem isn't that people don't use the relevant technology, it's typically that they're an asshole, or that they simply choose not to give a fuck about $issue until it there's a problem.

Exchange isn't - and can't be - a replacement for human beings taking responsibility for their actions, taking the time out to think about the various projects that needs be done, or actually taking the time to answer the various and sundry e-mails and communications that need answers. Making communications "more efficient" doesn't force people to actually acknowledge one another, keep eachother in the loop or convince the powers that be to make a fucking decision about something.

It absolutely doesn't force overworked people to sort their crap and "properly file" digital data. If you have problems with people using a single public share as a catch-all wastebin where they store everything "because everyone has access and it's more convenient" then public folders and/or sharepoint are just going to look the exact same. The issue there is the people, their habits and their workload, not the technological tools available to them.

When and where exchange can make a difference, I absolutely champion it's use. Exchange is one piece in the best groupware and productivity stack on the planet. Period.

But I do not champion it's use in most SMBs. I think that's ridiculous overkill. Hell, even Office 365 which is designed to be simple to administer (compared to Exchange) and offers only a subset of features is something where 98% of all SMBs I've worked with that use it simply don't change anything past defaults.

So, while I think Exchange is grand, I can't and don't recommend it for SMBs, unless the SMB has a definable need for it and they're willing to pay for it. Regular updates, proper amounts of sysadmin time, proper hosted AS and enough server licenses and hardware to make it all go.

I will never do another exchange install that doesn't have Exchange Enterprise Cal Suite for each user and hosted AS. There will also be a minimum of three server licenses involved: one dedicated hub transport server and at least two storage servers in a cluster. They will also be backed up using Data Protection Manager and monitored using System Center.

The floor cost for this is simply higher than most SMBs are willing to pay, to say nothing of the ongoing costs of keeping it ticking along.

Here's a great example: try running Update Rollup 3 if you'd disabled IPv6. Whole thing goes pear-shaped. Worked fine without IPv6 until then, then *bam*, implosion upon update. There are various reasons why IPv6 had to be disabled in one of the environments. Update happens along, murders exchange. Figuring out what went wrong, then applying the fix takes a proper sysadmin.

Ideally, you never encounter the error because everything exists in a test environment, all patches are vetted, etc. How often do you think that happens in an SMB where you don't have things like "dedicated Microsoft communications stack admins" or even "dedicated Microsoft admins?"

And so we get to the heart of it: Exchange is an example of a service that should never be run by an in-house SMB sysadmin. It needs to be outsourced. If you are going to run Exchange in-house then the sysadmins should have access to an MSP with a hell of a lot more experience, time and resources to do proper labbing of patches for that SMB's config and so forth. It is an application in a stack for which specialists should be used.

...or where it makes damned good sense to simply pack the whole thing up and go "cloud".

If Microsoft had "Office 365" for service providers and/or could make their own offering reliable enough that it isn't constantly experiencing outages, I'd say "use O365 service provider" and be done with it. MS refuses to release O365 to SPs and it can't keep it's own version working.

That leaves me with Gmail as the most stable offering for SMBs, followed by the more expensive hosted Exchange (assuming you can meet the floor cost), or simply hosted e-mail using open source MTAs without all the groupware faffery.

But the issue, 99.9% of the time isn't that "groupware will magically make things better." It is that there are bigger business and communication issues that need to be dealt with that no software can make better.

Anywho. Long ramble...

Re: Too much info in too few X-Spam headers

Actually, I have to disagree with you here. The reason for moving towards an X-Spam-Status header is that it is an industry standard. If the system is set up to accept these then it can be used with AS devices or services from any number of providers. Not all providers allow you to change the headers you are working with, so X-Spam-Status makes the most sense to stay with.

Now, the ability to change the old server to pop it's stupid BAYES info into a different header, that would be great...

Re: OPE vs FOPE . . .

Oh, I lied. The MPN support people only solved part of the problem within 32 hours of picking up the ticket. They fixed the part that was preventing me from generating quotes for new seats. They didn't fix the licensing issue with my MAPS. They *just* e-mailed me about that.

This makes it 48 hours to pick up the ticket and we're 72 hours past that point without the ticket fully resolved. And the ticket in this case not being some niggly complex technical problem, it's a billing/administrative issue that stemmed from a years-back uncaught authentication system screwup on their side.

I.E. the damned thing autogenerated me an Office 365 account without informing me, then assigned my MPN account to it. I was then able to create another Office 365 account that was somehow also attached to the same MPN account, but which couldn't get at the partner section, but which would accept my MAPS keys.

They fixed the bizzare double-attachment bit 32 hours after picking up the ticket, but solving the "regenerating me a new MAPS key" part of the ticket is two days past that and counting...

I think I'd have more confidence in Office 365 - which, from a technical standpoint is actulaly quite a good solution - if only authentication ever fucking worked. MPN never works the first time. Even straight-up .onmicrosoft.com Office 365 IDs never seem to work, requiring me to login two or even three times, sometimes requiring a log-out in between. There's something about session cookies they can't every get right.

Beyond that, I have all sorts of issues with Azure Active Directory. Sometimes it says it works, but isn't. Other times for reasons incomprehensible it just stops working, despite nothing having changed (and no reported outages on the MS side.) This makes hybrid setups very frustrating.

Microsoft is so close. Their hybrid solution will one day be the solution. But to be perfectly honest it's another 1-2 years away from being ready for primetime. Maybe when Server 9 comes out, they'll have added in the bits required to make it go reliably.

Re: ^^3 day SLA?

I get 48 hours for responses to queries calling the MS partner support network. Then up to 32 hours for them to fix it. I get similar responses for average customers with E1 and E3 licenses. Multiple events now, same timeframes for each.

Re: Its a regex !

It can use regex?

...must test this.

Re: Your mistake was...

Thanks for that. In addition to everything I have to remember about the hundreds of applications I manage, I'll just run along now and memorize every PowerShell command. It's not like money was paid so that there would be a reasonably easy to use and modestly intuitive GUI. Nope, rote memorization of more data than a human mind can actually hold for every application is absolutely the best possible path forward for systems administration.

Re: Precise language and vague language

There is no "exactly matches" condition. You would have to do "matches" then "except when". Still working on the details.

Re: OPE vs FOPE . . .

The Office 365 advisor program and I are having a disagreement. Specifically, I've been fighting with MS for the past five days to even make my bloody partner page work. MPN and O365 both hate me. I hate them right back in turn.

Office 365 is something I'll revisit when they A) beef up the reporting to levels that aren't complete ass. [Insert 8-page reporting rant here]. and B) Make the fucking thing work. When Microsoft can achieve Google Apps levels of uptime, we'll talk again.

As for SPLA; fuck SPLA. I refuse to host Exchange in my cloud. The hosted e-mail I offer my clients is Qmail, Postfix or Zimbra, front-ended by Barracuda and/or Netgear UTM. OPE can be got by the customer for their own site...but it's more expensive than competing solutions and not as good.

Re: What's with the Google fascination?

Oh, I just really, really hate exchange. E-mail in general, but exchange in particular. Loathe it with the burning passion of 10,000 suns. Most of my clients you Google Apps, Zimbra or a hosted exchange solution (that I don't have to manage, hee hee!)

If there's e-mail to manage I just want it to be a nice IMAP server. Postfix + Dovecot on virtualmin works like a hot damn. Or Qmail. For the love of $deity, why can't I just use Qmail? But no; exchange! Exchange, destroyer of souls. Exchange, the eraser of sanity. Exchange the requirer of resources 80x that of any other MTA.

And the cloudy alternatives? Well there are Linux-based IMAP mails...but I could run those in house, if allowed, with no real problems. There Google Apps with Just Works and works better than any hosted e-mail solution I've ever used. And then there's Office 365, which is the only solution I've used that makes me piss away more hours solving pointless problems (or waiting for Microsoft to do so) than Exchange itself.

Maybe I wouldn't have Office 365 so much if it weren't for the 48-hour lag on support calls, followed by 32 hours to resolve issues, but this is what it is. And when it's a "client down" scenario, 3+ days to get them back online isn't okay.

So yeah, Google Apps, when possible. Because it just works. If you read these pages, you know I'm not a big public cloud fan...but I trust Google to keep the e-mail working. Because they have a hell of a track record of doing so.

The solution, to my mind, is "have a critical service be bulletproof." I cannot offer that running on 10-year-old hardware using overly complicated MTAs with no funding for proper spam and antivirus scanning software. I am not convinced that Office 365 can offer it either. The only things I trust are Qmail, Zimbra and Postfix (which the client is allergic to) and Google Apps (which at least has something sort of like public folders, though you have to use a web UI to access them.)

Hence the desire to convince them that's the way to go.

When someone says "do this" and you aren't sure you can, the bigger mistake, I think, is spending your life just saying "yes". I've started to say "no", and this is a source of a lot of tension and conflict. "No, I can't do that" or "I don't think that will work." A decade ago I would fucking make it work...but a decade ago I only needed 2 hours a night of sleep...and I was only responsible for about 12 applications.

Now I am responsible for hundreds of applications, and I'm getting old. I need 8 hours of sleep or I am worthless the next morning. That young punk who could solve any technological problem using spit and bailing wire and sheer force of will is dead and buried. I used to know all there was to know within my sphere...but IT now encompasses a hell of a lot more than it did then. I could spend my entire day just trying to keep track of which companies exist in our industry, let alone what they do and how to implement their technologies.

So the scope of the project is beyond just software needs or desires for one vendor or feature. Who is going to look after this stuff? Especially once I'm no longer there to keep it ticking along? How will it all interact with everything else, and should it even interact with anything else?

The more I ask these questions, the more I want to pull core services off the local network. Some things need to be in house. But e-mail doesn't. There's already too much there for one person to handle; I'd prefer to pull everything that doesn't need be on-prem off, just so that it's feasible that one person with next-to-no budget can keep that place going for another decade.

Even if that means feeding the advertising behemoth of Mountain View.

Re: DailyWTF?

You know, I find this whole "never go live on a Friday" thing idiotic. I went live for a brief period of testing on a Friday. Someone found the error I missed on a Sunday. It was fixed before Monday. Staff came to work with a weekend of low-volume traffic where they had to check through the junk-email folder for (on average) about 15 e-mails to see if they were false positives. Not the end of the fucking worked.

If I had run that thing at 8am Monday morning, it would have taken about 4 hours for someone to notice that something was up. In that time an average of about 100 e-mails would have hit each person's box that they needed to check through.

And I'd rather work a weekend than have 50 people screeching at me demanding to know when the fix will be in, "How could I possibly have let this happen" and telling me how shit I am because I can't design a network that's more reliable than Google while being more accurate than Microsoft and more capable than Amazon, all for free.

Buncha great choices there.

Re: DailyWTF?

1) My spam servers worked just fine for years.

2) Putting things live during the day risks outages during working hours which has been emphatically affirmed to be an absolute no-no. There isn't much choice.

3) Exim? Really? I'm a bit of a QMail fan myself, though I have to admit that Postfix has come a long way. Honestly though, I've been working more and more with Zimbra and liking it.

I loathe exchange with the burning passion of 10,000 suns.

Re: Hosted AS?

@Vic; I'm on the long path to getting rid of a decade's worth of bandaids and nudging the client along towards a proper (though significantly more expensive/year) IT setup ahead of leaving. It's a long fight.

@Sampler

Oh, I tried that argument. I believe the response was "so we can take the cost of the hosted AS out of your salary?"

Re: Quote to long to put int title

I'm not in the UK/EU. I'm in Canada. And for this class of customer you cheerily can put them in Google Apps without consequence.

You get an exchange licence and 10 free CALs with your Action Pack. That's about $400/year. When you already have to have a virtual infrastructure to deal with all the other stuff you do....yeah, it's cheaper.

Internal IT doesn't exist just to support one app.

Re: actually..

I did. I fed it simulated data for days. Of course, the one thing I hadn't thought of was that the X-SPAM-HEADER info would be a problem, so the simulated data all had X-SPAM-HEADER data of either "yes" or "no".

@peteur

I try to discourage people working late, or on weekends. I have few enough maintenance windows as it is. If you work during off hours, well, I have no sympathy. There isn't a 24/7 global team of nerds to implement changes and patch things. So we have to sleep some time. If I have to be up for the 9-5 grind, then I'm not waiting until 3am to patch.

Besides, some folks start getting in a 4am...

Re: Friday ...

For a full "this is live and will stay that way", I agree. For a pre-permenant, data-gathering exercise that needs to run on live...this I prefer on the Friday EOB. Remeber, the goal here was not a permenant run, just a very brief test on live with just enough traffic to find bugs.

Found one.

Re: Speaking of language........

>_>

<_<

:(

//sads

Re: Hosted AS?

Hosted AS is ultimately where I want to go. The history is as follows:

1) Until recently, hosted AS was along the lines of "a few dollars per user per month" not "a few dollars per user per year." Which is more than the client would pay.

2) Until recently, relatively simple in-house open source AS systems worked just fine.

3) Having used the simple open source AS systems for so long transitioning away from them takes time. The existing system, for example, injects [SPAM ASSASSIN DETECTED SPAM] into the subject, rather than adding X-SPAM-STATUS

My goal is to get them using an in-house AS system that uses X-SPAM-STATUS for the rest of the year and then have them transition to a hosted AS system at the end of the year. This will be possible because both the system I'm trying to deploy for the in-house option and virtually all hosted AS systems use X-SPAM-STATUS.

Now, getting them to accept hosted AS will require getting them accept paying a subscription for an AS service when they're used to using free in-house stuff AND getting them to overcome their innate paranoia regarding having their e-mail hit servers in the states. I honestly don't know if I can "sell" that...and I'm pretty sure I don't care enough to try.

What I can do is get them migrated to a solution that uses X-SPAM-STATUS instead of subject injection which will make the transition to a proper hosted AS a heckofalot easier in that mythical future when the decide to just pay the tithe like everyone else.

That's the goal, anyways...

Re: Friday ...

Do it on a weekday and they'll have your hide if anything goes wrong. Do it on a weekend and there's not enough traffic to make it go 'ping'. Do it on a Friday, right after EOB and you have a few good hours of decent incoming traffic flow, a handful of folks who work late and are used to minor changes and an entire weekend to fix things if you bork them really badly.

Re: actually..

I watched it for about two hours. Nothing bizarre jumped out at me. I figured if something was going to go splonk, it would do so in a two hour timeframe. Guess I was wrong.

Re: "Our VDI panelists are diverse."

Agree 100%. I have a second webinar in the planning to address that. I am looking to do these in stages; start from the bottom (infrastructure) and work towards the top. The LoginVSI guys (and Eric, for that matter) know a fair amount about the user virtualisation issues, enough to hold their own, but user virtualisation is, of itself, a separate field entirely that transcends "just VDI".

User issues are present in non-persistent VDI, but also in "hoteling" desktop setups, multi-device/multi-OS setups and more. That's why that discussion will happen separately.

Re: roach motel

"he system PSOD'd and the HP firmware auto rebooted the box. Still waiting on support to analyze the logs first indication is it is a General protection fault (13)."

$20 it's a bad SPD chip on a single DIMM. Virtually impossible to isolate. The error manifests as a PSOD/BSOD only under certain very specific conditions. The issue is that the BIOS is configured to clock each DIMM (or at least each bank) independently. The bad SPD chip reports an incorrect speed for the capabilities of the DIMM. The result: an overclocked DIMM that goes squirrely seemingly at random, but especially when the temperature goes up.

The error will often show up as a set of ECC errors within your system, but when you go to memtest the DIMMs individually they're all fine. Alternately, you could have a system wherein timings are set per bank, not per DIMM, and the bad SPD chip is in fact on a DIMM that absolutely can handle the higher speed, but one of the other DIMMs in the bank can't, and that (perfectly fine, if tested on it's own) DIMM is the one that errors out.

Solution: attempt to isolate DIMM (hard) and RMA - or - manually set the speed and timings of all DIMMs in the system. (I typically downclock to just below rated anyways, just to avoid minor manufacturing defect issues.)

You can also use that Intel technology that allows you to RAID 1 your RAM. For file servers, this is what I do: RAID 1 the RAM and downclock it. Then the things run like a tank.

Re: Shut up take my money!

The 1.2TB Micron p420m is pretty much the most amazing thing that's ever entered my lab. There are not enough nice things to say about proper enterprise flash.

Re: "Cool things are the things that don't even know they are cool."

Run-of-the-mill twats I can cope with. Hipsters need to be liquified into series of short-chain polymers.

Re: "Cool things are the things that don't even know they are cool."

No true scotsman is all about someone trying to exclude others from their group by progressively narrowing the definition of the group until it includes people just like them.

I, on the other hand, and very specifically defining a single classification of attitudes and behaviors which I believe are deserving of being loaded into a cannon and fired into the sun.

I am not attempting to draw a narrow circle around people "like me" and excluding everyone else. I am drawing a circle around a small population and saying "the rest of humanity is good, but these fuckers can go to hell."

The no true Sctosman bit is about creating a clique. I'm on about discrimination against and identifiable group, where the group I've identified as needing to be discriminated is "entitled douche canoes."

Now, we could argue that I've chosen the wrong word for the group in question. In my experience, those who self-identify as "Hipster", or are most often identified by others as "Hipster" meet the qualifications for deserving to be trapped in a bubble at the bottom of the ocean with Barney playing on infinite repeat. But perhaps there is a fellow out there who self-identifies as Hipster that isn't part of that group. Okay. I've no problem with that. I think he's probably using the wrong term to describe himself, but that's no skin off my nose.

Unlike "No True Scotsman", I am stating my definition of the group which needs to be trapped in a glacier for all time, and don't actually care at all about the term used to describe them, whereas in "No True Scotsman" the focus is on keeping the term but progressively excluding those who don't fit exactly.

You can replace "Hipsters" in the above comments with "calamari-worshiping jelly fanatics" for all I care. So long as we're clear about who needs to be compressed into a singularity, then the term used to describe them is not relevant.

Cheers.

Re: "Cool things are the things that don't even know they are cool."

Is the lady a hipster? Because she can go the special hell too. The one reserved for people who pollute their coffee with cream and the blackguards who talk at the theater.

*Ahem*

There are five horsemen.

War, Pestilence, Famine, Death and Apathy. There may have originally been four horsemen, but Apathy has totally earned his stripes.