Posts by Trevor_Pott

@Pandy06269

We are running almost fully VDI. We did use templates. Office doesn't like bing cloned. Firefox has user-specific configurations and add-ons must be installed /per user/. We have industry-specific applications that absolutely require user specific configuration. PST files had to be loaded from the old forest's exchange server into the new forest's exchange server manually. The change from XP to 7 also brought with it the change in profile types. We changed from roaming profiles to folder redirection and did so across forests. That meant manually importing files from the old network and placing them into the appropriate places on the new network.

There were other things, but no. It wasn't something that could be automated. I promise you we automated everything we could, and made extensive use of template VMs. Sadly, the ability to filly configure applications which require user-specific setup depends on support from the application vendor. Two of the applications we use simply don’t support it, and I have zero say in the fact that we use them.

@Kempsy

Ubitmenu. <3.

Training?

As in where you spend money to get users trained? Not in the seven years I've worked there. Myself and the other sysadmin have gotten MCP exams paid for. The other sysadmin and the CFO periodically are flown out to learn some things about how the point of sales software works. One or two of our production folks attends a conference every year about how a piece of industry-specific software works.

As to users...no sir. It is generally considered our job as IT Operations to train them. The production fellow has to train users in the use of that application. That's as far as it goes. As to chaninging my job...why do you think I took up writing? It will be a slow process, but maybe in ten years...

Backgrounds?

Nein! We use RDP to access virtual machines. You vill use solid colours und you vill like ut!

Dude...

So many applications and settings for which that simply isn't possible. It sucks. Greatly.

@jackifshadows

I did it because it had to be done. I doubt I'll eve be trying something like it again. We didn't have the money for IBM Globalservices. We did have the money for a sysadmin who doesn't get paid overtime. What I did was make a decision that was good for the company, but bad for me personally.

If I have a flaw as a sysadmin it is honestly that I work too hard. When I put my “company hat” on, I push out my own needs and focus entirely on what is best for the company. The problem with that is that I then rapidly burn out, which doesn’t do the company any good and sure as heck doesn’t do me any good.

So the thing is to know when to spend the money and when to bur myself out. Sometimes the money isn’t there and so the call is taken out of my hands. Sometimes I make a bad call, sometimes I make a good one. I’m still learning where the balance lies.

I agree with your assessment though. The IT generalist is becoming an individual who manages a series of contractors and outsourcers. It’s something that saddens me, because it means the end of my career in IT. I don’t have a degree, nor any management credentials beyond running an SME IT department for seven years. I might be able to get a PMP designation or somesuch, but then I am fighting eleventeen squillion unemployed IT Operations guys with PMP designations for the small handful of jobs left on this continent.

Once the SME administration job market dries up, I honestly don’t have a clue how to make the jump to being an admin in a larger enterprise, nor do I think that I have the resume to go up against the many other vastly more experienced contenders for the “outsource manager” positions.

This is why I’ve taken up writing. I think that ten years hence there will be more of a career in writing than in SME IT administration. Have to change with the times. If I can stick with IT writing, then all those years of experience as an IT generalist won’t have gone to waste.

Hey SuperM

If you are looking for case study material you might find the reasoning behind the migration interesting.

The biggest reason for the move was the damage to the schema. In truth, some of it was caused by inexperience several years ago. Installing one product or uninstalling another caused all sorts of crap to accumulate in the scheme that I couldn’t clean out by hand no matter how hard I tried. (I simply didn’t know exactly where all the bits hung out, and Google wasn’t helping me much.)

Similarly, before we moved to virtual machines, it wasn’t uncommon for domain controllers to just up and die. As small as the company is (read the article again to get an exact count of how many systems we have) seven years ago we had one domain controller that was also a file server, firewall, FTP server and everything else all in one. When I started with the company that domain controller was being run off of a Pentium 4 desktop board with a single desktop-class hard drive.

As you can imagine it was a few years before we started to get DCs that didn’t experience random and sudden hardware failures; so there were remnants of these old DCs, despite my best efforts to nuke all references to them out of the AD. (The “Proper procedures” don’t get them all. Especially if your previously an heroed DC was a certificate authority, etc.) Not to mention that going from OCS 2003 -> 2005 -> 2005 R2-> 2007 R2 creates a whole bunch of zomfgwtf hanging about in the AD. Similarly Exchange 2000 -> 2003 -> 2007 -> 2010

In theory if I had thrown enough time at the old AD I might have been able to clean it. There comes a point however where you have to look at the whole mess and say “I only have 75 users. Let’s just restart from scratch; it’s significantly less effort.” I always have this sneaking suspicion that if I was a true active directory expert, I might have been able to avoid all of this. I’m not though. I’m not an “expert” at any one field of IT. I haven’t spend my career specialising. As an SME sysadmin I have to maintain all of those systems you read about (and more that didn’t make it into the article.)

There simply isn’t any possible way for one human being to develop true expertise in all of those various systems. (There is no such thing as a modern-day polymath.) The best I can reasonably hope for is to understand the fundamentals and as many of the commands/quirks/specialised ballyhoo of every application, operating system, hypervisor, file system, database, piece of hardware, networking, crypto etc as I can fit into my brain.

Where that falls apart is the truly in depth knowledge of things like Active Directory. I know more than your average bear; but by the same token I only really have to deal with it once every two or three years. It might have some real arguments for/against IT generalists like me. I believe I am a rare breed in IT; most folks seem to have specialised in some particular part of it (databases, AD, LAMP, whathaveyou) by this point in their career.

I am capable of dealing with a wider variety of systems than your average specialised IT body…but in order to have that capability I have to sacrifice a great deal of the super-specialised knowledge that comes from spending over a decade dedicated to a single type of product. At first glance, someone like myself might seem ideal for an SME. It certainly gives me a breadth of knowledge and experience that allows me to write about various topics here on El Reg.

What I begin to wonder however is if the truly efficient way to deal with SME IT administration isn’t to have SMEs handled by largish consultancies. A large consultancy can afford to have one (or more than one) of each relevant kind of specialist. When they run up against a challenge like I described above they don’t have to restart the whole AD from scratch. They summon their in-house AD super-specialist and he deals with it.

Where’s the line between the utility of an IT generalist and the advantages of a small cluster of IT specialists? I’ll be honest when I say that I don’t know. I am however exceptionally curious about the answer…

@Loogie

A large part of what we needed to do was shed the "cruft" a decade of disparate naming schemes, hirings, firings, e-mail address changes, remaining of users etc had caused to the AD.

You had USER_D who had been created as USER_A originally. The name was changed in order to ensure that USER_D could access USER_A's e-mail for business continuity purposes, but many of the various hooks for that user in the AD still reference USER_A. Similarly, many of the users were simply Firstname instead of Firstname.Lastinitial (which we started to make all new users several years ago.) There are other examples, but you get the idea. For this reason ADMT was pointless. We didn’t want to migrate the users. We wanted entirely new users with clean information that would follow the naming convention from the CEO to the digital janitors.

Recreating 75 users was the absolute least of our worries. That was about an hour’s work. Create the users, assign them a SIP address, make them an e-mail address. Link the home folders. Set the dial-in permissions on the small handful who needed them. Everything else was handled through GPO.

@AC

Two problems with that idea.

1) I actually have no idea where to rent server equipment from. Honestly not a clue where to even begin.

2) I didn't think of it until we were about halfway through the weekend. Hindsight and a great deal of “d’oh” suggests you might well be on the proper track, however…

@AC

The printers were largely upgrades to an extant lease of Xerox printers. Some were replacements for really, REALLY old workcenters (Think N32s, N40s, etc.) No futzing with forms was necessary, and I did spec delivery for the middle of the month. (Doomsday Weekend took place on the 20th of August.) Negotiations for the whole thing started at the beginning of August.

The nice lady seemed to be trying to do everything she could to get the widgets on time, but sometimes expletive happens. Sadly, it tends to happen at the worst possible time…

@The original steve

Actually, I *AM* the one who said it had to be done in a weekend. Here's the scoop:

We absolutely /had/ to have the updates done by September 1st. (This per CTO requiring updates and the fact that Sept – Dec is silly season around here.) As per above, I could see no way except to do it “all in one go.” I was not present through July, as I was in two of the other locations installing Wyse clients, recabling and prepping the locations local hardware for the changeover.

It took us the first two weeks of August to get the Domain Controllers, E-mail server, and BES/OCS/WSUS/Teamviewer manager server (yes those 4 share a VM) installed and for the new network. (I was going to have THOSE at least installed, if not configured by Doomsday.) In addition, we roughed out a template Windows 7 VM and a few template Windows XP VMs for render boxes.

We didn’t have to go ENTIRELY from scratch, but I promise you that even to get what we had prepped in advance we were scraping the bottom of the excess Virtual Server capacity barrel.

I managed to create all the user accounts prior to hitting the wall, but that’s about it. Exchange wasn’t configured, OCS was not only not properly configured…it had to be uninstalled THRICE and reinstalled specifically to get the blessed thing to cooperate. As to the rest…well…there’s more articles on that.

Suffice it to say that the actual call “hey guys, we need to do this all in one go over a weekend” was mine…however there were ZERO other choices that would have had us meet the deadlines imposed on us…

@AC

Old printers were giving up the ghost. (Some were pushing 10+ years.) We needed a new set of higher volume printers. Had the things arrived on time, the old printers would have gone away with the old network, and the new printers would have started working on Monday morning with the new network. Alas, this was not to be as the – get this – paper trays where backordered.

Yeah. Paper trays.

*sob*

Contractor. HAH.

Full time regular staff. Salaried, and I don't get overtime. There wasn’t enough gear to run both networks in parallel. We had to pull of the network change without new equipment; almost everything is virtualised, so the “new network” was largely a set of new virtual machines. Given that, I don’t actually see how you can move from one forest to another, migrating all services “one at a time.”

I would love to say I am totally in charge of making all such decisions, but unfortunately I do have to work with what I’m given. I do largely get to buy what want, but I have to justify it all; part of that justification is that all equipment be utilised 80%+ for it’s lifespan. Buying equipment just to handle the changeover then letting it sit idle would /not/ have gone over well. Most especially since we would have required somewhere around 30% of our yearly budget’s worth of gear to do it.

Sometimes, you just gotta do what you gotta do…

@AC

Bringing in other help. THAT is a whole other story. The interesting part is that I did try to get a buddy of mine who runs a computer consultancy to lend a hand (for appropriate remuneration) during the op. He bailed at the eleventh hour adding another layer of fun and happiness to this entire exercise. Additionally, another individual I was hoping to be able to talk into assisting got distracted by Starcraft 2. (He was at the time considering pursing IT as a career only to change his mind and head towards electrician in the last couple of weeks.)

So what I was seriously hoping was going to be a five man operation became a three man operation. Then Xerox got tripped up on the printer delivery due to backordered paper trays (sonofa…) OCS blew up. The Spam server ate half a night because I typoed the domain name. Office 2010 had some ridiculous upgrade guarantee thing that made getting the 2010 key from our 2007 installs a screaming nightmare and took five times as long as we figured it would. We discovered too late that Firefox add-ons are USER SPECIFIC and didn’t have time to figure out how to push those through a GPO, so ended up installing them manually.

To top it off, we had two different industry-specific applications that absolutely required user-by-user configurations and took about an hour each. There is no facility whatsoever for centralised deployment and configuration of those applications. Throw into the mix that each user is a “special case” with their own unique set of software requirements and you can’t just roll out one base image and be done with it.

It’s so very easy for people looking at something like this in hindsight, or reading about only part of the reasons and events that occurred to spew vitriol and negativity about the whole thing. Actually working in an SME environment like this is a completely different story…something I hope that my blog articles can help convey.

@Ammaross Danan

Amen brother. You have hit upon the purpose behind the entire exercise. There were some things we simply couldn't accomplish with the changeover except by logging in as those users and setting their profiles up for them. What folks with the anger-making don’t see to get is that had everything gone to plan, we would indeed have reset the users’ passwords and forced them to change upon first login that Monday.

Things went all pear-shaped when folk started showing up on the Monday…and oh, damn…we weren’t done customising their VMs. By the time we got done with the customisation, we were deep into Tuesday. This means the users had logged in for a couple of days with the “new” passwords they were given.

Going around and forcing a password reset on the Wed morning, after Monday and Tuesday had been filled with printer issues, incompletely-customised VMs and changes to new and bewildering versions of software like Office 2010 would have been heap bad juju for IT.

The solution in the end was to phone the users one at a time and walk them through the password changes whilst taking the time to hear out any complaints they had, teach them any bits the needed to know about the new software and help them customise Windows 7 to behave less new and scary.

The sheer APATHY towards the “changing passwords” part of that phone call shocked me however. Users really, really don’t give a damn about security. Enough that I am starting to come round to the opinion that they honestly do need a periodic kick in the ass about it.

Unlike Unix, Windows administrators don’t have the opportunity to completely abstract the security away from users, nor do they have the control that a Unix admin would have to customise profiles and the like from afar. This means some level of user cooperation is required in the Windows world. When you run up against the sheer user apathy we did…that’s an eye opener worthy of an article.

@Keith 21

Well, first off...we don't have compliance issues to worry about. Were i operating in an environment where that was a concern, then things certainly would play out differently. We did have a contingency plan if things went wrong: step through issues with the users one at a time. There are only 75 users.

Phoning them up one at a time to deal with their passwords, listen to their complaints and solve any problems they have one at a time really didn’t take that long. Less than a working week and we had dealt with each and every user in the company. A contingency plan doesn’t have to be a set of automated tools or a whitepaper. Sometimes it can be putting one’s nose to the grindstone or adding a human touch.

There are “proper” ways to do things. I agree with you wholeheartedly that if you have the time, the resources and the manpower then how things went down during doomsday weekend would look frighteningly inefficient. What my Doomsday Weekend articles are about is not how things work in a corporation of 300,000 users with a crack team of administrators and management that understands why IT Operations need time to execute.

What my Doomsday Weekend articles are about is what life is like at the coalface of an undermanned, underfunded SME IT department with nearly zero resources and a management staff that doesn’t remotely understand why IT Operations need time to execute. My Blogs aren’t a statement of how I have accomplished perfection and found the one true path. They are a demonstration of the mistakes I have made, the neat products I have encountered and the insights I have gained from it all.

I operate in a far less than perfect environment. Contrary to your stated belief, I am not lazy. I am willing to put the time and effort in whenever and wherever I can. If you read the article previous to this one, you will see the network I have to run, with only two sysadmins and a bench tech to hold it up. I promise you that I know the “by the book” ways to run a network. I also promise you that we don’t have the resources or the time to manage by whitepaper, as easy as that would be.

As to “lashing out,” I don’t believe I did any such thing. If anything, I think you will find I took a good poke at both users and systems administrators in my article. What I do have to question though is why you feel the need to be so negative towards me? What did I do to offend you so?

@James Pond

Biometrics and Wyse Thin Clients. *Wince.* I am not sure I want to start mixing magics like that.

Correction:

This should have read:

My statement was /merely/ that it was a common approach, but in all truth not one I support in any way.

My experiment of using my HTC Desire as the only source of comment input for my own articles over the past month is truly proving both how bad I am at spelling/grammar without squiggly little lines under everything and how far these devices really do have yet to go as regards text input.

More fodder for the smartphone-as-a-work-tool article I suppose…

@Timo

The pre-edited version of the article had this line: "I should have printed off a copy on each of the remote sites’ printers such that the information they required was on hand when they walked in."

Very small difference to the published article, but it essentially mirrors your advice. I have a few users who would agree with you that it was indeed the correct approach to have taken.

@AC

Overall this is excellent advice. The dynamics of this appraoch in a smaller company become far more complicated...but the advice is no less true.

@nichomach

Honestly, I don't believe in this approach. My statement was mere that it was a common approach, but in all truth not one I support in any way.

@AnachismEvolved

Would have been in compliance if we had managed to reset all the passwords before the bell. Sadly...

@Gordan

I can RAID drives. DIMMS, not so much.

As to redundancy...difficult to do on an SME budget, but I'd like to think we don't do too badly. I've learned the hard way some failure points, while others I was fortunate enough to see coming. The La Cie isn't something I would /ever/ use as a primary device. It was hoped however that the thing would serve as a temporary storage point during the move. Not so much, apparently.

As to fans issue; good fans will give me three years. The fans that don't give me a single year irk me greatly. We can't afford IT bodies in all sites, so we are reliant on yearly preventative maintenance. Fans that refuse to give me even that year...I'll not be buying from that company again.

Any decent sysadmin plans for redundancy within the budgetary parameters given them. There is that point however where you do have to trust hat some of your components will stay up some of the time; a minimum service life if you will. If only because the primaries have to be operational whilst you are upgrading the backups.

@Pavlov's obedient mutt

I can't speak to the "legal" part of the editing...but it definitely reads significantly different than written. My editor tries to compress my articles into as small a space as possible; I fear I have not yet learned the art of telling my tale in few enough words. ("Be ruthless, you know it makes sense!") Overall I think he's an excellent editor. I like to think that any improvement at all in my writing style is due to his hard work and the kind advice of several folks at El Reg.

I was hoping that this article would serve as an introduction to a short series of articles based on a particular hellacious network migration I recently underwent. Unfortunately, that introduction contained a lot of lead-in and ended up being about three hundred words over limit. I can’t really comment on the resultant article; it would be the ultimate height of arrogance to comment on one’s editor’s style. Frankly, I’m new enough not to have much experience in what “sells,” other than I write (and talk) way too much.

The ultimate measure though is whether the readers here like my articles or not. While for all other articles I’ve been merely posting links to them on my personal blog. I am allowed to post the full unedited articles there after a few weeks have passed since their publication on El Reg. It might be an interesting experiment to post the “Doomsday Weekend” articles in full and have some folk read the edited and non-edited versions.

That, and my editor has kindly offered to take me through editing one of my articles step by step. With luck, from both I will learn what people like and don’t like. Stylewise, my articles (when they reach my editor) read no different than my comments. I have a lot of learning to do before I can pare my writing down enough to produce articles that don’t need editing at all…but I’m learning!

@Mosquito

It is the introduction; first part of a set. There will be additional articles with details as to what was attempted and what we suceeded/failed at.