Stay out of my server room!

Re: And this story is probably not the worst there is to come! - I can relate!

When my old company split of its corporate parent, we moved to a new facility where I had the opportunity to build a brand new server room/data center. This contained the 4 data servers, the Exchange server, an SQL server, the two domain controllers plus the network switches, and phone system.

The racks were neat, cabling and power cords done nicely on ladders and up from the floor, and the room became a showpiece. When management would have company, meaning C-level visitors from business partners, they would get the cook's tour including the computer room. They would compliment us on the neat, clean and organized setup, and I got some nice compliments from the management as well for keeping things neat and tidy.

Then it happened. The company downsized as its business shrank, and we moved. This beautiful showpiece became a closet in the new building. I was given a brief tour of the shared facility and what we had to work with. I was told initially I would have a former locked office, but no that wasn't the case. Instead I got what used to be a former ladies bathroom which was converted to a storage room! The plumbing was gone, but next door the former men's room remained and leaked occasionally, and that had me on edge. Luckily nothing passed into the server room, though there was a puddle or two more often than not out in the hallway.

The other problem was there was no ventilation to speak of. Since this room went from a bathroom to a storage room, the building's owner had facilities removed the air-conditioning vents which made the room hot enough to roast a turkey. I made it be known that under no circumstances would my servers operate in a room like that. My manager looked at me in disbelieve because I'm usually quiet and didn't say much, but in this situation I knew exactly what would happen after about in hour in the oven! They conceded and put in a window air-conditioning unit! Yes, one of those home/office jobs you get at the DIY home goods and appliance centers! It worked and brought the temperature down to a reasonable level and I was able to set things up for about 9 months when it suddenly stopped working over a weekend!

I came in that Monday to find the users complaining they couldn't log in. I walked down the hallway to the old bathroom, I mean server closet, and could feel the heat on the doorknob. When I opened the door, the server had shut themselves down and there were alarms buzzing like crazy. As I said before, you could roast a turkey in there! Until facilities got around to replacing the A/C unit, I kept the room shutdown except for the domain controllers and two servers when needed with big fans running to suck out the heat, though that didn't work very well, and even at that point, the room was more than uncomfortable so I shutdown everything when everyone went home for the evening.

Anyway, this was not fun and I felt it was a slap in my face since this critical infrastructure which was the pride of the operation, became a third-place citizen in the end that was placed in a former bathroom, no less!

Beware cheap combo locks

I've encountered several of those mechanical number pad with round knob or handles to open the door. Unfortunately a number of these accept the combination in any order, so a quick polish of the keypad and a bit of patience will reveal which have finger prints on them and the code.

My wife's old school had one and I suggested she tried this out as there had been a recent debate as to what the code was and a colleague had been adamant my wife had the code wrong.

One quick demo later was enough to get all of the combos replaced.

IT are generally in charge of the access control, too.

A 500kg holding force maglock with battery backup costs a pittance and can be fitted by anyone.

If you can't have people in your room, don't let them.

Working in a school, the IT Office is access controlled (only IT can open the door, otherwise we have to buzz them in), and the server rooms are inside that room and access-controlled again (physical key).

Best bit - not only can you decide who gets access, you can monitor who tries too, and whether that site-manager who absolute must have access to every cupboard that he never goes in is sneaking in at night to have a gander round.

At that point, you fit a PoE CCTV camera tied to your smartphone in that cupboard too.

"But what if they won't install it?" Buy it, put it in. It's access controlled, right, so nobody should be able to get in there to see you even have it...

Punji sticks.

Basilisks

Electrified if necessary

oh, yes, it comes with keys to change the combination

you might also get a carpenter in to add a simple $30 keyed doorlock. they won't have that key. if needed, hey, firemen have axes. stockers with pallet handlers don't.

Rent a skip. Have it dropped in the car park. Tell those responsible they have one week to clear all the rubbish out of the Comms Room, after which it will all be in said skip.

Now the hard part - DO IT.

Once they see their stuff in the skip, you'll get action alright.

Claim it is a fire hazard

Tell them you've heard from a friend who had their datacenter audited who found that having flammable items like boxes, chairs, tires and golf bags in your server room nearly got them shut down without notice until it was all quickly removed and dumped before the auditor left. Heck, I haven't read IFC/NFPA codes for a data center, but that might actually be in there.

A business I own was recently put on notice for a few things I never realized were NFPA violations (the previous inspector liked us and must have given us a pass) like multiplug adapters plugged into ceiling outlets without support, flammable items stored within 18" of the ceiling, having a door that isn't a marked fire exit actually usable as a door (which we are fixing by putting blinds over it so people don't "know" it is a door...sometimes the NFPA requirements are weird)

Another option could be to report the theft of some item that had been received as a free sample from a vendor (whatever excuse needed for it not to show up in a PO or in inventory) and suggest that all non-IT people be barred from access so it can be narrowed down should the culprit strike again. Give the directors an 'emergency access only' scenario - i.e. a code kept in an envelope in the company safe, which is changed after any use as part of the new more secure access policy.

That way they can still give "tours", but the code will change so they can't go back and dump stuff in there without you knowing when their one time code shows up in your logs.

Re: Luckily I work in a large Telco

Our previous office had a nicely-cooled server room. So well cooled, in fact, that we used to store the champagne & beer for parties there, under the suspended floor just beside the cold air outlets.

New building is boring, big half-empty room, tightly-controlled access.

Re: The headline image...

It was int he way of my rockstar hair!

I assume things broke when stuff got hot and went down, like drives dying in your NAS? And that cost money, right? So you should have purchased something to automatically shut down everything when it got too hot in there. Then there'd be a quick correlation between whoever turned off the AC and everything going offline, and the finger of blame would point at the right place. And maybe if everyone got tired of that happening because people wouldn't learn, you could justify the investment for a separate cooling zone for the server room! :)

" before I could come in to turn the air con back on."

Oh, I'm sorry, it's been cooked,. It'll take 3 days to get replacement parts.

Watch how fast you get your own AC

Oh, they also painted over everything in magnolia. Yes that includes cables if they were up against a wall, colour coded ducts etc.

I had a similar problem in an old server room about ten years ago. The Estates dept decided we weren't up to reg on the lagging of the heating pipes, so they got a contractor in to lag all the exposed pipework in that particular building over one weekend. Come Monday morning, off we go to do an early check of all the gear, only to find that the extra couple of inches of lagging meant that two cab doors couldn't be opened due to excessive pressure on the lower hinges, and that a load of ad hoc free-running cables which had been tucked behind the pipework for safety's sake (it's not like the useless heating pipes had ever actually been hot) were now permanently trapped there.

After our complaints fell upon deaf ears, we got the knives out and fixed the problem.

Re: Ahhh...

Wait, have we worked together? That's my story. At least 5 different times.

Man, this industry is depressing.