nav search
Data Center Software Security DevOps Business Personal Tech Science Emergent Tech Bootnotes
BOFH
Lectures

back to article
Stay out of my server room!

Holmes

And this story is probably not the worst there is to come!

Let's face it, most non-techies see a nicely laid out server room as a possible space for all kinds of storage but the task of getting a nicely laid out room is a feat in its own right. I can recall having to move from one rather antiquated but serviceable server room to an area temp-walled off from an open plan office with no cooling and a ventilation system that opened onto what at that time was a building site. What I found thereafter in the various fans, filters and whatnot can be left to the imagination!

Or, if your imagination isn't up to the task... here's a hint!

4
0

Re: And this story is probably not the worst there is to come! - I can relate!

When my old company split of its corporate parent, we moved to a new facility where I had the opportunity to build a brand new server room/data center. This contained the 4 data servers, the Exchange server, an SQL server, the two domain controllers plus the network switches, and phone system.

The racks were neat, cabling and power cords done nicely on ladders and up from the floor, and the room became a showpiece. When management would have company, meaning C-level visitors from business partners, they would get the cook's tour including the computer room. They would compliment us on the neat, clean and organized setup, and I got some nice compliments from the management as well for keeping things neat and tidy.

Then it happened. The company downsized as its business shrank, and we moved. This beautiful showpiece became a closet in the new building. I was given a brief tour of the shared facility and what we had to work with. I was told initially I would have a former locked office, but no that wasn't the case. Instead I got what used to be a former ladies bathroom which was converted to a storage room! The plumbing was gone, but next door the former men's room remained and leaked occasionally, and that had me on edge. Luckily nothing passed into the server room, though there was a puddle or two more often than not out in the hallway.

The other problem was there was no ventilation to speak of. Since this room went from a bathroom to a storage room, the building's owner had facilities removed the air-conditioning vents which made the room hot enough to roast a turkey. I made it be known that under no circumstances would my servers operate in a room like that. My manager looked at me in disbelieve because I'm usually quiet and didn't say much, but in this situation I knew exactly what would happen after about in hour in the oven! They conceded and put in a window air-conditioning unit! Yes, one of those home/office jobs you get at the DIY home goods and appliance centers! It worked and brought the temperature down to a reasonable level and I was able to set things up for about 9 months when it suddenly stopped working over a weekend!

I came in that Monday to find the users complaining they couldn't log in. I walked down the hallway to the old bathroom, I mean server closet, and could feel the heat on the doorknob. When I opened the door, the server had shut themselves down and there were alarms buzzing like crazy. As I said before, you could roast a turkey in there! Until facilities got around to replacing the A/C unit, I kept the room shutdown except for the domain controllers and two servers when needed with big fans running to suck out the heat, though that didn't work very well, and even at that point, the room was more than uncomfortable so I shutdown everything when everyone went home for the evening.

Anyway, this was not fun and I felt it was a slap in my face since this critical infrastructure which was the pride of the operation, became a third-place citizen in the end that was placed in a former bathroom, no less!

5
0
Silver badge

Combination lock on the door. And after Facilities have installed it, change the combination. They'll probably have left it at the manufacturer's default.

22
0
Anonymous Coward

Beware cheap combo locks

I've encountered several of those mechanical number pad with round knob or handles to open the door. Unfortunately a number of these accept the combination in any order, so a quick polish of the keypad and a bit of patience will reveal which have finger prints on them and the code.

My wife's old school had one and I suggested she tried this out as there had been a recent debate as to what the code was and a colleague had been adamant my wife had the code wrong.

One quick demo later was enough to get all of the combos replaced.

13
0
Silver badge

IT are generally in charge of the access control, too.

A 500kg holding force maglock with battery backup costs a pittance and can be fitted by anyone.

If you can't have people in your room, don't let them.

Working in a school, the IT Office is access controlled (only IT can open the door, otherwise we have to buzz them in), and the server rooms are inside that room and access-controlled again (physical key).

Best bit - not only can you decide who gets access, you can monitor who tries too, and whether that site-manager who absolute must have access to every cupboard that he never goes in is sneaking in at night to have a gander round.

At that point, you fit a PoE CCTV camera tied to your smartphone in that cupboard too.

"But what if they won't install it?" Buy it, put it in. It's access controlled, right, so nobody should be able to get in there to see you even have it...

11
0
Anonymous Coward

""But what if they won't install it?" Buy it, put it in. It's access controlled, right, so nobody should be able to get in there to see you even have it..."

Then someone higher than you (or perhaps the police) demand access and call in a locksmith...

2
5
Silver badge

And now you know that they've entered the space they weren't allowed to, and you invalidate all the passwords in that room...

Physical access is compromise.

10
1
Silver badge

Re: Beware cheap combo locks

If the cleaner isn't too great you can usually tell which 4 buttons are pressed most often, just ignore "C" as nobody seems to understand that's for clear/cancel rather than part of the code..

4
0
Silver badge

Too late. They would've already made alternate access methods.

Compromise must be assumed to be COMPLETE compromise.

2
1
Anonymous Coward

Punji sticks.

Basilisks

Electrified if necessary

4
0
Silver badge

Electrified basilisks? You'll get shamed by the RSPCA and raided by PETA.

8
0
Silver badge

Re: Beware cheap combo locks

A rather nice hack for those cheap keypads is to put two buttons in serial.

Example:

Combination 1-2-3-4*, 4 and 7 wired in series,

becomes 1-2-3-[4&7 pressed simultaneously].

Nobody expects pressing two buttons at the same time!

*Yes, yes, you have the same combination on your luggage.

2
0

Re: Beware cheap combo locks

Ultimate combo/permutation lock

http://codesmiths.com/shed/locks/manifoil.htm

0
0

Re: Beware cheap combo locks

Some very witty comments here...

If you've the budget, electronic access control is way better than a £20 push button "any order" XYZ mechdigi lock. Obviously!

You can't set the XYZ up for simultaneously pushed buttons. You're thinking of the Unican range, which start at around £200 not fitted. And few seem to be able to master those four picking or decoding. (and the electronic version is superb!)

For maybe £400 you can get a basic electronic access control system. But do yourself a favour, ask a professional to design and install it! Yes, it'll cost more, do more and be better - just like your IT system design is better than the boss's "great idea".

0
0

Re: Beware cheap combo locks

We had one of those in one place I worked. The well used buttons were so dirty you didn't need to polish the surface.

In another place I was told to jiggle the knob a few knob a few times and press the numbers, which didn't matter which ones, and I could get in.

That was safe!

1
0

oh, yes, it comes with keys to change the combination

you might also get a carpenter in to add a simple $30 keyed doorlock. they won't have that key. if needed, hey, firemen have axes. stockers with pallet handlers don't.

0
0
Silver badge

Our comms room at the office - it's not big enough to grace with the term "Server room" - was carefully planned by us IT staff when we moved buildings 6 years ago. We were given free reign to design the best possible space, as we were starting with a new build.

It had three 42U racks full of servers, and a fourth 42U cab which held the switches, patch panels and routers which sat in the middle of the room with free access all round.

It had nice open worktop areas round two sides, with loads of power and network outlets, and plenty of storage space for cables, components, etc, etc.

It is currently full to bursting with assorted crap, including old office chairs, 4 car tyres, a pile of ceased network switches and routers from a branch office which closed, 2 large-screen TVs, and the bicycle, and full golf-trolley, belonging to one of the Directors.

We can't actually open the doors to the racks without moving stuff out of the room.

Oh, forgot to say, access is controlled by physical locks and electronic passkeys, only the IT team and Directors have access. Guess where most of the junk comes from...

18
0

Rent a skip. Have it dropped in the car park. Tell those responsible they have one week to clear all the rubbish out of the Comms Room, after which it will all be in said skip.

Now the hard part - DO IT.

Once they see their stuff in the skip, you'll get action alright.

12
2
Anonymous Coward

There's a risk the Directors are over IT'S head or can more easily reach such people. Meaning there's a risk of the stuff getting swapped and pink slips attached.

4
0
Silver badge

Claim it is a fire hazard

Tell them you've heard from a friend who had their datacenter audited who found that having flammable items like boxes, chairs, tires and golf bags in your server room nearly got them shut down without notice until it was all quickly removed and dumped before the auditor left. Heck, I haven't read IFC/NFPA codes for a data center, but that might actually be in there.

A business I own was recently put on notice for a few things I never realized were NFPA violations (the previous inspector liked us and must have given us a pass) like multiplug adapters plugged into ceiling outlets without support, flammable items stored within 18" of the ceiling, having a door that isn't a marked fire exit actually usable as a door (which we are fixing by putting blinds over it so people don't "know" it is a door...sometimes the NFPA requirements are weird)

Another option could be to report the theft of some item that had been received as a free sample from a vendor (whatever excuse needed for it not to show up in a PO or in inventory) and suggest that all non-IT people be barred from access so it can be narrowed down should the culprit strike again. Give the directors an 'emergency access only' scenario - i.e. a code kept in an envelope in the company safe, which is changed after any use as part of the new more secure access policy.

That way they can still give "tours", but the code will change so they can't go back and dump stuff in there without you knowing when their one time code shows up in your logs.

10
0
Silver badge

Well, it's nice to be qualified to the civilian nearest equivalent of a Fire Marshall and asking some fire inspector to drop by for some advice. Next up, the requirements of OSHA and state variants have the most interesting regulations which have Director frightening fines, including prison time attached.

Whenever I've got downtime, backups or on-call time fer instances, I read engineering manuals and regulations. Aside from the weirdest shit in them, CYA dontcha know.

4
0

Ah yes, telling a director that no, they can't do whatever they want then blame I.T. if something breaks.

Good fucking luck. I've yet to meet ANY "director" who understands that they have to follow the same rules as everyone else. The only real choices, in a real world, are "suck it up" or "quit". Because the "director" will never, never learn. They don't have to, they're the "director". They're never wrong, and they're never at fault.

(replace "director" with everything from "senior management position of your choice" to "owner")

4
0

Fortunately, most see sense when you explain the long prison term for corporate manslaughter.

Blocking the fire escape with crap in the room most likely to have a fire? The directors will be in court, explaining it, if someone gets hurt.

It's not just a big fine anymore.

0
0

Re: Claim it is a fire hazard

We have extremely tight data center access control, only the minimum personnel, aka engineers have access to the data center. Anyone else has to jump through everything from information security through change management, even to get a tour.

Needless to say, being in the information security department, I've successfully avoided a tour. I've lived inside of data centers long enough to happily skip that bit. Copious signage reminds the engineers to not leave anything that isn't a server or switch and such being authorized to be there inside of the data center. Paper and boxes are outright forbidden - devices are brought in without boxes or dunnage whenever possible.

About two months after I had arrived at the facility, having recently transferred there, one of our information security engineers wanted to retrieve a removable hard drive from a server, where some patches were installed from the device remotely.

He also wanted to sidestep change management in order to more rapidly retrieve the device. I wasn't having any of it and put it through change management, speaking personally with the manager in charge of change management in order to escalate and facilitate speedy removal of the offending device.

Said engineer then attempted, to surprising success, in having our management seek authorization for me to gain access to the data center.

A quiet conversation over lunch with the change management manager, suggesting an attempt to end run around the change management process put the kibosh on that access.

With a bootnote from myself through the entire management chain, "If you want me to have engineering access to the data center, you're going to pay me engineering wages, grant engineering access to our systems and title me an engineer, otherwise I'll have none of it, as it's above my pay grade".

0
0
Silver badge

"I've yet to meet ANY "director" who understands that they have to follow the same rules as everyone else."

The problem with being a director successfully is that you have to know when it's time to break the rules, and get it right. That's what directors are for, otherwise you're just a pen pusher. Even apparently open and shut legal situations can be more ambiguous than you thought once lawyers get involved.

The way I used to assess my directors was, do they break the rules for the benefit of the company or for their own gain? The first ones are the ones to work for if you can. When I was a director, that was what I tried to do. I still got shouted at by some of my engineers, but I used to explain to them that it was my job to decide when to break the rules and take the consequences, and it was my judgement that I was paid for.

I am not including in this arrogance and bloody-mindedness, mind you.

0
0
Silver badge

"Next up, the requirements of OSHA and state variants have the most interesting regulations which have Director frightening fines, including prison time attached."

I've been told that something like this actually happened at $orkplace many years ago, with the fire brigade inspector making the point rather forcefully and clearly.

0
0
Silver badge

"Blocking the fire escape with crap in the room most likely to have a fire? The directors will be in court, explaining it, if someone gets hurt."

Likewise if someone gets injured by the fire supression system.

Some of them are surprisingly sensitive and bottles of inergen run to about £1500 a shot to refill.

0
0

Unless you're a director, or have the support of your reporting director, it's hard to run a shop at all. Or you can play truth or consequences: Communicate clearly the problems caused by hijacking the server room or wiring closet for storage or - worse - janitorial supplies, and then just let it fail. When you come to fix it, stack the crap in the hall, make the repairs, lock the door, and leave their crap in the hall.

There's tremendous satisfaction in staying calm while an executive gets red-faced over your explanation that an errant mop handle - that should never have been there - shut down his office or plant for the afternoon. I learned to hate the self-important buffoons, and just let them dig their own graves, documenting every shovel full.

Allow me to excel and you'll have show piece, but if you won't listen, pardon me while I save some sanity and just put you're gear on life support. Failure to listen on your part does not constitute an emergency on my part.

2
0

Luckily I work in a large Telco

Where I am, the only real issue we have is that the A/C has been moved from what felt like -15C to more like 25. In the hot Aussie summers. the server room moves up a degree or two making for unpleasant conditions but still within the thermal envelope. Installers left their crap laying about in our room and it was relocated to the hallway duct taped to together with about 5 rolls. didn't happen again....

10
0
Anonymous Coward

Re: Luckily I work in a large Telco

Our previous office had a nicely-cooled server room. So well cooled, in fact, that we used to store the champagne & beer for parties there, under the suspended floor just beside the cold air outlets.

New building is boring, big half-empty room, tightly-controlled access.

6
0
Anonymous Coward

Re: Luckily I work in a large Telco

"So well cooled, in fact, that we used to store the champagne & beer for parties there, "

we did that too .. left-overs from hospitality supplies for leaving drinks etc, all went under the floor where the mainframe used to be (cooling had not been altered to reflect that half the room was no longer generating any heat) ...

2
0
Silver badge
Coat

The headline image...

Trevor, you shaved your beard off!!

3
0
Gold badge

Re: The headline image...

It was int he way of my rockstar hair!

2
0
Silver badge
Angel

It took time

...but we now have all of our SER/MER's secured, finally. Besides our IT team, only building security has keys ;-}

4
0
Anonymous Coward

I had the luxury of restricting access, but my problem was the design of the building. The server room, while secure, did not have a separate air conditioning system. Anything I set it to was overridden by a master controller located toward the rear of the main office. It would cost far too much to alter the air conditioning system to our actual needs, of course, but that wasn't a problem in summer while everyone enjoyed the luxury of cooler air.

Inevitably, that all changed when winter started to come around. Every few days, I'd come in to a very noisy sauna-like server room because some office flunky decided they needed to turn the heating on through the air con system or simply decided the whole thing should be turned off. 2 years later, I'd left the company with no amount of emails, briefings or passive-aggressive bits of paper taped over the master unit reminding people to bring a jumper rather than jeopardise the office infrastructure having made a dent.

Thankfully, we did have a cabinet in a proper datacentre to put production systems, but it was always somehow my fault when the AD, email or NAS systems went down because they were trying to operate at sub-saharan temperatures before I could come in to turn the air con back on.

8
0
Silver badge

I assume things broke when stuff got hot and went down, like drives dying in your NAS? And that cost money, right? So you should have purchased something to automatically shut down everything when it got too hot in there. Then there'd be a quick correlation between whoever turned off the AC and everything going offline, and the finger of blame would point at the right place. And maybe if everyone got tired of that happening because people wouldn't learn, you could justify the investment for a separate cooling zone for the server room! :)

2
0
Silver badge

"the finger of blame would point at the right place."

Whatever the right place might be it always points to IT.

10
0
Anonymous Coward

"And maybe if everyone got tired of that happening because people wouldn't learn, you could justify the investment for a separate cooling zone for the server room! :)"

Or they'll just break into the server room and pull out the offending sentinel, even if it says, "Fire Risk!" on it. The only way to stop that is to get to know the local fire inspector and be sure he doesn't take bribes. Sometimes, the only language they'll understand is legalese.

3
0
Anonymous Coward

Then there'd be a quick correlation between whoever turned off the AC and everything going offline, and the finger of blame would point at the right place.

No, it'd be pointing at the "right" place.

0
0
Silver badge

" before I could come in to turn the air con back on."

Oh, I'm sorry, it's been cooked,. It'll take 3 days to get replacement parts.

Watch how fast you get your own AC

0
0
Bronze badge

Not mine but a council running a care line contact centre from a converted flat. Comms room a cupboard running 6 fairly beefy servers and the equipment we supported. There was an extraction fan, but due to the noise they kept the door shut so it didn't disturb the staff in the office next to the room.

Constant statements about their support being revoked/null due to heat issues were ignored and their kit failed.

12
0
Anonymous Coward

it's your own fault!

all my datacentres, either as a first-job junior to my current position (private, LocGov x 3, private) have always had restricted access to the data centre zone(s).

my last local government stint, we removed the last remnants of staff (ops/sysadmin) and any stand-alone servers (replaced with rack mounted or VMWare) out of the datacentre and also had periodic purges of 'junk' with the full support of the IT Director (ie he was an assistant director of the authority, so we were represented 'at board level').

currently, I'm even more restricted fro the DC, as have to have a change-control record or installation number in order to enter and make changes.

5
0

I should post a photo of our newly 'renovated' server room. As part of ongoing upgrading of our 'internal spaces' they fitted a brand new suspended ceiling over the existing 60's woodchip and asbestos ceiling. Unfortunately some of the cabinets reach right up to the ceiling so they actually built the ceiling round the cabinets. In each case they actually blocked the cabinet doors from opening. Not only that, but the fire suppression was also boxed in, as were the aircon vents. Not that blocking the fire suppression would matter because although they moved the alarms onto the new ceiling, they didn't bother to actually wire them up. Not that THAT matters because it turns out they also managed to drill through some part of the roof guttering system, and about a week after putting the new ceiling up, the entire thing came straight back down the first time it rained heavily. No hardware was harmed thankfully.

36
0

Oh, they also painted over everything in magnolia. Yes that includes cables if they were up against a wall, colour coded ducts etc.

26
0
Silver badge

I had a similar problem in an old server room about ten years ago. The Estates dept decided we weren't up to reg on the lagging of the heating pipes, so they got a contractor in to lag all the exposed pipework in that particular building over one weekend. Come Monday morning, off we go to do an early check of all the gear, only to find that the extra couple of inches of lagging meant that two cab doors couldn't be opened due to excessive pressure on the lower hinges, and that a load of ad hoc free-running cables which had been tucked behind the pipework for safety's sake (it's not like the useless heating pipes had ever actually been hot) were now permanently trapped there.

After our complaints fell upon deaf ears, we got the knives out and fixed the problem.

5
0

"got the knives out and fixed the problem."

We're the knives used on the insulation or the installers?

6
0

I think the worst thing I've seen was shortly after we had a leak in the aircon unit in the "server room" (comms cabinet would be closer). The aircon was above the door and when it leaked it saturated the door to the point where it expanded and jammed itself closed. We had to drill out the lock, and a substantial part of the door and literally kick the thing open again. Oh well, these things happen right?

Anyway, got a heap of notifications a week later after the aircon (but not the door) was fixed about temps running high in a majority of the rack so wen't to investigate only to be faced with what I can only describe as a "wall of toilet roll" behind the server room door. Yes, the cleaners had their quarterly delivery of goods for the office, and without anywhere to store fifteen hundred toilet rolls, they thought they'd re-purpose our little server room for the use of storing not just the rolls, but also bottles and bottles of various cleaning chemicals to boot.

The door was quickly replaced after that.. and I think we have a new cleaning company too..

22
0
Silver badge

Ahhh...

A couple of years ago, management decided to have a big clean out of our offices- without telling anyone. They hire a bunch of monkeys to just come in and throw anything not locked away into a skip, on a weekend. We came in on the Monday to find our entire spares cupboard stripped clean. We complained but it felt on deaf ears.

Cut to two months later and the NIC failed on an "important" server (one that produced reports for management- so not critical). There was much screaming, wailing and gnashing of teeth, and the accusation was levelled at IT that we should have seen that kind of thing coming and we should have prepared for it- our answer: We did, but you lot threw out all of the spares that were being stored for that very reason. Queue: mumbling and grumbling and "well, you still should have been prepared"...

35
0
Gold badge

Re: Ahhh...

Wait, have we worked together? That's my story. At least 5 different times.

Man, this industry is depressing.

2
0
Silver badge

Re: Ahhh...

The cleanout event got nicknamed "The Purge". For ages after, any management request for hardware (new keyboard, printer cartridge etc) would get turned down with the response "well, we used to have that but it got thrown out during The Purge". Eventually management relented and gave us a budget to buy up new replacements and spares.

2
0

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

The Register - Independent news and views for the tech community. Part of Situation Publishing