back to article Stay out of my server room!

We always design our data centres and server rooms with the best of intentions. A nice, enclosed space with excellent ventilation and air conditioning, sufficient power points to ensure circuits are never overloaded, and enough space to get at both the front and back sides of the rack – ideally all cordoned off into its own room …

  1. Chika
    Holmes

    And this story is probably not the worst there is to come!

    Let's face it, most non-techies see a nicely laid out server room as a possible space for all kinds of storage but the task of getting a nicely laid out room is a feat in its own right. I can recall having to move from one rather antiquated but serviceable server room to an area temp-walled off from an open plan office with no cooling and a ventilation system that opened onto what at that time was a building site. What I found thereafter in the various fans, filters and whatnot can be left to the imagination!

    Or, if your imagination isn't up to the task... here's a hint!

    1. jcitron

      Re: And this story is probably not the worst there is to come! - I can relate!

      When my old company split of its corporate parent, we moved to a new facility where I had the opportunity to build a brand new server room/data center. This contained the 4 data servers, the Exchange server, an SQL server, the two domain controllers plus the network switches, and phone system.

      The racks were neat, cabling and power cords done nicely on ladders and up from the floor, and the room became a showpiece. When management would have company, meaning C-level visitors from business partners, they would get the cook's tour including the computer room. They would compliment us on the neat, clean and organized setup, and I got some nice compliments from the management as well for keeping things neat and tidy.

      Then it happened. The company downsized as its business shrank, and we moved. This beautiful showpiece became a closet in the new building. I was given a brief tour of the shared facility and what we had to work with. I was told initially I would have a former locked office, but no that wasn't the case. Instead I got what used to be a former ladies bathroom which was converted to a storage room! The plumbing was gone, but next door the former men's room remained and leaked occasionally, and that had me on edge. Luckily nothing passed into the server room, though there was a puddle or two more often than not out in the hallway.

      The other problem was there was no ventilation to speak of. Since this room went from a bathroom to a storage room, the building's owner had facilities removed the air-conditioning vents which made the room hot enough to roast a turkey. I made it be known that under no circumstances would my servers operate in a room like that. My manager looked at me in disbelieve because I'm usually quiet and didn't say much, but in this situation I knew exactly what would happen after about in hour in the oven! They conceded and put in a window air-conditioning unit! Yes, one of those home/office jobs you get at the DIY home goods and appliance centers! It worked and brought the temperature down to a reasonable level and I was able to set things up for about 9 months when it suddenly stopped working over a weekend!

      I came in that Monday to find the users complaining they couldn't log in. I walked down the hallway to the old bathroom, I mean server closet, and could feel the heat on the doorknob. When I opened the door, the server had shut themselves down and there were alarms buzzing like crazy. As I said before, you could roast a turkey in there! Until facilities got around to replacing the A/C unit, I kept the room shutdown except for the domain controllers and two servers when needed with big fans running to suck out the heat, though that didn't work very well, and even at that point, the room was more than uncomfortable so I shutdown everything when everyone went home for the evening.

      Anyway, this was not fun and I felt it was a slap in my face since this critical infrastructure which was the pride of the operation, became a third-place citizen in the end that was placed in a former bathroom, no less!

  2. Doctor Syntax Silver badge

    Combination lock on the door. And after Facilities have installed it, change the combination. They'll probably have left it at the manufacturer's default.

    1. Anonymous Coward
      Anonymous Coward

      Beware cheap combo locks

      I've encountered several of those mechanical number pad with round knob or handles to open the door. Unfortunately a number of these accept the combination in any order, so a quick polish of the keypad and a bit of patience will reveal which have finger prints on them and the code.

      My wife's old school had one and I suggested she tried this out as there had been a recent debate as to what the code was and a colleague had been adamant my wife had the code wrong.

      One quick demo later was enough to get all of the combos replaced.

      1. Halfmad

        Re: Beware cheap combo locks

        If the cleaner isn't too great you can usually tell which 4 buttons are pressed most often, just ignore "C" as nobody seems to understand that's for clear/cancel rather than part of the code..

      2. Captain DaFt

        Re: Beware cheap combo locks

        A rather nice hack for those cheap keypads is to put two buttons in serial.

        Example:

        Combination 1-2-3-4*, 4 and 7 wired in series,

        becomes 1-2-3-[4&7 pressed simultaneously].

        Nobody expects pressing two buttons at the same time!

        *Yes, yes, you have the same combination on your luggage.

      3. Pangasinan

        Re: Beware cheap combo locks

        Ultimate combo/permutation lock

        http://codesmiths.com/shed/locks/manifoil.htm

        1. YetAnotherLocksmith Silver badge

          Re: Beware cheap combo locks

          Some very witty comments here...

          If you've the budget, electronic access control is way better than a £20 push button "any order" XYZ mechdigi lock. Obviously!

          You can't set the XYZ up for simultaneously pushed buttons. You're thinking of the Unican range, which start at around £200 not fitted. And few seem to be able to master those four picking or decoding. (and the electronic version is superb!)

          For maybe £400 you can get a basic electronic access control system. But do yourself a favour, ask a professional to design and install it! Yes, it'll cost more, do more and be better - just like your IT system design is better than the boss's "great idea".

      4. jcitron

        Re: Beware cheap combo locks

        We had one of those in one place I worked. The well used buttons were so dirty you didn't need to polish the surface.

        In another place I was told to jiggle the knob a few knob a few times and press the numbers, which didn't matter which ones, and I could get in.

        That was safe!

    2. Lee D Silver badge

      IT are generally in charge of the access control, too.

      A 500kg holding force maglock with battery backup costs a pittance and can be fitted by anyone.

      If you can't have people in your room, don't let them.

      Working in a school, the IT Office is access controlled (only IT can open the door, otherwise we have to buzz them in), and the server rooms are inside that room and access-controlled again (physical key).

      Best bit - not only can you decide who gets access, you can monitor who tries too, and whether that site-manager who absolute must have access to every cupboard that he never goes in is sneaking in at night to have a gander round.

      At that point, you fit a PoE CCTV camera tied to your smartphone in that cupboard too.

      "But what if they won't install it?" Buy it, put it in. It's access controlled, right, so nobody should be able to get in there to see you even have it...

      1. Anonymous Coward
        Anonymous Coward

        ""But what if they won't install it?" Buy it, put it in. It's access controlled, right, so nobody should be able to get in there to see you even have it..."

        Then someone higher than you (or perhaps the police) demand access and call in a locksmith...

        1. Lee D Silver badge

          And now you know that they've entered the space they weren't allowed to, and you invalidate all the passwords in that room...

          Physical access is compromise.

          1. Charles 9

            Too late. They would've already made alternate access methods.

            Compromise must be assumed to be COMPLETE compromise.

    3. Anonymous Coward
      Anonymous Coward

      Punji sticks.

      Basilisks

      Electrified if necessary

      1. allthecoolshortnamesweretaken

        Electrified basilisks? You'll get shamed by the RSPCA and raided by PETA.

    4. swschrad

      oh, yes, it comes with keys to change the combination

      you might also get a carpenter in to add a simple $30 keyed doorlock. they won't have that key. if needed, hey, firemen have axes. stockers with pallet handlers don't.

  3. Alister

    Our comms room at the office - it's not big enough to grace with the term "Server room" - was carefully planned by us IT staff when we moved buildings 6 years ago. We were given free reign to design the best possible space, as we were starting with a new build.

    It had three 42U racks full of servers, and a fourth 42U cab which held the switches, patch panels and routers which sat in the middle of the room with free access all round.

    It had nice open worktop areas round two sides, with loads of power and network outlets, and plenty of storage space for cables, components, etc, etc.

    It is currently full to bursting with assorted crap, including old office chairs, 4 car tyres, a pile of ceased network switches and routers from a branch office which closed, 2 large-screen TVs, and the bicycle, and full golf-trolley, belonging to one of the Directors.

    We can't actually open the doors to the racks without moving stuff out of the room.

    Oh, forgot to say, access is controlled by physical locks and electronic passkeys, only the IT team and Directors have access. Guess where most of the junk comes from...

    1. AndrewDu

      Rent a skip. Have it dropped in the car park. Tell those responsible they have one week to clear all the rubbish out of the Comms Room, after which it will all be in said skip.

      Now the hard part - DO IT.

      Once they see their stuff in the skip, you'll get action alright.

      1. Anonymous Coward
        Anonymous Coward

        There's a risk the Directors are over IT'S head or can more easily reach such people. Meaning there's a risk of the stuff getting swapped and pink slips attached.

        1. Anonymous Coward
          Anonymous Coward

          Well, it's nice to be qualified to the civilian nearest equivalent of a Fire Marshall and asking some fire inspector to drop by for some advice. Next up, the requirements of OSHA and state variants have the most interesting regulations which have Director frightening fines, including prison time attached.

          Whenever I've got downtime, backups or on-call time fer instances, I read engineering manuals and regulations. Aside from the weirdest shit in them, CYA dontcha know.

          1. Alan Brown Silver badge

            "Next up, the requirements of OSHA and state variants have the most interesting regulations which have Director frightening fines, including prison time attached."

            I've been told that something like this actually happened at $orkplace many years ago, with the fire brigade inspector making the point rather forcefully and clearly.

      2. raving angry loony

        Ah yes, telling a director that no, they can't do whatever they want then blame I.T. if something breaks.

        Good fucking luck. I've yet to meet ANY "director" who understands that they have to follow the same rules as everyone else. The only real choices, in a real world, are "suck it up" or "quit". Because the "director" will never, never learn. They don't have to, they're the "director". They're never wrong, and they're never at fault.

        (replace "director" with everything from "senior management position of your choice" to "owner")

        1. YetAnotherLocksmith Silver badge

          Fortunately, most see sense when you explain the long prison term for corporate manslaughter.

          Blocking the fire escape with crap in the room most likely to have a fire? The directors will be in court, explaining it, if someone gets hurt.

          It's not just a big fine anymore.

          1. Alan Brown Silver badge

            "Blocking the fire escape with crap in the room most likely to have a fire? The directors will be in court, explaining it, if someone gets hurt."

            Likewise if someone gets injured by the fire supression system.

            Some of them are surprisingly sensitive and bottles of inergen run to about £1500 a shot to refill.

        2. Anonymous Coward
          Anonymous Coward

          "I've yet to meet ANY "director" who understands that they have to follow the same rules as everyone else."

          The problem with being a director successfully is that you have to know when it's time to break the rules, and get it right. That's what directors are for, otherwise you're just a pen pusher. Even apparently open and shut legal situations can be more ambiguous than you thought once lawyers get involved.

          The way I used to assess my directors was, do they break the rules for the benefit of the company or for their own gain? The first ones are the ones to work for if you can. When I was a director, that was what I tried to do. I still got shouted at by some of my engineers, but I used to explain to them that it was my job to decide when to break the rules and take the consequences, and it was my judgement that I was paid for.

          I am not including in this arrogance and bloody-mindedness, mind you.

        3. Smedley54

          Unless you're a director, or have the support of your reporting director, it's hard to run a shop at all. Or you can play truth or consequences: Communicate clearly the problems caused by hijacking the server room or wiring closet for storage or - worse - janitorial supplies, and then just let it fail. When you come to fix it, stack the crap in the hall, make the repairs, lock the door, and leave their crap in the hall.

          There's tremendous satisfaction in staying calm while an executive gets red-faced over your explanation that an errant mop handle - that should never have been there - shut down his office or plant for the afternoon. I learned to hate the self-important buffoons, and just let them dig their own graves, documenting every shovel full.

          Allow me to excel and you'll have show piece, but if you won't listen, pardon me while I save some sanity and just put you're gear on life support. Failure to listen on your part does not constitute an emergency on my part.

    2. Anonymous Coward
      Anonymous Coward

      Claim it is a fire hazard

      Tell them you've heard from a friend who had their datacenter audited who found that having flammable items like boxes, chairs, tires and golf bags in your server room nearly got them shut down without notice until it was all quickly removed and dumped before the auditor left. Heck, I haven't read IFC/NFPA codes for a data center, but that might actually be in there.

      A business I own was recently put on notice for a few things I never realized were NFPA violations (the previous inspector liked us and must have given us a pass) like multiplug adapters plugged into ceiling outlets without support, flammable items stored within 18" of the ceiling, having a door that isn't a marked fire exit actually usable as a door (which we are fixing by putting blinds over it so people don't "know" it is a door...sometimes the NFPA requirements are weird)

      Another option could be to report the theft of some item that had been received as a free sample from a vendor (whatever excuse needed for it not to show up in a PO or in inventory) and suggest that all non-IT people be barred from access so it can be narrowed down should the culprit strike again. Give the directors an 'emergency access only' scenario - i.e. a code kept in an envelope in the company safe, which is changed after any use as part of the new more secure access policy.

      That way they can still give "tours", but the code will change so they can't go back and dump stuff in there without you knowing when their one time code shows up in your logs.

      1. Wzrd1 Silver badge

        Re: Claim it is a fire hazard

        We have extremely tight data center access control, only the minimum personnel, aka engineers have access to the data center. Anyone else has to jump through everything from information security through change management, even to get a tour.

        Needless to say, being in the information security department, I've successfully avoided a tour. I've lived inside of data centers long enough to happily skip that bit. Copious signage reminds the engineers to not leave anything that isn't a server or switch and such being authorized to be there inside of the data center. Paper and boxes are outright forbidden - devices are brought in without boxes or dunnage whenever possible.

        About two months after I had arrived at the facility, having recently transferred there, one of our information security engineers wanted to retrieve a removable hard drive from a server, where some patches were installed from the device remotely.

        He also wanted to sidestep change management in order to more rapidly retrieve the device. I wasn't having any of it and put it through change management, speaking personally with the manager in charge of change management in order to escalate and facilitate speedy removal of the offending device.

        Said engineer then attempted, to surprising success, in having our management seek authorization for me to gain access to the data center.

        A quiet conversation over lunch with the change management manager, suggesting an attempt to end run around the change management process put the kibosh on that access.

        With a bootnote from myself through the entire management chain, "If you want me to have engineering access to the data center, you're going to pay me engineering wages, grant engineering access to our systems and title me an engineer, otherwise I'll have none of it, as it's above my pay grade".

  4. marky_boi

    Luckily I work in a large Telco

    Where I am, the only real issue we have is that the A/C has been moved from what felt like -15C to more like 25. In the hot Aussie summers. the server room moves up a degree or two making for unpleasant conditions but still within the thermal envelope. Installers left their crap laying about in our room and it was relocated to the hallway duct taped to together with about 5 rolls. didn't happen again....

    1. Anonymous Coward
      Anonymous Coward

      Re: Luckily I work in a large Telco

      Our previous office had a nicely-cooled server room. So well cooled, in fact, that we used to store the champagne & beer for parties there, under the suspended floor just beside the cold air outlets.

      New building is boring, big half-empty room, tightly-controlled access.

      1. Anonymous Coward
        Anonymous Coward

        Re: Luckily I work in a large Telco

        "So well cooled, in fact, that we used to store the champagne & beer for parties there, "

        we did that too .. left-overs from hospitality supplies for leaving drinks etc, all went under the floor where the mainframe used to be (cooling had not been altered to reflect that half the room was no longer generating any heat) ...

  5. Alister
    Coat

    The headline image...

    Trevor, you shaved your beard off!!

    1. Trevor_Pott Gold badge

      Re: The headline image...

      It was int he way of my rockstar hair!

  6. chivo243 Silver badge
    Angel

    It took time

    ...but we now have all of our SER/MER's secured, finally. Besides our IT team, only building security has keys ;-}

  7. Anonymous Coward
    Anonymous Coward

    I had the luxury of restricting access, but my problem was the design of the building. The server room, while secure, did not have a separate air conditioning system. Anything I set it to was overridden by a master controller located toward the rear of the main office. It would cost far too much to alter the air conditioning system to our actual needs, of course, but that wasn't a problem in summer while everyone enjoyed the luxury of cooler air.

    Inevitably, that all changed when winter started to come around. Every few days, I'd come in to a very noisy sauna-like server room because some office flunky decided they needed to turn the heating on through the air con system or simply decided the whole thing should be turned off. 2 years later, I'd left the company with no amount of emails, briefings or passive-aggressive bits of paper taped over the master unit reminding people to bring a jumper rather than jeopardise the office infrastructure having made a dent.

    Thankfully, we did have a cabinet in a proper datacentre to put production systems, but it was always somehow my fault when the AD, email or NAS systems went down because they were trying to operate at sub-saharan temperatures before I could come in to turn the air con back on.

    1. Anonymous Coward
      Anonymous Coward

      I assume things broke when stuff got hot and went down, like drives dying in your NAS? And that cost money, right? So you should have purchased something to automatically shut down everything when it got too hot in there. Then there'd be a quick correlation between whoever turned off the AC and everything going offline, and the finger of blame would point at the right place. And maybe if everyone got tired of that happening because people wouldn't learn, you could justify the investment for a separate cooling zone for the server room! :)

      1. Doctor Syntax Silver badge

        "the finger of blame would point at the right place."

        Whatever the right place might be it always points to IT.

      2. Anonymous Coward
        Anonymous Coward

        "And maybe if everyone got tired of that happening because people wouldn't learn, you could justify the investment for a separate cooling zone for the server room! :)"

        Or they'll just break into the server room and pull out the offending sentinel, even if it says, "Fire Risk!" on it. The only way to stop that is to get to know the local fire inspector and be sure he doesn't take bribes. Sometimes, the only language they'll understand is legalese.

      3. Anonymous Coward
        Anonymous Coward

        Then there'd be a quick correlation between whoever turned off the AC and everything going offline, and the finger of blame would point at the right place.

        No, it'd be pointing at the "right" place.

    2. Alan Brown Silver badge

      " before I could come in to turn the air con back on."

      Oh, I'm sorry, it's been cooked,. It'll take 3 days to get replacement parts.

      Watch how fast you get your own AC

  8. wyatt

    Not mine but a council running a care line contact centre from a converted flat. Comms room a cupboard running 6 fairly beefy servers and the equipment we supported. There was an extraction fan, but due to the noise they kept the door shut so it didn't disturb the staff in the office next to the room.

    Constant statements about their support being revoked/null due to heat issues were ignored and their kit failed.

  9. Anonymous Coward
    Anonymous Coward

    it's your own fault!

    all my datacentres, either as a first-job junior to my current position (private, LocGov x 3, private) have always had restricted access to the data centre zone(s).

    my last local government stint, we removed the last remnants of staff (ops/sysadmin) and any stand-alone servers (replaced with rack mounted or VMWare) out of the datacentre and also had periodic purges of 'junk' with the full support of the IT Director (ie he was an assistant director of the authority, so we were represented 'at board level').

    currently, I'm even more restricted fro the DC, as have to have a change-control record or installation number in order to enter and make changes.

  10. cyclical

    I should post a photo of our newly 'renovated' server room. As part of ongoing upgrading of our 'internal spaces' they fitted a brand new suspended ceiling over the existing 60's woodchip and asbestos ceiling. Unfortunately some of the cabinets reach right up to the ceiling so they actually built the ceiling round the cabinets. In each case they actually blocked the cabinet doors from opening. Not only that, but the fire suppression was also boxed in, as were the aircon vents. Not that blocking the fire suppression would matter because although they moved the alarms onto the new ceiling, they didn't bother to actually wire them up. Not that THAT matters because it turns out they also managed to drill through some part of the roof guttering system, and about a week after putting the new ceiling up, the entire thing came straight back down the first time it rained heavily. No hardware was harmed thankfully.

    1. cyclical

      Oh, they also painted over everything in magnolia. Yes that includes cables if they were up against a wall, colour coded ducts etc.

    2. Rich 11

      I had a similar problem in an old server room about ten years ago. The Estates dept decided we weren't up to reg on the lagging of the heating pipes, so they got a contractor in to lag all the exposed pipework in that particular building over one weekend. Come Monday morning, off we go to do an early check of all the gear, only to find that the extra couple of inches of lagging meant that two cab doors couldn't be opened due to excessive pressure on the lower hinges, and that a load of ad hoc free-running cables which had been tucked behind the pipework for safety's sake (it's not like the useless heating pipes had ever actually been hot) were now permanently trapped there.

      After our complaints fell upon deaf ears, we got the knives out and fixed the problem.

      1. Toolman83

        "got the knives out and fixed the problem."

        We're the knives used on the insulation or the installers?

  11. asphytxtc

    I think the worst thing I've seen was shortly after we had a leak in the aircon unit in the "server room" (comms cabinet would be closer). The aircon was above the door and when it leaked it saturated the door to the point where it expanded and jammed itself closed. We had to drill out the lock, and a substantial part of the door and literally kick the thing open again. Oh well, these things happen right?

    Anyway, got a heap of notifications a week later after the aircon (but not the door) was fixed about temps running high in a majority of the rack so wen't to investigate only to be faced with what I can only describe as a "wall of toilet roll" behind the server room door. Yes, the cleaners had their quarterly delivery of goods for the office, and without anywhere to store fifteen hundred toilet rolls, they thought they'd re-purpose our little server room for the use of storing not just the rolls, but also bottles and bottles of various cleaning chemicals to boot.

    The door was quickly replaced after that.. and I think we have a new cleaning company too..

  12. Andrew Moore

    Ahhh...

    A couple of years ago, management decided to have a big clean out of our offices- without telling anyone. They hire a bunch of monkeys to just come in and throw anything not locked away into a skip, on a weekend. We came in on the Monday to find our entire spares cupboard stripped clean. We complained but it felt on deaf ears.

    Cut to two months later and the NIC failed on an "important" server (one that produced reports for management- so not critical). There was much screaming, wailing and gnashing of teeth, and the accusation was levelled at IT that we should have seen that kind of thing coming and we should have prepared for it- our answer: We did, but you lot threw out all of the spares that were being stored for that very reason. Queue: mumbling and grumbling and "well, you still should have been prepared"...

    1. Trevor_Pott Gold badge

      Re: Ahhh...

      Wait, have we worked together? That's my story. At least 5 different times.

      Man, this industry is depressing.

      1. Andrew Moore

        Re: Ahhh...

        The cleanout event got nicknamed "The Purge". For ages after, any management request for hardware (new keyboard, printer cartridge etc) would get turned down with the response "well, we used to have that but it got thrown out during The Purge". Eventually management relented and gave us a budget to buy up new replacements and spares.

  13. This post has been deleted by its author

  14. Alien8n

    We sort of inherited our server room from the previous occupants so it's a lot bigger than it needs to be. This means when the bottles of champagne and advent calendars are stored in there ready for December there's plenty of cupboard space for them near the door. Which is well away from the racking for the spares and build PCs which are also well away from the server racks and switches.

    Best thing is, because of who the previous occupants were, the air con unit in the server room is isolated from the rest of the air conditioning. Bit chilly in there right now, but I'm sure it'll be lovely next summer ;)

  15. Rodrigo Valenzuela

    About two years ago, I went to visit one of our branch offices. Is not so far, about an hour driving.

    That branch does not a have a server room, just a communication rack with a router, switches and telco equipment, plus a couple of UPS.

    It has never been a model of a room, because it is simply a converted broom cupboard.

    For that reason, the door is never locked. That way, in the hot days of summer, which easily brings the mercury to 35°C (~95F), the door can be fully opened to allow for some air circulation.

    I have asked for an AC for the room, but of course the answer is yet to materialize.

    On my visit, I could barely open the door.

    When finally got it open, I discovered that the room was full, from floor to ceiling, of old newspapers.

    Everywhere: on the rack, around the rack, inside the rack... it was impossible to reach switches, connectors, cables.

    I was apoplectic.

    There was so many risks... communications failure, overheating, fire...

    I talked to the office manager and sure enough, the newspapers were promptly removed.

    Only to be replaced with the janitor's stuff.

    Now, the room is used as closet.

    R

    1. Alan Brown Silver badge

      "It has never been a model of a room, because it is simply a converted broom cupboard.

      For that reason, the door is never locked."

      Hasp and padlock.

      Vent grilles on the door, with a suitable fan attached.

      And a policy of "anything not authorised to be in here will immediately go into the dumpster"

  16. dl1

    All that nice warm air blowing out the back of the racks is great for drying motorbike leathers.

    1. Anonymous Coward
      Anonymous Coward

      I have a similar use... I go swimming several times a week - there is a pool very near the office.

      So the server heat combined with dehumidifying effect of aircon, does a grand job of drying my gear.

    2. Andrew Moore

      I've been thinking of building a biltong box on the back of one of ours...

  17. adam payne

    MD gets his hand in his pocket but only after a few tricks from IT

    At the place I worked at a long time ago we had our servers in the IT office with no AC and just a small window we could open. Summer temps in there were typically anywhere between 30°C and 40°C.

    Asked for AC and of course it got rejected by the MD as an unnecessary expense that simply wasn't required.

    The IT manager and I decided enough was enough so we came up with a plan to keep rebooting the Exchange box at least twice a day when we knew the MD was in the building. After a week of the MD getting disruption to his email and us telling him the Exchange box was overheating he soon got his hand in his pocket.

    1. Anonymous Coward
      Pint

      PAOFH

      Passive Aggressive Operator From Hell

  18. Nolveys
    Gimp

    Let's see, there's the server closet in which it rains every spring. The servers in there are placed on little chunks of wood to prevent them from sitting directly in pooling water.

    There's the server closet which contains a furnace, water heater and hydronic equipment to heat the entire office. Oddly enough the temperature is somewhat reasonable in the summer. In the winter it's probably pushing 40C. With the massive amount of complex and occasionally sketchy piping I'm just waiting for the day when the server, switch and comms are introduced to a torrent of boiling water.

    Then there's the server...thing...which consists of a 12 ft by 3 ft area with a shelf mounted on one side at neck height and another mounted, on the opposite side, at gut height. Getting in there requires some interesting contortions. It's also filthy as dust from the attached shop accumulates in thick layers.

    Another one has a little broom specifically to get rid of the vermin feces. I recommend against sitting on the chair in that particular closet.

    The best one I've seen was a crawl space full of our old friend vermin feces along with fiberglass insulation and broken glass. The server was placed on some old 2x4s to prevent it from resting directly on the dirt floor. Luckily I was only at that place twice. I wouldn't be surprised if they had relocated their server to the bottom of a literal cesspit by now.

    1. Anonymous Coward
      Anonymous Coward

      I can top your rain filled cupboard...

      Back in a previous life I worked in telecoms. I went to customer site one day to assist another telecoms engineer (from a large international telephone switch manufacturer) to commission some of our gear.

      Upon arrival I was taken down to the telecoms room, which was in a basement room. The room was nice and cool... with a stone tile floor... Curious I thought. Then I noticed all the telephone switch gear was raised off the floor by about a foot on stacks of breezeblocks.

      I asked the other engineer what was going on... He replied "You see that man hole cover? Sometimes the drainage gets blocked, and the room gets flooded...".

      I looked, and indeed, in the middle of the floor was a very large manhole cover.

      I said "Oh perfect! I bet that's nice job to clean up".

      "Yes" came the reply "Especially with the business they are in here"...

      The customer was a meat processing plant!

      (Luckily I never "experienced" a flood there).

      1. Mark York 3 Silver badge

        Slaughterhouse Blues

        Yup that happened (before my time) at the slaughterhouse I worked at - The PFI (Intern) was sent in to clean it out.

    2. Alan Brown Silver badge

      "Another one has a little broom specifically to get rid of the vermin feces. I recommend against sitting on the chair in that particular closet.

      The best one I've seen was a crawl space full of our old friend vermin feces along with fiberglass insulation and broken glass. "

      Sorry, can't work on that, Environmental health breach.

  19. phuzz Silver badge

    In my last job we had a whole ex-office as a server room, which of course meant that not only were we expected to keep all the IT equipment in there (from spare PSUs to laptop bags), but we also had to share with accounts, who kept boxes and boxes of old paperwork in there. Of course, being bean counters they didn't want to pay for shelves, but the cardboard boxes collapsed if stacked more than three high, so their collection of fire hazards slowly grew to take up half the floor plan.

    At least I had one of the few keys to the room. On quiet days when the boss was away I could let myself in and catch up on kip in the warm spot behind the racks, knowing that nobody else could get in :)

  20. yoganmahew

    Fires and floods...

    Worked at one national airline that stored old CP listings in a network room. They caught fire, and the fire brigade couldn't get through the mantrap...

    Worked at another where there were two datacentre rooms connected by a very thick cable strung between buildings. The monkeys loved the cable. One monsoon season, the roof of the backup centre leaked a little. Water and mainframe do not mix well... One of the plant rooms was so cold everyone condensed on entry and exit during the buildup to monsoon. The UPS was a series of truck engines with their exhausts sticking out the wall. They belched huge fumes of black smoke when they started up, so you'd always walk a little quicker past that section of wall! Happy days :-)

  21. Anonymous Coward
    Anonymous Coward

    Never have a window

    Lost our nice and big computer room and was made to stick it in our store room.

    The old computer room in now a meeting room and because we can't store paper of cardboard nowhere to store brand new equipment.

  22. Chris King

    In an early job, it was decided that the MicroVAXen would live in a cupboard behind the reception area.

    A sparky from HQ was duly sent down to sort out "aircon" for the cupboard.

    He fitted one of those little extractor fans people use in toilets, with a thermostatic control.

    Unfortunately, he wired it all up so the fan stayed ON until a certain temperature was exceeded.

    One unusually warm Easter weekend later, the whole cluster got cooked.

    1. WonkoTheSane

      We had a similar situation. Luckily, I noticed before any kit was installed, and was believed when I pointed out how dumb this was to my PHB.

  23. Evil Auditor Silver badge

    Has your precious data centre been turned into storage? Not exactly "mine" but yes, that happened in the past. It was especially nice with all the combustible material stored that in case of fire would render the whole extinguishing system useless. (That wasn't just an ordinary server room but a data centre of one of the 30 FSB G-SIBs).

    More recently, and on a much smaller scale, it was the storage room to which a server rack was added. And why installing an aircon when we simply can remove the door and walls of the rack to allow for free air circulation?

  24. Anonymous Coward
    Anonymous Coward

    Did you beat back the barbarians at the gates?

    no, we failed totally. A quiet phone call to building security later, whenever Chief Financial Barbarian's deputy demanded access to offsite server room (ok, it was space in a colo) the building security would not let them in without a ticket / reference number, which they could only raise if they were on the authorised list (they weren't). The CFO/B went on the list, after being given a piece of paper that said if you let anyone in, from that moment until one of the IT guys has signed saying they didn't break anything, all fault reports go to Finance. :-)

    strangely, no-one ever asked ..

    1. Anonymous Coward
      Anonymous Coward

      Re: Did you beat back the barbarians at the gates?

      And IT couldn't be overridden by the rest of the Executives? Because IT is rarely on the board, who usually CAN override.

      1. YetAnotherLocksmith Silver badge

        Re: Did you beat back the barbarians at the gates?

        No, it's a co-located server, so the physical infrastructure is under the control of the co-lo company (though you may or may not own the actual computing hardware) so access and the like is down to them and their security team. And so out of direct control by your boss.

  25. Stoneshop
    Facepalm

    The computer room was kindof okay

    Some twenty racks, with assorted servers, disk shelves, tape drives and comms stuff, all reasonably orderly. Not at the 'neat' level, but I've seen worse.

    The problem was with the aircon. One of the two units had a leak, and they had already used up their quotum of CFC-based coolant so it was out of action, with the other running flat out. It had to: the room was in a 'temporary' wooden single storey building, knocked up in the 1950's or maybe early 1960's (this was 2003, so ...) with a low, tarpaper-covered roof. Temps in the adjoining offices could easily reach 35C on a moderately sunny day. To increase aircon performance they had installed a pair of garden sprinklers underneath the heat exchanger, which fortunately happened to be mostly in the shadow for a large part of the day. Around 10 o'clock someone would have to open the tap, and the last person to leave had to turn it off again. On warmer days this required staying overtime, occasionally until 20:00. Even hotter days required the back door to be opened and half a dozen floor-standing fans adding a feeble breeze to the existing airflow.

    My suggestion to visit the nearest hardware store and drop a couple of buckets of white paint on the relevant section of roof, or, even better, deploy a few rolls of reflective bubblewrap foil, was dismissed with "this is a temporary building". Which was a) actually the root cause of the problem, and b) had been its status for the past four decades.

    Yes, this was a (semi-) government department

    1. Anonymous Coward
      Anonymous Coward

      Re: The computer room was kindof okay

      Let me guess. The way this government department's budget worked, it didn't pay for water or for electricity, but would have had to pay for the paint or reflective foil?

      1. Stoneshop
        Facepalm

        Re: The computer room was kindof okay

        I suppose so, and there's that OPEX vs CAPEX thing as well.

        Mind, they could also have bought one of those lawn sprinkler timer valves which would have paid for itself in one single day, by letting the Tap Controller go home two hours sooner, but see above.

  26. Tempest8008

    Too many problems to count. I'm responsible for numerous sites around my Province of Ontario (Canada).

    Location 1: Single rack with network gear, a DVR, UPS, and ancillary equipment. The room also houses the demarc and the phone system. By company fiat ONLY IT equipment is to be in this room.

    But...

    The site has put a drinks cooler in there, propped the door open 24x7, and is now using it to store janitorial supplies and tools.

    Location 2: Same as above for the IT gear, plus a small print server.

    But...

    The site has removed all the venting from the door, resulting in NO air flow, and the night cleaning contractors put their half full mop bucket in there the other night. Hello humidity alarms at 3am...

    Location 3: The MDF has a couple of servers, 3 separate network racks, the PA system, the phone system and acts as the demarc for two buildings.

    But...

    The site has decided to store sales paraphernalia, including sweaters, t-shirts, golf bags, posters, flat packs of floor displays and everything else they can think of in there. That's in addition to all of the PCs that I have recently lifecycled that they are requiring "be stored in a secured area" before I (literally) chuck them in a big wooden crate and the company we contract with comes and takes them away...in 6 months.

    There is no longer any such thing as sacrosanct when it comes to IT gear. It used to be considered inviolate, sacred, and feared.

    Now it's in the way...until it goes down.

    Fun times...

    1. Alan Brown Silver badge

      "By company fiat ONLY IT equipment is to be in this room."

      That's your authorisation to transfer the non IT stuff to a dumpster.

  27. Anonymous Coward
    Anonymous Coward

    Piggin 'eck

    I once walked into my carefully and lovingly crafted server room, to find a pigs leg in a box on the floor. It was a huge Jamón ibérico, which cost around £500. The company had been sent it as a Christmas gift from a supplier. The leg couldn't fit in the fridge, so some bright spark had the idea that the server room was so cold, it could be used as a giant fridge.

    The crazy thing was that the company was owned by a jewish family, and most of the people there were jewish friends and family, so the thing never got eaten. It was surreal, the intended recipients couldn't / wouldn't eat it, but because of it's value it wasn't simply given to whoever in the office could have divvied it up and taken it home. It just sat there, in my server room on the floor for weeks until it had to be thrown out.

    1. Daniel Voyce

      Re: Piggin 'eck

      haha I just posted something similar, except mine was 12 x 1kg containers of Beef mince! smh

  28. js.lanshark

    Your room has been repurposed...

    A section of the raised floor is walled off so the IT department can have a decent lab (everything is tested in the lab before going into production. Everything.). Comes time to move in and we find the room full of boxes. It appears that the designated storage room (on a slab, shares office aircon) was deemed too small, so they took over the next largest space. The lab was built in the former storage area. After some heat related equipment failures, they finally relented and installed aircon. And overhead wiring trays and power runs. Everything the new storage room has in fact!

    1. Wexford

      Re: Your room has been repurposed...

      That's great! When they fill up the storage room and need more space, there's an ideal spot with identical environmental conditions waiting to be used.

  29. GlenP Silver badge

    I don't run to a server room (when we moved in to the current office I specified a minimum size, fortunately somewhat larger than I really needed as what I got is 50% of that). I do have a server "cupboard" with a 7KVA air conditioner and, in case they're tempted to use it as a fridge, a Chubb safe door! There are only two keys, one in my pocket and one in the HR department who have strict instructions to only use it with my authorisation.

  30. Anonymous Coward
    Anonymous Coward

    Server Room, we should be so lucky!

    At a Russell Group university our server room has had all the servers removed and is going to be a big open plan office for all the IT staff.

  31. Anonymous Coward
    Anonymous Coward

    Worked for a small company as their IT guy a few years ago. They decided to expand from a top floor storage unit to an entire warehouse. I was promised a shiny new server room, with air conditioning and enough space to store all my kit.

    I dutifully went down a few weeks before opening as the office was being put together, measured and installed all the cabling, wired in the UPS and set our single server rack up.

    All that would be left was a Friday evening shutdown and journey 5 miles down the road.

    The boss was conspicuously absent all day. As I arrived I found him trying to unclip all the wiring and reroute it to a stationary cupboard and my racks thrown out because they wouldn't fit in said cupboard.

    His wife had decided she wanted to have a job and needed an office for all 8 hours a week she worked. I was very angry. and refused to rewire or help. Quit soon after.

    1. Anonymous Coward
      Joke

      > His wife had decided she wanted to have a job and needed an office for all 8 hours a week she worked. I was very angry. and refused to rewire or help. Quit soon after.

      You should have calmly offered to let her work on the help desk. :-)

    2. Robert Helpmann??
      Childcatcher

      Quit soon after.

      The boss was conspicuously absent all day. As I arrived I found him trying to unclip all the wiring and reroute it to a stationary cupboard and my racks thrown out because they wouldn't fit in said cupboard.... I ... refused to rewire or help. Quit soon after.

      This! This is how it's done! I have had to do this myself. Either the people you work for and with will back you and value the service you provide or they are going to continue to make whatever you do worthless. Much better to find something worthwhile to do than to stick around in a messed up situation that you know is going to stay that way.

      1. Disk0
        Thumb Up

        Well put

        "Either the people you work for and with will back you and value the service you provide or they are going to continue to make whatever you do worthless."

        You put it into perspective nicely!

        I've found myself in the latter situation too often, and wondered how come I got demotivated, stopped producing anything meaningful, and felt generally worthless despite normally being reasonably capable, productive enough to pass general muster, and capable to be cheerfully cynical.

        I've always written it off to lack of inspiration in unimaginative environments, but the notion of value seems much more relevant.

        1. Anonymous Coward
          Anonymous Coward

          Re: Well put

          Trouble is, with the job market as bad is it is, sometimes you're against the wall. You either put up with it or can no longer support oneself because there are no other opening (all the good ones already taken).

  32. Anonymous Coward
    Anonymous Coward

    Here's what my company does...

    They have a big impressive Server Room full of blinking lights and twirling fans.

    But then they run the whole company off of an under-clocked Raspberry Pi.

    At least that's what it seems... LOL.

  33. Alistair
    Gimp

    @header pic

    Whomever chose that header image for the article has ...... a kink or three to work out.

    <appropriate icon?>

  34. ecarlseen

    Paper trails are your friends.

    I think that it's important to remember that part of our jobs in IT is risk management. This is part of the lingo that most upper management speaks and understands, and so if you can communicate these things in their language you'll have an easier time getting your point across. The other thing is to NEVER have risk management discussions verbally, or if you do be sure to IMMEDIATELY follow up with a memo to everyone involved outlining everything that was discussed. Be fair about it and give everyone's position a respectful treatment (regardless of how stupid it may be), or it will backfire. This sort of thing is just as important as running proper backups. This way when poor compromises are made about physical access (or a myriad of other issues) and it eventually comes back to bite everyone, you have clear documentation of the decision-making process. If done properly these sorts of disasters wind up enhancing your position rather than leaving you as a scapegoat.

    It's important to remember that no matter how smart you are technically, your position will always be limited if you can't interface your ideas with other departments. Like most areas of contention, you will be most effective if you are capable of respectfully making the other sides' case better than they can (even if it's dead wrong). Once you can do that, you will be better able to persuade them of your position. Think of it as a hacking challenge or something like that.

    1. Anonymous Coward
      Anonymous Coward

      Re: Paper trails are your friends.

      Just remember that a paper trail should include real paper for the case of utter disaster. Whenever I was developing rules and procedures, I'd have a folder with all the inputs for the ultimate question :Who fucked up? Why?

      1. Anonymous Coward
        Anonymous Coward

        Re: Paper trails are your friends.

        So what happens when that room you keep the paper trails in has a FIRE? And given the environment, odds are passing fair any fire there would beat a fire safe.

  35. Alistair
    Windows

    storage of IT equipment.

    Functional or not as the case may be.

    Most of the IT equipment I deal with lives in proper DC space.

    Every once in a while something in a network/comms closet gets thrown my way.

    Things I've found in said network/comms closets in my travels:

    Bicycles (one closet had four. One of which was in numerous parts, including what looked like having had part of the frame cut using a hacksaw, the metal shavings from said operation still lying on the 'raised floor' tiles in the room)

    Old TVs/Display panels (usually stood up long ways against the backs of the cabinets.)

    Bundles of fibreglass insulation, pink, yellow and green. In one case, neatly taped into packs of three slabs, and stacked in the space between the back of the racks and the wall. Tightly packed of course.

    Cases of empty beer bottles. (in that one, 248 two-fours, stacked against the aircon unit, blocking airflow *into* the unit ....)

    dead PCs. from a cleanout of a callcenter as far as i could tell. I made it to 150 before I stopped counting.

    Broken toilets (just sitting in the middle of the room, chunks of shattered ceramic)

    Unused desks, unused chairs. A very large, sloppily decorated plywood box. As far as I could tell it was a magicians "cut someone in half" box.

    An 8 cylinder diesel engine, and I'm guessing the clutch to match it.

    7 years worth of tax filing documents for a subsidiary of a company that my company bought. Each year was in a several file folders. The file folders were laid flat in a rack, and stacked on top of one another. It appeared that the computer rack was dedicated file storage. (there's a joke in there somewhere)

    4 (4'x4'x4') cardboard boxes full of nothing but those stupid styrofoam peanuts.

    Once, an old old OLD freezer. Locked. Rusty. most definitely was NOT empty.

    in one case in excess of 80 bottles of photography chemicals, loosely arrayed on shelves in a rack over the PDUs that fed the equipment in the next 2 racks.

    I try not to go to weird locations to recover hardware any more. It has left me with various nightmares.

    Oh- and the birds nest. There were eggs in it. It was rather dusty so I doubt that the eggs were likely to hatch, but it also made me wonder where the parent(s) were.

    I know someone that had the 'farm computer' experience. At least I'm not dealing with *that* level of ........

    1. MonkeyCee

      Re: storage of IT equipment.

      "Once, an old old OLD freezer. Locked. Rusty. most definitely was NOT empty."

      My bad. No carpet in the van that day.

      1. Alien8n

        Re: storage of IT equipment.

        A friend of mine was tasked with disposing of an old fridge at a very prestigious university frequented by the higher echelons of Parliament. He took one look inside and immediately refused to have anything else to do with it. They had to get a hazardous waste team in for it. Turns out the fridge still contained samples of Anthrax that had been forgotten about.

  36. The humble print monkey

    Serfver cupboard

    A photography company I worked in, early 2000's, went from analogue (E6 line in the basement) to digital, and suddenly we required proper server, storage and back ups. Small outfit, with a very hands on approach. Researched and ordered server, raid, external backups (we were new to this, and on a tightish budget). Installed said on shelf in cool, dry basement in a part of the office (the noise was fine - the music from the studios drowned out the drone of the fans and drives).

    Went away one fine easter bankholliday, safe in the knowledge that everything would tick over in my absence. Came back a few days later to find the server and drives no longer visible. It being early morning, the 'togs were not in, and the music off. The boss - something of a tidy fairy - had taken all the ugly computer kit, and placed it in an under the counter cupboard. I located it by a combination of the sounds of the fans screaming, the drives clicking and screaming, and the worksurface temperature of being in the too hot to touch range.

    1. Anonymous Coward
      Anonymous Coward

      Re: Serfver cupboard

      Did you leave it in-situ, screaming as before, then ask the perpetrator to put their hands on that work surface?

      Sometimes it takes a practical lesson to explain why something is wrong.

      1. Anonymous Coward
        Anonymous Coward

        Re: Serfver cupboard

        And sometimes you find practical lessons are lost on some people: like someone who routinely cooks and is used to hot surfaces.

  37. Kernel

    The best I've found

    was back in the days when electro-mechanical PABXs were the standard - in this specific case, what was known as a BPO* 100-type.

    My off-sider and I went into the PABX room at a small business and found the manager's outboard motor leaning against a wall, complete with a partially full tank and a general aroma of petrol. Cue very prompt removal of motor and petrol followed by ventilation of the room.**

    We prevented this from happening again by removing the all external and internal covers from the PABX (especially and particularly the 'pulse set') , taking the manager into the room and letting him watch calls being made - with the lights out, so he could see the sparks more easily.

    * British Post Office, for all you young'uns.

    ** This was many years ago - nowdays we'd have to have evacuated half the block and called the Fire Service, Police and $deity knows who else to deal with it, as well as filing sundry H&S reports.

  38. SteveK

    In [old job] the servers were under a table in the open plan IT office. On hot days we'd open the fire exit door out to a strip of grass alongside the building. On one hot summer I'd brought a couple of old 120VAC industrial fans that I'd ripped out of something and wired together and had them in front of the door to vent hot air.

    One day heard a strange vibrating noise coming from the fans and discovered that an escaped tortoise had wandered in through the open door and was busy sticking its head into the fan (which was missing its protective grill).

    The tortoise was completely fine (and was returned intact to its owner), it moved its head so slowly that the tip of its nose was just being brushed by the fan so it would move backwards then slowly extend head again, which was fortunate as I'm not entirely sure how I could have explained it otherwise.

    "The blood all over the carpet and servers? Well, you know how you always said that getting SCSI to work first time required a sacrifice..."

  39. kain preacher

    I was on call when My beeper went off. Server room is 100 then 120. Buy the time I got into the car it said 220 and stopped . I get With in a mile and I see fire trucks with lights blazing. I'm like please don't turn left please don't turn left. It did along with five more trucks. There wire fire engines from other cities. Oh did I mention that they had diesel generators in the basement. I call my boos up. He was pissed that I woke him. I told he did not have a job as the building burnt to the ground.

    1. Kiwi
      Mushroom

      He was pissed that I woke him. I told he did not have a job as the building burnt to the ground.

      Any chance someone stole your stapler a few days beforehand?

      (Someone please tell me what that movie is called! I wanna see it again!)

      1. ISP

        (Someone please tell me what that movie is called! I wanna see it again!)

        Office Space.

        1. Kiwi
          Thumb Up

          Thank you most kindly.

          May the fleas of a trillion camels infect the armpits of a thousand of your enemies!

      2. kain preacher

        I would of burn his house for that infraction.

  40. Daniel Voyce
    FAIL

    Server room - dont you mean Fridge?

    Yeah I'm not even exaggerating here, I got some temperature warnings from our Modest server room (it was basically a 6 x 12ft studded walled off area with an air-conditioner in it that had a border line thermal profile vs the BTU of the aircon unit, the downstairs had a lot of CD burning robots in there and in summer the whole of the downstairs block got pretty warm.

    So I ambled downstairs to see if someone had left the door open, I walked in to the server room to find about 12kg of mince sitting on top of the Tower RAID we had built (that co-incidentally had a top exhaust fan).

    There was apparently no room in the office Fridge and this was the next coolest place....

  41. Triggerfish

    Claymore

    Mine preferably, sword will do.

    Alternatively get the guy at the top, show em the rack currently being used as a coat rack / file storage space, point out how much all the stuff cost him. NB Also like a claymore do not stand in front of after activation.

  42. Wexford
    Facepalm

    Racks are for files, not switches

    Routine visit to our communications closet, only to discover the switches hanging at the side of the rack by their power leads, with the rack shelves themselves occupied by lever arch files.

  43. mirza.aziz

    Access control system controlled with access cards and permission to small group of tech staff on that particular reader.

    Couple of CCTV cameras recording 24/7 activity inside server room will save your ass.

    1. Anonymous Coward
      Anonymous Coward

      How do you avoid getting overridden by people over your head who can demand "your card or your job"?

  44. Demosthenes Locke

    Probably the most important rule about your server room is...

    ...don't let the office manager design it.

    Seriously, nobody but an IT professional should be designing a server room. Case in point, a law firm I worked at for a summer bought a lovely PDP-11 for internal data processing (yes, this was a while ago). They set up a lovely little room in the office with a picture window into the conference room, so they could show it off. There was lots of air conditioning and electrical stuff, everything that anyone could want.

    When they unveiled the new computer, they showed it off to the staff, letting us oooh and ahhh at it through the lovely picture window. But when I trooped dutifully past the window to pay homage to our new electronic overlord, I turned to one of the firm's partners and said: "Why is there CARPETING in that room?" Indeed, they didn't put down a proper floor. They left the same beautiful thick-pile carpeting in the room that was in the rest of the office.

    He didn't get it. "What's carpet got to do with anything? That's expensive wool carpet, none of that cheap synthetic stuff!"

    So I reply, "But it generates STATIC ELECTRICITY! There's a reason why computer rooms have those nifty raised floors!"

    He tut-tutted it away. "Those floors are expensive hoo-hah. There's no reason for them, and they're ugly. We want this to be a centerpiece for the office! You can't use those silly computer floors for that!"

    So I finally said, pleadingly, "Please tell me you at least had it treated with an anti-static preparation? The first spark that hits that computer could fry it!"

    "Pish and tosh," he said. He actually SAID "pish and tosh", even though this was in Chicago. "We've been assured that this is a perfectly adequate installation. See, I'll show you!" He walked over and opened a door into the computer room. I followed, staying as far from the machine as I could, lest I be accused of what was to come. He strode purposefully to the machine, proudly, and waved a finger toward the front panel. "This is a state of the art..." He didn't finish the sentence, because a fat spark leaped from his finger to a switch fitting on the front of the machine. There was a loud SNAP. And the machine went DARK. It stopped whirring powerfully. It went silent.

    I said NOTHING.

    He turned around and looked at me and said "Don't. Say. It."

    I didn't have to. I simply smiled a little smile and beat a hasty retreat. I hear it cost about 5 grand to fix the machine, and another 10 grand for a proper floor.

  45. Slabfondler

    Had a server room (for academic computing at a small college campus in the late 80's) that was shelves behind my desk. We grew beyond that small space and had to move them to racks in the main electrical distribution room, which had open vents to the outside, no issues with cooling, there were snowdrifts in the room in the winter :)

  46. Unicornpiss
    Meh

    Cardboard and Hypocrisy

    At a past job, a self-proclaimed CIO decreed that anything paper or cardboard must be removed from the server room. Apparently for fire safety, or maybe he just didn't like the look of cardboard. We never found out. We got dinged for having a paper calendar with our backup schedule on one of the cabinets and a tiny cardboard box with new backup tapes.

    Several months later, a conference room was being remodeled. (taking part of our server room to make more space) All the carpet, boxes of new fixtures, and misc. crap was stored, you guessed it--in the server room, along with drywall. (and associated dust) I guess it's not a fire hazard or eyesore if it's part of progress.

  47. John Geek

    many many years ago (ok, was about 1975), I was working for a small computer company who had a customer that was a exotic steel alloy mill. the factory was full of these enormous arc furnaces being used to make fancy aerospace alloys. the computer was in a portable building on stands in the middle of this factory. we had to cover the whole porta building with copper sheeting, soldered together and grounded at multiiple points with 2" wide braid, heavy copper screens over the windows and door, to keep the EMF from crashing the system. we had to isolate the power with a motor+generator where the outside power grounds were NOT connected to the copper cage, only the generator was (and massive ground stakes). thankfully this was before networking was common, so there were no external connections to this room, not even modems. lets not even get into the problems we had caused by microfine metal dust getting through the ventilation filters and into the systems, those were mostly mitigated by stacking multiple air filters and changing them weekly, and keeping positive pressure in this room by way of enormous fans (outside the faraday cage, powered by the building power).

    1. Alan Brown Silver badge

      "lets not even get into the problems we had caused by microfine metal dust getting through the ventilation filters and into the systems"

      Next time you have to deal with that kind of thing, bear in mind that air turns corners better than microfine dust does and add a centrifugual intake filter followed by an electrostatic plate. It makes the world of difference.

  48. Potemkine Silver badge

    BTU? Pweah!

    Never heard of 'Joule' to express energy?

    1. GruntyMcPugh Silver badge

      Re: BTU? Pweah!

      The datacentre modelling software we use measures in BTU.

  49. returnstackerror

    hot to the touch

    Customer was slow in adding capacity to the server room so the new multi-million dollar system was placed in a temporary room which was cooled by portable air con units (their idea)

    Came in on Saturday morning to do some work, touched the door handle and burnt my hand.

    The portable aircon had failed for some reason (circuit overload?) but amazingly after shutting the system down and letting it cool naturally for a few hours, it all sprung into life.

    Did diags on everything (even the drives) and it all checked out.

  50. GruntyMcPugh Silver badge

    Current place of employment has a lovely neat airconned, and fire suppressed data centre, with card and PIN door access, and we have a spare hall we use for storage, so all is good.

    Previous job when I worked in schools was a world away however. Servers and comms crammed into cupboards, and one time I went to diagnose a networking problem, opened up a comms cabinet, to have footballs and hula hoops fall out. Don't fill all those U with Cisco, and some teacher finds the space irresistible.

  51. Anonymous Coward
    Anonymous Coward

    Anon, oh yes.

    Following a large fire, thought to have been caused by a 300KVA UPS overheating, we moved into temporary 'lodgings'. IT got the former broom cupboard in what was formerly a Unigate bottling plant, situated in the South Western corner of the building. Year round the building was cold, until they cut the tree's that shaded the building down, the broom cupboard got just a bit too warm on summer afternoons after that.

    After a few years, and £55m our new building was ready to occupy, great, server rooms on several floors, atleast one for each dept. The Computer Science dept. got the top floor and the biggest server rooms, we got one on another floor. Aircon was just that a con, the system installed by Merits monkeys used cooling water to cool the room, the water being cooled by the same chillers, screw and an abbo that needs hot water at 97C to work properly and huge roof mounted cooling towers that also cooled the process equipment, so when we needed it most the process kit was heating it too much, especially when the cheap screw chillers started failing. Much complaining by the IT team and technical staff had little effect.

    First building wide shutdown for 'plant' maintenance, after being told we would lose all cooling and power to the racks we kicked off, so they wheeled in 3 foot diameter fans to blow air through the server rooms, bang went any security at that point, all powered by the same extension leads from the next building that would keep the racks running. Over the shutdown the fans were unplugged several times, once by workers doing the shutdown due to the noise, and several more times by s-too-dense who though it was funny to unplug the leads in 'their' building. Having pointed out the server rooms needed independent power and cooling previously the management relented and installed some old split unit aircon units, fighting all the way not to install proper aircon. These were to back-up the water cooling system, not replace it, after 8 years we finally got proper aircon, but as you might have guessed it's not independently powered... Then we discovered Estates had issued keys to every server room, so the contractors could check on the cooling without calling on the IT pro's first.

    So when your computer science lecturer teaches you about computer security, and how easily you can pick the locks, so therefore so can others, remember physical security comes first, not just to 'secure' the equipment but to keep the cleaners and all the other dross depositors out, 'space, the final frontier' takes on a whole new meaning.

    1. Alan Brown Silver badge

      Re: Anon, oh yes.

      "Then we discovered Estates had issued keys to every server room, so the contractors could check on the cooling without calling on the IT pro's first."

      Change the locks. Don't give copies to Estates.

      1. Anonymous Coward
        Anonymous Coward

        Re: Anon, oh yes.

        Estates calls locksmith, forwards bill to your department. I may be wrong, but Estates probably has legal responsibility for the physical plant, meaning they have the authority to overrule you regarding locks.

  52. Kiwi
    WTF?

    If you're sensitive, DON'T READ!

    Came in to my work early one day - systems are down and need to get in and fix them.

    Why?

    Well.. The same manager who was wanting machines up and running immediately (the factory was kinda dead after all!) also wanted a larger office. And the server room had a wall that could be moved if those racks were moved.

    It gets worse.

    It wasn't IT staff called in to shut down the machines and move them so the wall could be moved. No. It was the fucking builders. Who, given the dust build up, had been at work for some time with saws and who-knows-what-else before the first machine was shut down. Shut down? I mean, the power plugged pulled out while they "gently" moved it.

    But hey, the're only computers. Not like the housed any important data (accounts, designs, other unimportant material like that) or any important machines (data for the various control machines, coms, etc etc etc).

    We did get some newshiny as a result, but at a hell of an expense.

    Oh, and I have held back considerably in this as well - not just the detail of the offence but what was said (and some of what was done ;) ) to those responsible afterwards. I do still have some rage-events at the memory of this, and it was a very long time ago.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like