back to article Heatwave shmeatwave: Brit IT departments cool their racks – explicit pics

Never let it be said that techies aren't agile or innovative – and perhaps a little slapdash – when solving problems on a tight budget. Only this week, El Reg was given early sight of a pre-patented swanky new cooling system. IT departments at businesses that asked to remain anonymous, for obvious reasons, have devised a …

  1. Anonymous Coward
    Anonymous Coward

    Once upon a time

    I, and half a dozen other people, worked in a stuffy windowless basement which was where the prototype hardware we were developing ROM code for resided.

    In vain we pleaded for aircon. The temeperatures were up to 27 degrees - even 30.

    I was working the ICE - In circuit emulator to the initiated, A vast box of power hungry ECL (emitter coupled logic) and schottky TTL that on a fair day with a following wind could pretend to be an 8086 sufficiently convincingly to be of use debugging the firmware.

    Except when it wasn't. One day code that had previously run well, stopped. Analysis of breakpoint audit trails showed nonsensical behaviour. Opcodes were simply not being executed....as they should.

    I reported to my boss 'The ICE has gone west' .

    "Why?"

    "Dunno. It is getting pretty hot. "

    And we grabbed the manual and leafed to the back

    OPERATING TEMPERATURE.15C-27C AMBIENT

    Now I came from a hardware background, and I knew as well as anyone that those chips had capabilities to go over 100C without popping, and the thing was fan cooled, so the only reason they would have the temperature rating so low was that it was on the bleeding edge built out of selected chips and the timings would not hold out beyond that.

    The thermometer one of the permies had installed on the wall said 28C

    "Leave it with me" said the Boss

    The next Monday we arrived to find air conditioning had been installed.

    Its always nice to know management values its ICE more than it does its coders

    1. Anonymous Coward
      Anonymous Coward

      Re: Once upon a time

      "Now I came from a hardware background, and I knew as well as anyone that those chips had capabilities to go over 100C without popping"

      Ah, ECL. Runs hot but the problem is, as you note, that being non-saturating its parameters are very temperature sensitive, so unless you have an expensive piece of kit with cooling, the temperature range is limited.

      IIRC there was a Nat Semi ECL minicomputer which had a self test routine where the tail voltage and frequency was adjusted for the CPU periodically.

    2. John Brown (no body) Silver badge

      Re: Once upon a time

      "Its always nice to know management values its ICE more than it does its coders"

      Bloody wimps whinging about heat in a room full of ICE!!

  2. Keith Oborn

    Some years ago-

    At a large machine room in Tokyo owned by a well known consumer electronics manufacturer whose (short) name begins with "S":

    A long row of very heavily loaded racks, with front and back doors off, and in front of each one a standard tall office fan set to maximum.

    The entire division's operations depended on those fans working.

    1. Yet Another Anonymous coward Silver badge

      Re: Some years ago-

      Hopefully all plugged into a series of daisy-chained extension leads back to a single wall socket ?

  3. Andytug

    Never mind the H&S of attaching fans with cables

    If that's a serial cable (as used for attaching laptop to back of switch for maintenance) then at our place it would be a security breach as could be used to access, reset config etc.

    1. steamrunner

      Re: Never mind the H&S of attaching fans with cables

      If someone can get close enough to plug a serial cable into your switches, you have a more serious problem than unauthorised console port access.

    2. Stoneshop
      FAIL

      Re: Never mind the H&S of attaching fans with cables

      If someone can access the equipment at all you're just waiting for a security breach.anyway, cable or not.

      1. Korev Silver badge
        Coat

        Re: Never mind the H&S of attaching fans with cables

        But they need to err 9 pin it on someone

  4. wyatt

    Glad they're using fans.

    Customer 1) 6 servers in a converted kitchen with 1 extractor fan (there due to the previous function) taking heat out. Warned on many occasion they wouldn't be supported due to overheating which did end up costing them for repairs to overheated kit.

    Customer 2) Survey completed in a partitioned off area to add a server. No aircon, room was already warm to hot and they wanted to add more heat? Advised they'd need to provide cooling before we'd be able to provide a quote which they wouldn't.

    Customer 3) our server has gone down. On site and no air con, it had failed months ago. Manglement had refused to pay until we refused to fix (at their cost..).

    Seems there's a trend with cooling not being taken as seriously as it should.

    1. Anonymous Coward
      Anonymous Coward

      "Seems there's a trend with cooling not being taken as seriously as it should."

      All time favourite? Company with managed sir conditioning system. The system developed a fault in the server room one night, the technician turned up to repair the fault...to find nobody there because only the IT department could get into the server room and none of them answered the call.

      It was an expensive mistake.

      1. Brad Ackerman
        Devil

        Re: "Seems there's a trend with cooling not being taken as seriously as it should."

        The fire department can open that door faster anyhow.

      2. StargateSg7

        Re: "Seems there's a trend with cooling not being taken as seriously as it should."

        The BEST company server room I have ever seen is a local one in Vancouver, Canada which I had a tour of last year. It used a Silicone Oil liquid immersion system. They rack many quad-cpu or eight-way SuperMicro motherboards into what looks like large home refrigerators but are in fact filled with silicone oil which is NON-CONDUCTIVE and is allowed direct contact with the electronics of the motherboards including the GPU's, RAM, DRIVES, etc. A 16 kilowatt power supply (ALSO submerged!) powers each large box which is about 2 metres tall by one meter wide by 75 cm deep. The silicone oil is pumped into the bottom and flows up through each VERTICALLY PLACED motherboard and comes out the top warmer than it was at the bottom. A external radiator and condensor assembly uses huge radiator fins on the roof of the warehouse-like building to get rid of the heat which then recools the silicone oil which is pumped back to the bottom of each sealed rack system for reuse. Since BC Hydro supplies electricity as cheap as 8 cents per kw/hr if you buy blocks of it up front, their mains cost are VERY LOW compared to other parts of the world which is WHY they are located in the Vancouver area!

        There were 150 racks containing 12 motherboards each with over 7200 8-core CPU's (that's 57600 CORES!) doing fast algorithmic day-trading. They could afford it! I was told by a 3rd party that they were pulling in over Two Million Canadian Dollars GROSS RECEIPTS A DAY 24/7/365 for the less than 10 shareholders...NOW EVEN WITH TAXES AND EXPENSES THAT IS STILL A MONSTER INCOME from a bunch of CPU's dipped in silicone cooling oil!

        Some people take IT cooling VERY VERY SERIOUSLY!

    2. tfewster
      Facepalm

      I've seen many repurposed broom cupboards. Plus a specialised computer room, which would have been great if the computers hadn't been moved in before it was finished. The servers were lifted (and dropped) by the contractors laying the floor covering. And covered in little piles of brick dust where shelves were being put in.

      One of my employers saw the light, and moved the servers and noisy high speed line-printers out of the general office to their own room. With not just an extractor fan, but external air from the cool side of the building sucked in! Unfortunately wasps built a nest near the intake one year, and we had a computer room full of dead wasps.

    3. Yet Another Anonymous coward Silver badge

      Our customer partitioned off part of the kitchen for the server racks, no AC or extractor fan but lots of steam, fat and smoke - because the printers were on the other side of the wall and so they could just knock holes in it for the cables.

  5. Anonymous Coward
    Anonymous Coward

    FAKE NEWS!

    That "The Bristol Air Scoop" corrugated paper box should have the markings of Amazon on it ...

    It's the little details that give it away...

  6. Giovani Tapini

    I've done this too

    Over 20 years ago, so no smartphone pics.

    Aircon popped in computer room. Getting toasty rapidly.

    Prop open all the fire doors etc, grab, commandeer, steal deskfans from all over the place, step carefully over the spaghetti of cables and watch is roar. It worked, just about, for long enough. Back in the day of the really big hard drives without all these fine tolerances :)

    although there was another occasion (minus the fans) in a different computer room where aircon poppage caused the heat buildup to be so intense that ingress to the room to switch off the old minicomputer was itself tricky and required hands to be covered with jumper to execute. This resulted in melted 8" boot floppy, and I am sure you could have cooked breakfast on the top of the racks. took a day or so to cool down! The drives were built like engine blocks and held their heat nicely!

    1. Stoneshop
      Flame

      Re: I've done this too

      Over 20 years ago, so no smartphone pics.

      Aircon popped in computer room. Getting toasty rapidly.

      Close to 35, in my case: I'm in the computer room for some reason when I hear the dull roar soundscape changing. After a couple of seconds it dawns on me that it's the aircon rumble that's absent now. Which means that there's now close to 100kW going in every second that's not taken out, which is clearly suboptimal for the continued operation of the equipment present. I sprint out the door and into the sysadmin pen, and alert those present. Half get assigned to raiding the offices and confiscating any fan they see, the others shut down and switch off all the systems and storage not deemed absolutely essential. It's the one time I've seen a thermograph move, climbing 10 degrees in as many minutes.

      We lost maybe three RA81 HDAs and one logic board out of well over a hundred, and one memory board from one of the systems.

      1. GlenP Silver badge

        Re: I've done this too

        We lost maybe three RA81 HDAs and one logic board out of well over a hundred, and one memory board from one of the systems.

        It was tough kit! Once had a VT-240 terminal that had been in a fire (sadly the VAXes didn't survive). The case had partly melted but when we powered it up it still came up with a reassuring "VT-240 OK"

      2. Alan Brown Silver badge

        Re: I've done this too

        "After a couple of seconds it dawns on me that it's the aircon rumble that's absent now. Which means that there's now close to 100kW going in every second that's not taken out"

        You really _REALLY_ have to wonder why with millions of squids worth of kit and redundancy up the wazoo, manglement cut corners and allow a single point of failure for the cooling.

        1. Stoneshop

          Re: I've done this too

          You really _REALLY_ have to wonder why with millions of squids worth of kit and redundancy up the wazoo,

          Eh, what? Redundancy? At that time there were a dozen 785/8600/750 systems, half of them in a cluster, the others standalone (although the two PDPs could run the other's tasks if one of them failed) plus a scattering of MVII's. Disks: lots. Four HSC50s, maxed out, for the cluster. Plus some RAs for the standalones. PDPs ran off 4 RP06es each. I don't think there was disk mirroring in use (can't be arsed to look up if that was even an option with VMS 4.5), but there were a couple of unused disks which were used as lukewarm spares when needed. They didn't even have a no-break until the site became crucial as a comms hub for NW Europe. Which was close to a year later.

    2. Doctor Syntax Silver badge

      Re: I've done this too

      "Back in the day of the really big hard drives without all these fine tolerances"

      I've seen a similar thing but back in the day of massive optical disk library which did have fine tolerances. I think the robot had to be recalibrated.

    3. Chris Evans

      Re: I've done this too

      "Over 20 years ago, so no smartphone pics."

      Back in the mist of time you get a gizmo that was normally black with silver dials and buttons on it about the size of a smartphone but a lot thicker. They were called Film Cameras! I wonder what ever happened to them...

      1. Anonymous Coward
        Anonymous Coward

        Re: Film cameras

        You’ll recall that film cameras have no more than 36 photos available before the SD card, sorry, film canister, needs to be ejected and then developed.

        So, we weren’t really in the habit of snapping every random thing that took our fancy, with the gay abandon that we are today.

        Every frame was sacred!

      2. John Brown (no body) Silver badge

        Re: I've done this too

        "They were called Film Cameras!"

        No, they were just called Cameras. There was no other type until those new fangled electronic Digital Cameras came along :-p

        1. Richard 12 Silver badge

          Re: I've done this too

          A canister full of film with 36 photographs!?

          You youngsters with your new-fangled "film" cameras don't know you're born!

          One glass sheet is all you used to get.

    4. prodromos65

      Re: I've done this too

      Got a late night call out on New Years Eve as the Comms room for a Law firm in one of the buildings we looked after had gone down. It turned out that they had scheduled maintenance on the Comms room cooling system that night as network traffic was at a minimum. In order to keep things cool while the crack units got serviced, several portable A/C units had been brought into the Comms room which the A/C contractor had plugged into power outlets on the wall of Comms room, all of which were backed up by the Comms room UPS.

      Queue UPS overload alarm followed by immediate power loss to all computer equipment in the room.

      It took me a while to get there as the CBD was closed to all traffic on New Years Eve and I had to walk from a couple of kilometers away, so by the time I got there they had already figured out their mistake and had everything running again.

      Had another callout to a bank branch that was suffering from heat related issues. The A/C plant on the roof had broken down so an A/C company had brought in portable units to keep things cool, with the hot air being pumped into the false ceiling which acted as the return air of the currently not working A/C plant. Thus, all the hot air came flowing out of the return air vents in the ceiling and the place just got hotter and hotter, since no one had actually thought about how they were actually going to remove the heat from the building. Their computers were already slowing down due to CPU throttling and it would not have been long before their network equipment shutdown.

      I explained the basic thermal dynamics to the manager, and the fact that they would soon be unable to trade when their network died. They craned a large temporary unit onto the roof the next day.

  7. Anonymous Coward
    Anonymous Coward

    At my old place, management had declined requests for aircon in the equipment room - apparently they considered the existing arrangement, whereby hot air was simply fanned/ducted into the gents' loo next door, was a perfectly adequate solution.

    During a particularly hot summer a piece of melted/distorted hardware was ceremoniously dropped on the MDs desk with the statement "that used to be part of a business-critical server, and that's why we need aircon"

    1. CrazyOldCatMan Silver badge

      management had declined requests for aircon in the equipment room

      One place I was at we were combining two computer rooms into a new building. I'd (privately) done the power and heat calculations and so requested a certain UPS and aircon setup. Unfortunately, they'd already bought the UPS and aircon setup as part of the building fit-out.

      Both of which were nowhere near enough. They would have been OK for either of the old computer rooms but not for the new, merged, room.

      By the time we got to 60% of the move the UPS went into power bypass because the power draw was too high. And because of the room configuration, we couldn't add extra capacity even the the UP was expandable. And the aircon was a two-chiller design that, in an ideal world, would have let the whole room run off just one chiller so that we could take the other one down for maintenance. But again, we had to run both units of full to dissipate the heat - and soon discovered that the 'architect' who had designed the room had put the chiller drip-tray drainage above the server racks..

      Not one of the finest fit-outs that I was involved with - I left that place two months later.

      1. Alan Brown Silver badge

        "Both of which were nowhere near enough. They would have been OK for either of the old computer rooms but not for the new, merged, room."

        BTDT. The result was a server room which was maxxed out thermally at 1/3 it's physical capacity with manglement wanting to put desks in there. Explaining the legal ramifications of putting people in with _lethal_ fire suppression kit made them go a little white. ("manslaughter charges" were mentioned)

  8. Anonymous Coward
    Anonymous Coward

    Desperate times demand desperate measures

    Where I work earlier this year, at the beginning of the heatwave, the machine room just below where I sit suffered a total air conditioning failure.

    By the time someone followed up the alerts, the ambient temperature was heading towards 40oC. What it was inside the racks we don't know, but it would have been much higher.

    As we had just recovered from a major storage failure a few months earlier, and did not want to have to do the same again, the managers put everyone on an emergency footing, but decided not to do an EPO, but wanted a rapid ordered power down.

    Well, that made a very busy afternoon, shutting down 100+ servers and the associated comms. and storage gear in the correct sequence, and in order to try to slow the rise in temperature, every single desk fan was commandeered, as was the portable air-con unit that was keeping the coffee shop cool. All the doors and windows in the corridors around the machine room were wedged open, and security were told to ignore all of the warnings they were getting. The managers themselves manned the doors and open windows to make sure that nobody tried to get in, while we did the technical work.

    Fortunately, most of the systems in the room were for test and development, so the external facing systems were not affected.

    Never seen the managers, project managers and PMAs do so much physical work!

    It must have worked, because we did not suffer any more than the normal number of failed disks following a power down, and in general, it all came back cleanly once the air-con. was fixed. Took a lot longer to bring up than it did to shut down, though.

    1. Anonymous Coward
      Anonymous Coward

      Never seen the managers, project managers and PMAs do work!

      FTFY!

      1. Sir Runcible Spoon
        Devil

        Re: Never seen the managers, project managers and PMAs do work!

        Rapid power down of 100+ servers -> Breaker Switch >:)

    2. Alan Brown Silver badge

      Re: Desperate times demand desperate measures

      "By the time someone followed up the alerts, the ambient temperature was heading towards 40oC"

      Our rooms have an emergency power crowbar at 38C for _exactly_ this reason.

  9. Stoneshop
    Facepalm

    Additional cooling

    One contract I had was with a branch of the Dutch Ministry of Agriculture, housed in (then) 40-year old barracks. Single story, low tarpaper roof, and even on just moderately sunny days the offices were already uncomfortably toasty. Cooling to the computer room was half-knackered, as one of the two units had a pinhole leak that apparently wasn't fixable, which had caused them to use up their freon-based coolant quota already. This lack of cooling caused serious swings in the computer room temperature, with a resultant well-higher than average component failure rate.

    Their method of trying to keep the temperature within acceptable values was by installing garden sprinklers underneath the heat exchangers, and turning them on when temps in the computer room hit 25 degrees. Time to turn the tap off was often well into the evening, requiring one of the sysadmins to clock several hours of overtime. On really warm days boosting the cooling capacity like this was insufficient, and relief had to be brought by opening the back door and pointing half a dozen floor-standing fans at it.

    (My suggestion to dump a few buckets of white paint on the roof, or tack a couple of rolls of alu-coated foil over it was dismissed, as 'this is just a temporary building'. It had been temporary since the mid-1950's, and it continued to be temporary for at least five more years after I left)

    1. This post has been deleted by its author

    2. H in The Hague

      Re: Additional cooling

      "My suggestion to dump a few buckets of white paint on the roof, or tack a couple of rolls of alu-coated foil ..."

      The top floor of H Mansion has a flat roof with black bitumen roofing felt. That floor used to get uncomfortably hot but I assumed that most of the heat came in through the windows and not through the roof as that's insulated. But applied special aluminium bitumen paint to it last week and it's made a real difference. Unfortunately I didn't measure the before and after temperatures.

      Tip: paint the roof before the weather gets too hot. In hot weather the primer, even when dry, is very sticky and that makes work difficult.

  10. Peter2 Silver badge

    . . .

    Ok. Racks get hot because the equipment in them draws cold air from the front, and then blows hot air out of the back. As wallmount cabling cabinets are poorly designed for airflow the hot air in the back can escape in two ways, firstly through escaping out of the tiny holes drilled in the side which probably wouldn't let enough air in to keep a hampster alive, and secondly by escaping out of the front in the gaps around the mounting rails. This hot air then mostly gets sucked back in the air intakes at the front of the equipment in the rack. The rack then gets hotter and hotter until the equipment generating the heat melts.

    We can all agree that's the problem? Cool.

    So why in the name of $DEITY do people blow cold air into the hot aisle they've created to cool the cabinet down? It's just pushing the hot air out the front, hopefully faster than it's being pulled back in again. A more elegant solution is to pull the air out of the hot aisle so that cool air is sucked in the front and hot air goes out of the back.

    A PC case fan is ideal for this, and most IT departments have at least one dead PC waiting to be properly disposed of that can part with it's case fan. Figuring out how to get 12v to it should be an exercise left to the imagination of the reader, but I shall comment that most supermarkets will sell 12v PSU's for a few quid. This combination provably lasts years running 24/7 as long as the H&S people don't see the spliced wiring connecting these two parts.

    There are of course other methods such as using transformers already owned by your organisation, such as running a long cable from the 12v rail on a nearby PC, buggering the dead PC's PSU so that it'll run without anything connected, sacrificing one of those 12v transformers in that box of surplus to requirement parts saved for a rainy day, or wiring 4 (12v) fans from the (48v) output from the PoE switch which is generating most of the heat in the cabinet for Cooling Over Ethernet, but a dedicated 12v supply is by far the best option.

    1. Sir Runcible Spoon

      In my experience cold air is usually pumped into the underfloor space, allowing the racks to vent warm air out the top and drawing the cooler air in from below.

      1. Peter2 Silver badge

        With a half dozen top mounted case fans in fullsize cabinets, yeah. Bog standard arrangement.

        But the article was showing pictures of those little wallmount comms cabinets which are generally mounted as close to the ceiling as the cabling contractor could get, which means you can't get access to the top. The article showed people having the same heat management problems with the comms cabinets as with larger cabinets and trying to control the temp by blowing ambient air into the hot space at the back with a normal deskfan.

        My point is simply that controlling tempretures in a small cabinet is best done in exactly the same way as with a larger cabinet; kick the hot air out with a case fan or two and cooler air gets sucked in the other opening.

  11. Chris Miller

    August 1990 broke all sorts of UK records for high temperatures. I was responsible for mainframe systems across two locations, both with well-specified (we thought) aircon units. We started getting over temperature alarms from the backup system, and when we checked we found that temperatures in the sun on the roof (the computer room was in the basement of a multistory office block) were pushing 50C. As a result the heat exchangers were actually heating the water coming from the machine room rather than cooling it.

    Fortunately this was on a Friday afternoon, and though we had a hectic weekend sourcing additional aircon (like hens' teeth, because of the high temps across the country), but the weather had broken by Monday. (We sunsequently replaced the 'dry' heat exchangers on the roof with 'wet' ones that use water evaporation to provide additional cooling.)

    1. The Real Tony Smith

      I've had similar and found that a hosepipe spraying a fine mist of water over the heat exchangers worked well.

      That and removing all the debris from the (horizontally mounted) cooling grids!

  12. spold Silver badge
    Facepalm

    Just blow on it

    Once reviewed a Health Testing laboratory IT room. While the main servers had backup generators for power failures the Air Conditioning units didn't. Ah no problem they said - we have some big household fans we can bring in.... OK so it's a power failure... where are you intending to plug them in?

    Not the largest problem really... the fridges for the medical testing specimens had no backup power so things would get rather smelly rather quickly. And what was the point of the IT kit running when they had nothing to test...

    1. Ozzard

      Re: Just blow on it

      Yeah, our -80C freezers in the lab are on the UPS and, for bonus points, have big CO2 cylinders behind that they can vent over several hours to keep cool.

      Of course, at that point the entire floor's full of carbon dioxide and we don't have breathing equipment...

  13. SirDigalot

    cooling is for wimps!

    I have an 8 port POE cisco pro-sumer gigabit prosumer switch acting as my core switch in my (unconditioned) loft/attic here is toasty old Florida, solid as a rock, the temps up there get well over 100 F every day, been there untouched for a couple of years now, i know it is still working because if it stops i get text alerts at work from the W.E.W.S (Wife Early Warning System ) and also more alerts from the U.K.W.N.I.S (Upset Kid With No Internet System)

    1. Korev Silver badge

      Re: cooling is for wimps!

      I used to work somewhere with a similar system called Clive. No matter what we did to enhance alerting we still weren't as fast as him.

  14. Kenny_10_Bellys

    Aircon fail

    I was called out last Sunday evening when our refinery monitoring systems went dark. The links between the monitoring network and normal data network turn out to rely on 18 year old Cisco kit that had overheated when the aircon in a server room failed. Cue the open windows and as many stand up fans as I could find. What fun!

  15. saxicola

    The thing with fans is that they just move air about, (and warm it whilst doing so) they don't cool anything. They work on humans because we sweat and thus cool by evaporation of the sweat, moving air helps this process. AFAIK servers don't sweat.

    1. Paul Shirley

      True but hot air tends not to leave enclosed spaces very fast without being nudged and cool air prefers to kip on the floor instead of working :)

  16. Mike Lewis

    You can use picture hooks for cooling

    https://i.imgur.com/vo2zlBm.jpg

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like