Sign in / up

back to article Are you sure there are servers in this cold, dark basement?

Welcome to On-call, our getting-a-bit-more-regular look at the odd things readers experience when called out to do things at night. This week, reader Pauly tells us that “a couple of Mondays ago I was asked to go and install some temperature-monitoring equipment at a prestigious hotel in Knightsbridge”. That's the posh bit of …

COMMENTS

Post your comment
House rules Send corrections
Add to 'My topics' unstar *

Page:

  1. The Mighty Spang

    first job

    partially entailed doing ops on a room full of Vaxen. 780, 2x785, 8600, some microvax, lots of drives.

    one day when changing a tape over thought 'feels a bit warm behind the 8600' so go and tell the ops manager. we go in, agree it feels a bit warm. go out and look at the aircon control panel. in those days it wasnt a nice lcd panel, it was a grey metal box with panel lights and a power switch. None of the 'unit failed' lights were on. there were 3 units running the area.

    still we are coming out of winter and again a week later i'm thinking 'this is too hot, can't be right'. tell the ops manager, we do the same thing again.

    then a fortnight later it was early may and the first lovely warm day of the year. I'm in a diferent building and cant seem to log on to anything in the cluster. rush over to the computer room to see all the loading doors open and people wafting air in. The aircon had failed. the console printers were going mad. as a vaxcluster everything major that happened came out on all the printers. but the heat was affecting the connection and the machines were dropping out, causing all the machines to report on the status not just what it saw, but the status that the other machines in the clusters saw as well, so paper was just churning out.

    eventually they took the decision to just turn the power off as they couldnt regain control for an orderly shutdown. not good.

    when the aircon engineer turned up, it turned out due to a blown fuse in the control panel, the 'unit failed' lamps were not working. to the best of his knowlege he reckoned we were working on 2 units most of the winter, one failed a month ago (when i noticed it getting warmer) and the last one failed that day when it struggled in the heat.

    the aircon panel was re-wired so instead of 'unit failed' lamps we had 'unit running' lamps. If the lamp was off we had a problem - or a blown bulb!

    thing to be learned is if you've got a hunch there is a problem, don't trust your instrumentation.

    13 0 Reply
    1. Yet Another Anonymous coward Silver badge
      Reply Icon

      Re: first job

      You should have upgraded to the unit that had a "lamp failed"-failed lamp.

      7 0 Reply
      1. J P
        Reply Icon

        Re: first job

        Fail safe systems only fail when they fail to fail safe.

        There's an anecdote to the effect that the UK importer of De Tomaso cars employed a related tactic to deal with the reported overheating of the cars in UK traffic - he removed the bulbs from the dashboard warning lights, and owners stopped coming back to complain about the warning light.

        1 0 Reply
    2. Stoneshop
      Reply Icon
      Flame

      Re: first job

      One day I happened to be in the computer room that housed a similar-sized pile of systems: a VAXcluster, some standalone VAXes, MicroVAXes and a pair of PDP11's, all with their disks and tapes, when the aircon failed. The first thing I noticed was the computer room sound changing, with a low rumble now being absent. About three seconds later I realised that "FSCK! The aircon's gone out!", followed by a dash out the door to the sysadmin pen. Then back to the console room with four sysadmins in tow. The head honcho decided that "this, this and that machine HAVE to stay up, that comms rack too, shut down the rest, then power down all the disks except those, those and that other one. You (pointing to two colleagues) round up all the fans from the offices and bring them here. Keep the computer room doors open. GO!"

      We managed to shut the lot down before any of the overtemp safeties triggered.

      Given the multitude of RA81's and 82's present I expected at least a dozen casualties there, but over the next two weeks we only had three failed HDAs (70-18491-01), and one microprocessor module a couple of weeks later.

      4 0 Reply
  2. Stevie

    Bah!

    Best IT story on the interwebs this week. Would have been a surefire shirt-winner if submitted to the Sharktank.

    1 0 Reply
    1. Anonymous Coward
      Reply Icon
      Anonymous Coward

      Re: Bah!

      What on earth is Sharktank?

      0 0 Reply
      1. Tom Maddox Silver badge
        Reply Icon
        FAIL

        Re: Bah!

        "What on earth is Sharktank?"

        Hard to say. One day, perhaps someone will invent a simple way of looking up publicly-available information from the convenience of your desk or "tele-phone"; until then, just try asking around.

        11 0 Reply
  3. Borg.King

    Wanted: IT Manager, must have own servers and cat 5 crimp tool.

    200 Cat 5 connectors a bonus.

    0 0 Reply
    1. Yet Another Anonymous coward Silver badge
      Reply Icon

      Re: Wanted: IT Manager, must have own servers and cat 5 crimp tool.

      A real ad would read. "IT manager wanted, must have 5 years experience of fixing yellow Cat-5 cables on a Dell switch in an HP rack."

      2 0 Reply
      1. Anonymous Coward
        Reply Icon
        Anonymous Coward

        Re: Wanted: IT Manager, must have own servers and cat 5 crimp tool.

        ..."despite everything else being Cisco. And don't ask why there's still thicknet in the walls."

        0 0 Reply
    2. Pookietoo
      Reply Icon
      Headmaster

      Re: Wanted: IT Manager, must have own servers and cat 5 crimp tool.

      RJ45 crimp tool.

      0 0 Reply
  4. Jock in a Frock

    I work for a telco as a Field Engineer. I got called to a shared office block in Bristol to investigate why our rack in the basement was off the air. The basement had 2 feet of shit floating around, and it had taken out our power and battery backup.

    The customer wanted me to hose it down and re-power it within 2 hours!

    0 0 Reply
    1. Destroy All Monsters Silver badge
      Reply Icon
      Trollface

      "I could do in 1 hour. But for that, I need to short the circuit breakers first!"

      0 0 Reply
  5. Anonymous Coward
    Anonymous Coward

    Idiots

    Steal hard drives, others pull the disks, tap them on the floor and plug them back in.

    Some where (higher ed) I worked really pissed me off, before I quit I installed a hidden web server with jokes about the management, waste of resources/fraud etc and let everyone know the IP, it amused the in-mates including new staff who were introduced to the pages. That's the worst I would ever do.

    2 0 Reply
    1. phuzz Silver badge
      Reply Icon

      Re: Idiots

      It'll take more than a tap on the floor, I've dropped harddrives from above my head onto a wooden floor and they were fine.

      Well, I managed to install the windows image and then they went out in customer's machines, so fine for at least 30 mins....

      1 0 Reply
      1. Anonymous Coward
        Reply Icon
        Anonymous Coward

        Depends

        As I understood it, 3.5" platters are made of metal but 2.5" platters are made of glass.

        0 0 Reply
  6. Vinyl-Junkie
    Holmes

    The game's afoot....

    LArge amounts of kit and the IT manager both go missing twice in a year? Perhaps they should be combing the woods or dredging the Thames - has anybody actually SEEN either of the IT managers since their alleged "resignation"?

    1 0 Reply
  7. adamsharif

    A few years ago I was working at an SME in Shoreditch, which was, back then, an area notorious for its poor power supply and frequent blackouts...

    We had just created a brand new infrastructure, and apart from mail (hosted by Google Apps), everything was hosted on-site in the same office as where the employees were working.

    One day I had a call - everything was offline! So I rushed in, and the situation was very odd...

    Seemed that there were two power feeds coming into the building, and the consequence of this was that there was power to the comms room (thankfully), however our actual office had no electricity whatsoever (was on a faulty EDF feed), and according to the building management there was no way of routing the other feed into our office (it was a managed office). Subsequently, we (~40 staff) had to set up camp in a shared meeting room which did have power, since EDF told us that the issue wouldn't be resolved for 24-48 hours! The issues here were: we didn't have Wi-Fi, there were only a few ethernet ports in the meeting room, there were about 8 mains sockets AND... Only around 10-15 staff had laptops!

    All we could do was patch our core switch down to the meeting room, daisy-chain a ton of extension cables, and then also daisy-chain a few 4/8-port switches!

    That was a really grim two days, although we were SO lucky that the servers still had power.

    1 0 Reply
    1. x 7
      Reply Icon

      I once had the interesting sight of a building with two feeds....that were on different phases.

      Somehow one PC ended up with the monitor on one phase and the base unit on the other.

      Ended up with 400 volts going down the VGA lead, with PC and monitor both becoming "live".

      Bang! Imploded monitor, and burnt out motherboard and lots of other damage,including several sets of brown underwear

      Rather urgent survey of the rest of the site was carried out and several other buildings were found to have the same problem. Once we'd cured the problem and put each building on a single phase we ended up with around 15% reduction in electricity used

      0 0 Reply
  8. imanidiot Silver badge
    IT Angle

    Pleasant hop across the pond on the clients dime

    I've recently had the pleasure of making a trip to the US on the clients dime because THEIR global support team was having troubles with the submodule we make for them. By the time we actually got their (2 hour car journey, 9 hour plane ride and another 4 hours by car again) the team had decided waiting for us was going to take too long (What with the customer breathing down their neck and 5 to 6 figure losses for each day of standing around) and that maybe they should just use those service instructions they DID have. When we actually got there all the physical swapping work had been done, leaving me to just do the recovery and some basic testing and qualification. I spent the next few days in some safetymeetings (about those qualifications I or anyone else wasn't strictly licensed to do in the US and what would be needed to BE qualified), doing some sightseeing, giving some training on recovery techniques and going to a gunrange. Best trip I've had in a long time, and it didn't cost me a dime! (Except for the gunrange, that got a bit expensive... but FUN!)

    1 0 Reply

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

Forgotten password ?

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like