back to article Sysadmin held a rack of servers off the ground for 15 mins, crashed ISP when he put them down

Welcome to the ninth edition of Who, me? In case you’ve come late to the party, it’s The Register’s Monday column featuring readers’ tales of stuffing things up. This week, meet “Adam”, who told us “I worked for a regional internet service provider back in the mid 90s. The internet was just becoming a ‘big thing’ and we …

  1. AndyS

    tic-tacs have zero calories.

    Nice. So, if a company is 'clever' enough, they could define a minimum outage period as that which is longer than the time to contact the help-desk and trouble-shoot the issue. They can then have as many outages as they like in a year, and still keep their official stat as 100% up-time.

    Of course, only a cynic would suggest that out-sourcing the call centre to India, lowering the number of agents, and randomly dropping calls would lead to an increased allowable time...

    1. Anonymous Coward
      Anonymous Coward

      Re: tic-tacs have zero calories.

      You clearly don't work with enough outsourcers. This is completely normal unless you have really good measures in place.

      1. Anonymous Coward
        Anonymous Coward

        Re: tic-tacs have zero calories.

        My employer outsourced some routine, but very important, IT business to a company in India. They have a habit of asking the requestor if it's ok to change the required date of a request whenever that date gets close. Most people go ahead and agree without thinking. So their SLA metrics show that they routinely do everything on-time - even when it's a "do within the next week" item that took 3 weeks to do.

        (AC to protect the EXTREMELY guilty.)

  2. Anonymous Coward
    Anonymous Coward

    I prefer "have you tried re-configuring the primary power coupling?"

    1. Anonymous Custard
      Trollface

      Or check if the pins on the power plug are clean...

      1. Anonymous Coward
        Trollface

        Or : have you tried turning it off and on again ?

      2. big_D Silver badge
        Mushroom

        I remember a BOFH from the 80s, not enough salty deposits on the power cord connector, remove it from the terminal and stick it in your mouth, the electrolytic salts in your saliva will re-balance the system. Krrzzztt!

        Don't try that at home kids!

        1. The Oncoming Scorn Silver badge
          Boffin

          Power....

          Electronics tutor when I was doing my C&G224, told the story of a dead TV he went to service in the home, the fault was traced to the power socket which appeared to be dead, but was corroded inside the plug terminals & 240V was proved to be present as his AVO\DVM probes pierced through the crud & made contact with the metal.

  3. Lee D Silver badge

    I once found myself holding up a small cabinet.

    Working in a school, was up a stepladder in an disused store room that held one of the network cabinets (it was about 10-12U but fully occupied). Right above one of the servers, which became quite an issue as I later realised.

    Without doing anything stupid (I never pulled myself up by, or leaned on, any cabinet that was wall-mounted) I was standing on the stepladder and just reached into the cabinet to retrieve a stray power cable and then realised that, actually, the cabinet was moving with my arm. And that when I pulled back it came with me. Not at all sure that I could hold the full weight, especially at arm's length, up a stepladder, I just pushed it back against the wall. Because of the design, I could literally put my whole arm through it and touch the wall it was dangling precariously from, so I braced myself against the cab with a hand flat against the wall behind, holding the weight on my forearm and elbow. Underneath, you see, was quite an important server that probably wouldn't take kindly to a sharp metal edge hitting it with severe force.

    After a few minutes my polite enquiries turned into a bit of a yell until my boss (non-IT) came running.

    We didn't really fancy having to rewire the entire cabinet - with some fairly important fibres - either, and I wasn't convinced there was enough slack for it to be pulled out and put on the floor in front of the server to do so without disconnecting something. The only solution, therefore, because of the height and access was to continue to press it while he went to get help, returning with several site staff who helped take the weight.

    There was a series of hurriedly battening-down to create a makeshift shelf and then my arm was finally released about 20-30 minutes later, slightly sore. It wasn't fun, and the box weighed enough that it was a two-man lift and one-man fit when it came to re-fixing it (to a very much better battened wall).

    What got me was the people who walked past in the meantime, smiling and saying "You alright up there?"

    1. John G Imrie

      Working in a school, was up a stepladder in an disused store room that held one of the network cabinets

      What happened to the sign saying 'beware of the leopard'?

      1. Anonymous Custard
        Trollface

        It was in the filing cabinet just behind the planning applications...

    2. fnusnu

      What got me was the people who walked past in the meantime, smiling and saying "You alright up there?"

      Got to love British humour :)

    3. Chris King

      I once found myself up a stepladder in a disabled toilet, because some bright spark had decided that the best place to put the cabinet was just above the door, on the inside of said toilet.

      Unfortunately, I had forgotten to lock the door, and someone in a wheelchair flung in open.

      I grabbed the cabinet to steady myself, and fortunately the steps stayed in place- but I did have visions of them falling to one side, and me holding on to the cab for dear life like a modern-day Frank Spencer.

      Bettyyyyyyyyyyyyyyyyyyyyy !!!!

    4. Anonymous Coward
      Anonymous Coward

      Reminds me of the.....

      ....wall mount rack that was put above a desk a user sat at. We never thought it was a good idea but were ignored. Lucky for them the user heard a noise and moved. A min later that same rack came crashing down where he'd been sitting and left a big hole in the desk. It would of killed him if he hadn't moved. What annoys me the most is said company got away with this instead of being pulled over the coals by the HSE. From what I heard, they kept it quiet and never told the HSE about the incident.

    5. The Oncoming Scorn Silver badge
      FAIL

      W**F Medical - *elus

      I have found a number of dodgy installs inside medical offices in Canada, total lack of getting to them safely (Why do we need a stepladder it's your equipment? (It wasn't) including racks mounted:

      9 foot off the floor.

      Over lavatories

      I had small comms equipment fall & hit my head in one location as it all fell as one cable was tugged & in one extreme case, everything including a small UPS on a shelf that was balanced at one end on the top of the door frame & a tiny 1"bracket at the other.

      1. Montreal Sean

        Re: W**F Medical - *elus

        I worked on a server that had the switches and patch panel installed in a wall mount rack above the server rack.

        Said wall mount rack was very loosely mounted to the wall with a couple of small screws, with most of the support coming from the cat5 cables they had tied around a water supply pipe...

  4. highdiver_2000

    How did you slide the equipment into the new racks without losing power or connection? Don't tell me you move one power cord at a time?

    1. John Robson Silver badge

      You assemble the new rack around the servers...

      1. The Oncoming Scorn Silver badge
        Stop

        With a rats nest of wiring in the front in goverment offices, I ended up had to slide the old switch out & the new switch in from the rear of a very small patching closet.

        Doing it out of hours wasn't a option, neither was tidying up the cables.

        Cost me a bit of skin too fighting it in too as I recall.

    2. Paul Westerman
      Trollface

      Hello to Jason Isaacs

      You just slide the equipment into the new racks

    3. A K Stiles

      build around them or open front and back and slide it in whilst 2x4 support staff steps though the cabinet?

  5. Edwin

    Trumpet Winsock on Win95?

    I remember using it on Windows 3.1, but thought Win95 had its own sockets?

    1. BinkyTheMagicPaperclip Silver badge

      Re: Trumpet Winsock on Win95?

      It does - it shipped with TAPI that considerably simplified things (provided the modem inf file was correct, which it wasn't in all cases, especially if you wanted to do anything other than connect to the Internet).

      What I didn't know is that there was also a TAPI release for Windows 3.1. No idea how well it worked..

      1. Alan Brown Silver badge

        Re: Trumpet Winsock on Win95?

        What I remember from the bad old days was how every ISP wanted customers to manually configure their settings and provided CDroms or floppy installers which did this without telling the poor user what had happened - effectively nobbling their connections if they were jumping around between ISPs.

        Which led to "interesting" fun'n'games with queries to or from other outfits' DNS servers. I took the simple option of redirecting locally-originated queries using competitor ISP DNS to the local servers (for speed) and blocking remote queries after I found an entire Australian ISP using my servers for DNS to the tune of about 1/4 of my total bandwidth at the time.

        Even Trumpet Winsock understood PPP server-provided options and if you told users to "just leave it alone" things usually worked better than when anyone attempted configuration.

  6. Anonymous Coward
    Anonymous Coward

    Customer ordered a mainframe upgrade - in the days when that consisted of several 6' (1.8m) high heavy cabinets. The installation was scheduled for a weekend with little margin for delay.

    There was a known quirk - the computer room was on the second (US third) floor and the lifts were too small to take the equipment. Therefore when the machine was originally installed - a large doorway had been cut in the outer wall on that floor - with a projecting hoist mechanism. The door was tested to make sure it still opened.

    Come the critical day the delivery truck arrived. Then it was discovered that a bicycle storage shed had been built against the wall - immediately below the hoist.

    1. Anonymous Coward
      Anonymous Coward

      Early 1980s we had our first VAX 11/780 delivered. We were in an ordinary office building without a goods lift, so there was much headscratching about how to get an 800kg system up to the 2nd floor.

      Someone eventually realised that if the system was uncrated in the lobby it would just fit the passenger lift, with about 1cm clearance each side. It was substantially over the lift's allowed weight, but it was assumed there was a x2 safety factor, so would be OK for a one-off.

      The system was uncrated and carefully slid into the lift, which it filled completely. The "2" button was pushed, and as the doors closed the delivery crew sprinted up the stairs to wait for it. The lift pinged, the doors opened, and they realised that the extra weight meant that the lift had stopped with the floor about 2cm below the building floor. Several hours later some very hot and bothered people finally managed to lift the front of the 800kg system up those 2cm, and hauled it out onto the landing.

      One of the DEC guys present then regaled us with a story about how a similar delivery had had to be made via a window, there was just no internal option. The window & frame had been removed, the system placed in a sling, and a portable crane used to winch it up. It got to 3rd-floor height when the sling slipped... No-one hurt, but after impact the system was only 2/3 the height it had been, as it was taken sadly away back to the factory. Maybe our delivery could have been worse...

      1. Anonymous Coward
        Anonymous Coward

        A heavy 6' (1.8m) tall cabinet arrived after being shipped to a distant land. It had taken some time for the upgrade order to be fulfilled - so the customer was eager for it to be commissioned.

        On opening the door the engineers discovered that the internal rack was no longer a neat rectangle - it was now a diamond rhombus. A mark in the paint on one side showed where a sideways fall had been stopped by some obstruction. The internal rack frame had continued to fall with its own momentum until the top corner touched the case - the bottom of the frame being securely anchored to the case's base.

        It was several months before a replacement arrived.

        1. CrazyOldCatMan Silver badge

          that the internal rack was no longer a neat rectangle - it was now a diamond rhombus

          I had pretty much the same with a Cisco Catalyst 5509 rack (shipped from the US as a spare as we were doing a major building move).

          As soon as we saw the pallette we knew something was wrong and so held off opening the crate until someone found a camera to take lots of pictures.

          I don't even want to think of how much force was required to bend the chassis of one of those. Most of the cards were U/S too - snapped in a neat line down both sides.

          That was an expensive day for that courier. With the pictures we had taken (including the original pallette) they couldn't really argue. And a fully-loaded 5509 wasn't a cheap beastie.

          1. Chris King

            "I had pretty much the same with a Cisco Catalyst 5509 rack (shipped from the US as a spare as we were doing a major building move)".

            The Cat 5000's were built like tanks. I herded a 5500 in my last job (almost fully populated apart from the dedicated LightStream 1010 slot), and had homicidal dreams of dropping it from a great height on to our remaining 3Com kit. (If you've ever had to manage NetBulider II's, you'll understand)

            Sadly, this never came to pass. I'd never get a trade-in on the 3Com kit if I smashed it flat.

            1. Long John Brass

              3Com kitsmashed flat.

              If you've ever had to manage NetBulider II's, you'll understand

              I think I remember those bloody things; Am I right in thinking they were configured via 3.5" floppy and every once and a while they would spontaneously reboot and wipe the floppy disk?

              1. Chris King

                Re: 3Com kitsmashed flat.

                "I think I remember those bloody things; Am I right in thinking they were configured via 3.5" floppy and every once and a while they would spontaneously reboot and wipe the floppy disk?"

                Yep, that's the bast^H^Hdger.

                The "core" of that network was a Linkbulider MSH when I arrived. I thought the thing was a total piece of junk, but I got decent trade-in on it a couple of years later because the NHS couldn't get enough ot them.

          2. Anonymous Coward
            Anonymous Coward

            Shipping expensive stuff

            Pro tip from the in-circuit tester world: when shipping bulky but expensive cargo like that, specify that it goes by "air-ride van only" (US terms, not sure what the UK terminology would be). I asked about why they did that, and was told that the Air Ride guys had slightly less crazy forklift jockies. The actual ride on the truck didn't matter, but the freight forwarders that qualified were a bit more gentile with the cargo.

            Oh, the other item is if you can get a dedicated LTL ride somewhere versus using a freight forwarder. Less trips on and off of the truck means fewer chances for disaster.

        2. Alan Brown Silver badge

          "On opening the door the engineers discovered that the internal rack was no longer a neat rectangle - it was now a diamond rhombus. "

          On a similar shipping note, a newspaper in a town I lived in acquired a "new" (secondhand) computer markup and typesetting system from another country for a "not insubstantial" sum of money. This was shipped internationally in a 20 foot container and then for reasons only known to the shipping company, unpacked on the dock and loaded into the back of a large open topped truck for the last 400 miles.

          Apart from the issues of things moving around during the trip on the twisty roads in that part of the world (did someone mention sliding across the deck?), the icing on the cake was that the truck had so much extra space compared to the container that the company packed out the load by adding extra cargo headed in the same direction - large bags of superphosphate fertilizer.

          By the time the shipment arrived, the system was not only battered and bent, but there was a fine coating of corrosive powder in every single part of the entire system. The entire thing had to be written off at a seven figure cost.

          The kicker is that the delivery contract specified container service door to door (to ensure things arrived in good shape, everything was well packed inside the container), so legal claims went in against the haulier - who promptly went out of business and then phoenixed - meantime the seller demanded payment for what was a working system when it left their premises. It took several years to resolve.

      2. Stoneshop

        System delivery

        The window & frame had been removed, the system placed in a sling, and a portable crane used to winch it up. It got to 3rd-floor height when the sling slipped... No-one hurt, but after impact the system was only 2/3 the height it had been, as it was taken sadly away back to the factory.

        One customer I was assigned to as a contractor had ordered, shortly before I came in, two racks, one with an Alphaserver ES40, one with storage. When it was delivered, the delivery driver found that he had slightly misaligned his truck with the loading dock. So he decided to move away from the dock, then back up correctly. However, he had already undone the straps securing the cargo to the truck, and anyone with a modicum of knowledge of Newton's Laws would realise that an object at rest will remain at rest until acted upon by an external force. And indeed, the racks remained at rest while the truck moved forward underneath them, until the force of gravity got its chance when the tailgate cleared the racks.

        Two parallelogram-shaped racks went back to DEC, and a struggle ensued when it turned out that transport insurance, as arranged by some purchase department of the Ministry of Agriculture, was only good for Dfl.500, maybe one percent of the actual value.

        1. Bowlers

          Re: System delivery

          Similar story. Mainframe site on first floor long used to delivering large bits of kit through removable window. This time it was an EMC 5500 full of 5 1/4 drives, batteries, PSs etc and it was a heavy bugger (in both imperial and SI). The box was to be lifted, inside a cage, by a crane. Large man in the cage ready to push box out when it reached the window opening. Unfortunately this time the cage was not quite high enough and when the large man pushed the box caught the lip of the window and the cage moved backwards, box fell out of the cage ...oops and was effectively destroyed. Later it was revealed the box was insured by weight not value...oops2. (Disclaimer: I was not the large man)

      3. Stoneshop

        One of the DEC guys present then regaled us with a story about how a similar delivery had had to be made via a window, there was just no internal option.

        One I heard from a colleague in the UK was when an 11/780 was to be moved from the lobby where it had been delivered to a few floors up via the staircase. The system was put on stairwalkers, and a lot of muscle was gathered to power the move. Which went reasonably well until between the second and third floor someone lost his footing and the system crashed. Back down that flight of stairs and through the glass facade of the stairwell. And one couldn't quite say that the director's car cushioned its impact with terra firma very much.

        1. Chris King

          "And one couldn't quite say that the director's car cushioned its impact with terra firma very much".

          I once got to dispose of a dead MicroVAX I (one of the desk-side pedestal jobs) by helping a colleague heave it out of a first floor window into a skip (This was well before RoHS, let alone WEEE).

          The MD's Jag was parked next to the skip, and my partner-in-crime helping me teach the MicroVAX to fly [1] asked "Do you think he'd give it a lift back to head office if we could drop it on to the back seat ?"

          "Nah. We can't get the right angle from here - front passenger seat, maybe".

          [1] Make it fly? That's easy - even pigs fly, given sufficient thrust - but we knew it would never nail the landing.

      4. StargateSg7

        I remember those giant 780 cabinets from the mid-to-late 1980's in Calgary, Canada's Oil Patch and having to do a backups where the TAPE storage cabinet was MUCH heavier that the VAX computer.

        Our SysAdmins were smart! They put 2" thick x 12" wide boards in a cross-hatch pattern as the supports for the raised floor with holes drilled in for all comms cables and electrical. Weight was not an issue since we estimated the underflooring could take over 30,000 lbs!

        What was an issue was when a LARGER server-system eventually went in and the ENTIRE FRONT OFFICE window front had to be taken out to fit it in! It was lighter in weight but physically bulkier than the VAX 780 and could NOT be taken apart. That was another $15,000 later in Glazier costs or about $35,000 in today's dollars AND it had to be done over a three day holiday weekend otherwise any computer downtime would have cost us a lot more than $15K!

        1. Anonymous Coward
          Anonymous Coward

          The computer room was originally part of the factory floor - thus it resembled a high ceiling hangar. The mainframe had been assembled there - and then the area was partitioned off from the factory.

          An office suite was built on the outside wall for the operations staff - whose human sized entry was now the only way in to the computer room.

          In 1970 a 600MB hard disk was a big brute weighing about 1.5 tonnes. The delivery team managed to squeeze it through the office suite doors into the large computer room - and breathed a sigh of relief. They wheeled it down the centre aisle - but as they aimed for its side destination the false floor started to collapse. The centre aisle was not a false floor - it had a concrete strip underneath it.

          Fortunately they managed to haul it back until the floor could be strengthened. Any further and they would have had to take the roof off the building to give a crane access.

      5. Niall Mac Caughey

        Many (OK, many, many, many) years ago I worked with an ex-Post & Telegraphs engineer. One of his favourite stories was about a newly-extended telephone exchange in a rapidly-expanding suburb south of Dublin. It was a large, shiny new building (late 1960s) with solid concrete floors designed to take plenty of weight; which was a good thing as they had manoeuvred a considerable number of substantial racks filled with crossbar switches (I think) into place on a Friday afternoon.

        It was all very neatly done and they headed home pleased with themselves, prior to returning on the Monday to begin the cabling. Unfortunately some bright spark had specified this new-fangled stuff called HAC (high-alumina cement). People of a certain age may remember this.

        When they arrived on the Monday morning they found their shiny new equipment a little dusty and worse for wear, as it was now in the basement.

    2. Anonymous Coward
      Anonymous Coward

      Bike sheds

      See? This is why executives spend so much time on bike sheds! If they are tending to high level strategy, executive oversight, and long term business planning then things like this get missed.

  7. Anonymous Coward
    Anonymous Coward

    Back in the late 90s and working in the main IT building for one of the UKs largest councils...

    Every day, once or twice a day one of the Coax network segments would disconnect along with all the users. Then a few minutes later it would come back up again. Various troubleshooting techniques were tried and the cables were all tested fine, but it would still continue to lose connection. The fault was eventually traced to one of the junior network guys who while stretching his legs rolled his feet along the floor and stretched/twisted the cable just enough to terminate it under his desk. When he brought his feet back he put it back into the original working position and everything was fine again...

    1. Anonymous Coward
      Anonymous Coward

      A naval air station had one terminal that had intermittent network problems. It was finally noticed that there was a correlation with a Harrier jump jet landing or taking off from a nearby concrete pad. The 10base2 wall socket had an internal loose connection.

      1. Anonymous Coward
        Anonymous Coward

        Coax woes

        When your large 4 buildings on separate segments per floor, cox network looses on floor intermittently

        And all your TDR troubleshooting fails to show any issues.

        Your standing out side one office when the noise hits you of users protesting the networks down, followed by the cries of its back again, when you realize the lady in the office had just opened and shut a filing cabinet drawer.

        Check the cable and yes someone had put the the back right edge cabinet on top of the cable,

        and due to some wonder of physics when any of the bottom 3 drawers were moved nothing happened but when the top one was pulled out the shift in weight caused a break of the cable, when it was put back in the cable worked again.

        A quick new cable run tacked to the wall securely and the problems was solved.

    2. Anonymous South African Coward Bronze badge

      And the game of Hunt-the-Terminator on unknown (somebody else's install and mess) coax networks.

      1. Mike Timbers

        had an intermittent problem with a client's coax ethernet network that was eventually traced to one of the segments had a terminator that was for Arcnet* so was 75ohm instead of 50ohm.

        *anyone else remember that?

        1. Phil O'Sophical Silver badge

          *anyone else remember that?

          Oh yes, although I think ARCnet was 93Ω, nothing as "normal" as 75. I spent weeks trying to track down an intermittent problem where a cluster would grind to a halt under heavy network load, eventually finding 5m of ARCnet coax (in the underfloor trunking) that had been used to extend the ethernet network to a workstation across the room. It was really bizarre because I don't think we'd ever had any ARCnet in that office!

          Then we had the sysadmin who "discovered" that she could add more systems to the network in an office by using "Y" phone adapters plugged into the RJ45 ethernet sockets, instead of having to install more 10baseT and extra hubs. It took me a while with a whiteboard to explain how ethernet collision detection worked (or didn't, in that case). It's amazing that it seemed to work at all, I guess that TCP was just too good at cleaning up the mess.

        2. John F***ing Stepp

          My very first mess with that. The supplyer gave us a combo of various cable. One week of testing before anyone looked.

          (all just co-ax, right?)

          "The air likes you," I told the supplyer, "the ground likes you but if I throw you of the top of this building the impedance missmatch between the air and the ground might cause you some problems."

          He actually thought I was making a joke.

        3. Martin Taylor 1

          'Cos coax comes up in Somerset...

          Can't remember the precise resistances involved, but I was called to a site in Somerset where they were experiencing Ethernet problems. On lifting the floor tiles I was confronted with a mess of cabling of 3 different varieties, only one of which was 10Base2.

        4. Long John Brass

          terminator that was for Arcnet*

          @Mike Timbers - Ahhh ARCnet, how I miss thee. No for realz; ARCnet was pretty cool in it's time!

  8. phuzz Silver badge
    Unhappy

    Three finger salute

    Have I ever told customers to reboot, knowing that the problem was actually at my end, but that it would be fixed by the time the clients had rebooted?

    No of course not, where did you get that idea, who have you been talking to?! I DENY EVERYTHING.

    1. imanidiot Silver badge
      Trollface

      Re: Three finger salute

      If the problem is gone after a reboot the problem must be at the customers end right? -->

    2. Captain Scarlet

      Re: Three finger salute

      To be fair early ADSL routers often required a reboot, they would not bother to try and login again unless instructed and it was easier to reboot said devices.

      I would be honest and say if I knew what the fault was (As many people who called knew more than I did in my early days on an ISP support desk) and state it should be back. It was better than 56k where the users would then have to put the phone down as I could then get them to check the lights and if they had the knowledge login and check their logs.

      1. Stevie

        Re: the users would then have to put the phone down

        Wot, no VoiceView?

  9. TheSkunkyMonk

    its good to see that my lack of trust in tech call centers in well founded, thank you :P

    1. CrazyOldCatMan Silver badge

      its good to see that my lack of trust in tech call centers in well founded

      Anyone who has worked on a helldesk or in any sort of call centre knows exactly what goes on. That's why we tend to try to fix things ourselves..

  10. Shady

    Co-location

    Reminds me of a time about 15 years that our "nominal CTO" changed colo hosts to a cheaper provider, and we just had the servers shipped there. The colo wired them up, switched 'em on, and gave us a call to tell us they were running.

    One day we had cause to go on-site there (needed to install a new hard drive or some such), and we were, given the monthly cost, shocked but not surprised to find our servers supported by milk crates in a disused outdoor dunny at a semidetached house in Leicester, with cables running in through the air bricks that had been creatively enlarged (read - smashed through).

    Although we'd never had an issue with reliability or connectivity, the servers were moved pretty sharpish to a more reputable host.

  11. Anonymous Coward
    Anonymous Coward

    I wonder how common this was.

    I remember being visiting an ISP on the UK central south coast that fits that description perfectly.

    Desktop computer being used as servers stacked up and sometimes on selves in racks

    Bunches of US Robotics modems cable-tied together as point of presence

    All of which was in a back bedroom of a modest dormer bungalow.

    Even when they moved to dedicated (non-residential) premises they still used desktop computers in racks and small APC standalone PC UPSs and are still in those premises, although I suspect their infrastructure has improved along with their scale.

    1. Phil O'Sophical Silver badge

      The Computer History Museum in MountainView used to have (may still have, I don't know) one of the original Google racks on display. A quick search found these photos, I'm sure there are others: https://www.flickr.com/photos/sparker/173567346

      https://www.flickr.com/photos/nationalmuseumofamericanhistory/9623001486

    2. Mark 85

      I think that was pretty common back then. There was a large BBS* here in the States, maybe the largest at the time(?) that had something like 300 PC's (may have been more) stacked in a bedroom of the house the owners of the BBS lived in. Everything including modems were just packed and stacked from floor to ceiling with cabling across the floor, taped to the walls, etc. They even proudly had pictures available of their "state of the art server room" available for downloading.

      *I think it might have been Event Horizons but I'm not certain.

    3. Alan Brown Silver badge

      "I wonder how common this was."

      Very. Prior to Telcos taking over ISPs mostly grew out of hobbyist BBSes and to be frank it's easier to have a bunch of cheap, redundant desktop systems than a few expensive servers, especially when in the 1990s getting parts for said desktop systems was usually much faster (read: less than two hours) than for the servers (read: weeks)

      1. Anonymous Coward
        Anonymous Coward

        small ISPs

        Not to mention, "home" phone lines were *much* cheaper than "business" phone lines (at least in Baby Bell land). So you buy one business line for your 33.3K modem pool, then set up call forward-busy to a bunch of residential lines.

        Fragile as all hell, hammers your lead modem while the top few modems in the stack stay silent, one bum line can hang the whole system. PRIs were a godsend.

        Then you get into rural ISPs where it was most economical to have a point-of-presence every 20 miles or so (thanks to in-state long-distance calling). Put in a couple of PRIs to a modem bank (PM3s FTW!) and a router to a T1 backhauling to the main office. The world was a really strange place in the Wild West days of the Internet...

    4. Anonymous Coward
      Anonymous Coward

      hardly unusual

      C'mon...Hillary's email servers were in a bathroom, not that long ago. Standard practice, I think.

  12. Alistair
    Facepalm

    I recall the arrival of a pair (very early in the release process) of a pair of HP V2250s. Were *supposed* to; show up on Thursday, get racked and cabled on Friday, tested from Saturday to Wednesday, including the OS build and storage configuration, and go live the following weekend. Since they'd rolled off the assembly line on the Monday, they'd been flown up to us, fork lifted off the plane and into a 42' dual axle truck. I'll point out that the two of them (sometimes referred to as paint drying machines) were the *only* two objects put in the truck. Now, being 42" on a side and roughly 40" high, these are not small critters. Or light weight. Why in any deity's name the forklift driver or the truck driver or whomever decided to try and *STACK* them in the truck we will never know. But the large dip to below 3/4 line of the box in one of the corners, and the large dent in the bottom of the other made it clear we were in no way going to hit the Go Live on those two. The HP on-site engineer demonstrated quite an extensive vocabulary that afternoon.

  13. Stratman

    Many years ago

    in the days of analogue telly, we were covering a football match for MOTD as we did most weekends. At the time I was a vision engineer and during the match our main function was exposure and colour matching of the cameras. The cameras themselves were Philips LDK5s, which had a Camera Control Unit and power supply unit in the truck. The PSU was basically a big FO transformer (and some electronics in a separate narrow unit) and two fitted side by side in a ninteen inch rack. They had one control on the front panel, an illuminated push on push off switch which required a decent bit of force to operate. They were mounted just above floor level.

    Being Philips, there was a guard around the switch to prevent accidental operation. Being Philips they often bent or fell off. The engineer sitting next to me, let's call him Billy, was bored with the game and had a mid-half stretch which included his legs. We all heard an ominous 'click' as his size nine planted firmly on the guardless switch, and laughed as we knew what was coming next. The switches didn't change state until released, so Billy had to spend the remaining twenty minutes of the first half keeping the switch pressed in and therefore the camera operational.

    How we laughed.

  14. Anonymous Coward
    Anonymous Coward

    Microsoft Exchange of fluids

    During my sandwich degree some years ago, I was working on placement in an IT department at a very well known and quite large national mapping organisation in Southampton.

    While there, I started courting a colleague... I say courting, I mean nipping back to hers at lunchtime for a quickie. When time was short, on a couple of occasions we made use of the deserted and usefully-chilled server room - don't panic, she had her own access, I wouldn't breach security, what do you take me for?!

    Anyway, on one horny occasion we nipped in there, and after doing the deed, in the process of pulling my trousers back up, I stepped back onto an extension cable carelessly and unprofessionally draped between racks, tugging the plug out of the socket (for the second time that day, arf arf), and cutting the power to the entire rack, which most vitally included the Exchange server for all 4000+ users.

    Hearing the sound of banks of fans and drives spinning down, a wave of dread washed over me as I sensed something had gone horribly wrong. Given the compromised position we were both in, my now-wide eyes looked over at my bae silently communicating "oh fuck", and I proceeded to evacuate my face entirely of blood.

    Once the panic subsided, I quickly mashed the plug back into the socket, and we quickly but outwardly-calmly exited the room, straight past our support phone monkeys who, unbeknownst to them, were about to suffer an influx of incoming calls in 3....2....1.

    Anon, because to this day only me and her know about this. Oh, and you ugly lot. And maybe the CCTV guy. Actually, I should probably check Pornhub.....

  15. liquidsmoke66

    Lol-ing hard at this one.

    Great story :)

  16. ICPurvis47
    Boffin

    Diamond Trapezoid

    Not IT related, but when I was working as a development engineer for a very large electrical manufacturer in the Midlands, we had a rush job to refurbish two cabinets of circuit breakers for BR Southern Region. The final assembly was being done in a bay we called "The Elephant House" because of its high lift capability, by our two Elephant Trainers, Sid and Sam. Because of a delay in procuring some vital parts, the build was not completed until late on Friday afternoon, and Dispatch and Transport were waiting impatiently outside with the Commer TS3 (sounded lovely, ever heard one on full song?) low loader. Eventually, Sid and Sam completed buckling up the cabinets, which were about four feet square and eight feet high, and lifted them with the overhead crane onto the low loader. Without waiting for any strapping down, the driver (who was anxious to get home) drove off down the yard to D&T. Everything was fine until he started to back it into their loading bay, which involved negotiating a slight ramp up from ground level to the building floor level. As he was approaching at a 45° angle to the threshold, one side of the semitrailer rose and tipped both cabinets off the other side, where they crashed to the ground, destroying all of the circuit breakers inside and distorting the cabinet frames. They were eventually lifted back onto the low loader and returned to the Elephant House, but, needless to say, they weren't delivered that week (or the next, either).

    On a similar note, when I was an apprentice at the Trade School, I was one of only two apprentices that had been checked out on the fork lift and worksaver, so we had to do any heavy lifting and transporting around the school. One day we were tasked with lifting a large (about 500lbs) machine vice onto the worktable of a shaping machine. We completed this task but, as it was tea break, left bolting it down until we had returned the worksaver to its charging station and had our tea break. As we walked back into the machine shop there was an almighty crash. The apprentice who was working on that shaper was rather keen to get on, and had returned early from the canteen. He had placed his workpiece in the vice and knocked it up (ie set it level and tightened the vice), adjusted the tool height for the first cut, and as we walked in through the door, he pulled the lever and the tool hit the workpiece, sending it and the vice off the end of the table. Luckily the operating lever was at the side of the machine, otherwise it would probably have killed him. As it was, it made a very large hole right through the concrete floor, which took several days to get Site Services to repair. We were hauled over the coals for not bolting it down before we left it, but he shouldn't have been in the building on his own either, so we were just given a slap on the wrist each and told to be more careful next time. The lad was moved to a less dangerous machine, but left soon afterwards as he was not really suited to an engineering apprenticeship.

  17. J.G.Harston Silver badge

    Holding up the furniture

    I've had that as well.

    Early 1990s in Hong Kong. Company had moved into a new set of offices which had been wired with floor networking in conduits that had been concreted over. My first job was finding the outlet points that had been left and wiring in the socket plates.

    All of the findable sockets done, and continuity continued to insist there was at least another one somewhere. Eventually found one behind a cupboard - that was easy enough, and then the last one was underneath the display tables in the reception area.

    Crawling under the table I found a table leg over the corner of the outlet box. Ok, so I raised my shoulders slightly to reduce the weight of the table and punched the bottom of the leg to shift it a half a centimetre - at which point the whole leg fell out leaving me holding the entire setup on my shoulders.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like