back to article Are you sure there are servers in this cold, dark basement?

Welcome to On-call, our getting-a-bit-more-regular look at the odd things readers experience when called out to do things at night. This week, reader Pauly tells us that “a couple of Mondays ago I was asked to go and install some temperature-monitoring equipment at a prestigious hotel in Knightsbridge”. That's the posh bit of …

Page:

  1. Peter2 Silver badge

    How do you end up with two IT Managers stealing all of your equipment within a year?

    It sounds like this company has both severe systematic problems with recruitment, and also with how they are treating employees for that to happen to the same place twice within a year. I haven't encountered that sort of theft once during my career.

    1. Destroy All Monsters Silver badge
      Paris Hilton

      Even more, how does that company even survive? Do they have a special relationship with Big Money/The Queen/The Mob/YouGov?

    2. This post has been deleted by its author

      1. Anonymous Coward
        Anonymous Coward

        If I were the suspicious sort (spoiler: I am) I would wonder if it was really the IT Manager each time. Happenning twice at the same location does seem like the connection is something to do with the location. Perhaps it's not that they treat their IT Managers badly enough for them both to take identical revenge, but rather that someone else on the staff is stealing their stuff and blaming recently fired personnel.

        1. ecofeco Silver badge

          "but rather that someone else on the staff is stealing their stuff and blaming recently fired personnel."

          This as well. (see my latter post)

        2. Anonymous Coward
          Anonymous Coward

          If I were the suspicious sort (spoiler: I am)

          I think that suspicion and mild paranoia are positive traits working in IT.

          It doesn't fit a supplier taking equipment. They'd just take their servers and leave the cabling untouched.

          I remember having a joking discussion with some colleagues when we were about to be made redundant when the big recession hit as to how much damage we could do in 2 minutes to the IT infrastructure.

          1) Cutting the cables leading to the patch panel in two places. (maximum repair time for minimal time input)

          2) Cutting the BT fibre for the ISDN30.

          3) Taking the hotswap drives from the servers, exiting via a fire exit and depositing them in the nearby river. (as one person could fit the lot in a carrier bag)

          Our conclusion was that we could inflict near total damage to our infrastructure within a minute. They'd "know" it was us, but wouldn't have been able to prove it to a standard required in a court of law. We didn't do anything, needless to say though "AC because..."

          The point being, that taking the servers is slow, hard heavy lifting for a single person and not a massive blow to the business since a DR plan could recover to a new server as quickly as the equipment can be delivered and the tapes produced. Cutting patch leads is just petty but even a moderately large site could resolve problems in a few hours in the time it takes to build new servers with two people with radios, one at the board and one walking around calling back "port 1 is a computer, port 2 is a fax machine, port 3's a phone- the user says Ext 123"

          It doesn't make sense. The destruction job itself was unprofessionally pathetic. Unless he did the backup tapes as well, in which case I might be inclined to rate the job slightly higher.

          1. Paul Hovnanian Silver badge

            "It doesn't fit a supplier taking equipment. They'd just take their servers and leave the cabling untouched."

            Good point. And the supplier would just hide the old servers somewhere on site. Then, when they get the call to put in new units, they just reinstall the old ones, bill the customer for the new hardware and put it (still in the box) on eBay.

            Twice, by the looks of it.

          2. Alan Brown Silver badge

            "Unless he did the backup tapes as well"

            You'd be surprised (or not) at the number of outfits who either don't run backups or "backup to disk" (and those who do use backups invariably don't keep enough generations because it costs too much)

            A mounted disk can be scrubbed or removed in short order

            RAID != backup

            Replication != backup

            etc etc.

    3. Fink-Nottle

      > How do you end up with two IT Managers stealing all of your equipment within a year?

      Exactly! And these strange monthly bills they've run up.

      Err ... what is a data centre anyway?

    4. Doctor Syntax Silver badge

      "the same place"

      I'm not sure about this. It seems that the author was working for a service business or a leasing business. It was the 2nd time this had happened to that business. It doesn't say it happened at that site. It might have been but we're not told that.

    5. Peter Simpson 1
      WTF?

      "It sounds like this company has both severe systematic problems with recruitment, and also with how they are treating employees..."

      I'd say the complete lack of lighting in the "server room" was a good clue that they're running on a shoestring and cutting every possible expense.

      If I were the author, I'd make sure to get paid in full before any work was started.

      1. Fatman
        Joke

        Manglement strikes again.

        I'd say the complete lack of lighting in the "server room" was a good clue that they're running on a shoestring and cutting every possible expense.

        But, we have to increase shareholder value!!!

        Icon says it all.

    6. Anonymous Coward
      Anonymous Coward

      Oh, the 'call', from a financial company (legal loan sharks)

      'We need you here right now, our RAID has failed and nobody can log in'

      'OK, which server?'

      'We don't know, but can you come and look for us anyway, bring some disks'

      'OK, no problem, I'll look and see what you've got listed and bring a few but if it's died then it might not just be a disk'

      So, on arrival, I find eleven HP DL380 servers and a pair of MSA1000 storage arrays, all flashing merrily with every amber light.

      And not a single hard drive in any of them.

      And no backup tapes, or tape drives.

      Somewhere in the region of 80 72GB Ultra 320 SCSI disks.

      Turns out the 'IT' manager had got a bit miffed at being sacked and had stolen the lot.

      From three different sites.

      Or the other one, where the first on call guy had thought it was a good idea to 'Initialise' the EVA5000 because it wouldn't come back online and present volumes.

      The fully loaded one for the prestigious client who's name I can't even hint at for legal reasons.

      Or the major client who had relocated their own data centre and managed to connect the disk shelves incorrectly and overwrite the metadata on the array for 3000 plus users' Exchange store because the IT manager thought 'changed configuration' was OK to accept...

      All in a day's work. Full, 24 hour day's worth of work.

    7. James Cane

      Was the office manager called Edina, by any chance?

  2. chivo243 Silver badge
    Trollface

    New Installment

    Sounds like a BOFH in the fold, are they taking one of Simon's pages?

    1. Fatman

      Re: New Installment

      Sounds like a BOFH in the fold, are they taking one of Simon's pages?

      A real bastard would have done some additional, inconspicuous damage - like short the pairs together in some hard to find location. Imagine the blue wire from one cable being cross circuited to the orange-white of a different cable altogether? Try figuring that one out.

      1. Anonymous Coward
        Anonymous Coward

        Re: New Installment

        "A real bastard would have done some additional, inconspicuous damage - like short the pairs together in some hard to find location. Imagine the blue wire from one cable being cross circuited to the orange-white of a different cable altogether? Try figuring that one out."

        Close. True, a real bastard would have shorted some cables. But first they'd leave a handy metal stepladder near the rack. Then they'd cut the cables at a length necessitating use of said ladder. The BOFH would arrange so that the condensate line from the air-conditioning unit is routed to deposit condensate around the comms racks (and would spill a bottle of table salt on the floor to enhance conductivity over the normal semi-distilled state). Then they'd short some wire, but none of this blue to green stuff. No, they'd make sure to short a few random cables to mains power.

        Last step on the way out: disable the lift, making the use of the metal stepladder that much more inviting when compared to lugging a company fiberglass unit down the steps.

      2. Alan Brown Silver badge

        Re: New Installment

        "Imagine the blue wire from one cable being cross circuited to the orange-white of a different cable altogether? "

        Imagine someone banging panel pins (those are 15-30mm long 1mm diameter nails with quite sharp points) into a few cable bundles and trunks in hard to notice areas.

  3. Tezfair
    Holmes

    Something else is happening here

    I can't for one second, think that anyone would take servers home with them or cut cables unless those items were owned by the support company. Other than theft, it will come under the computer misuse act. No IT person regardless of how bitter they are would go this far upon dismissal.

    I suspect the kit was supplied and there is a disputed bill. If the invoice states ownership remains with the IT company until the bill is paid, then (s)he would remove the equipment. Perhaps the IT person was on a contract, and at the end of the minimum period wasn't fully paid up to date, so took the personally owned / unpaid equipment to recover losses.

    1. illiad

      Re: Something else is happening here

      Point is, they may have realised how inept the bosses were (how stupid do you have to be, to 'trust' someone TWICE???), gave a false address in their CV, etc, etc...

    2. Steve Knox

      Re: Something else is happening here

      No IT person regardless of how bitter they are would go this far upon dismissal.

      Are you sure about that?

      Sometimes people do strange things, for strange reasons.

      Sometimes it's apparently just greed.

      Every field has its share of people who are disturbed, distraught, or jut plain jerls, including IT. In this instance, if it wasn't the leaving sysadmins, it was someone else with the access and the knowledge of what was valuable.

      Not too long ago, some rag did a writeup on more examples of this stuff, and how to deal with it.

      1. Stuart Castle Silver badge

        Re: Something else is happening here

        "Every field has its share of people who are disturbed, distraught, or jut plain jerls, including IT. In this instance, if it wasn't the leaving sysadmins, it was someone else with the access and the knowledge of what was valuable."

        This, combined with the access they have to potentially important systems is why it's important to do what they told us to do in my degree when firing contractors and sysadmins. Put simply, if you need to fire one, disable all his or her accounts, cancel his or her access cards (assuming your building uses them) and find a couple of burly security guards to escort them off the premises. All of which happened within 20 minutes of an ex-colleague at another company being fired, and the only reason it took 20 minutes was that his line manager needed to question him first.

    3. Alan Brown Silver badge

      Re: Something else is happening here

      "Perhaps the IT person was on a contract"

      Cutting the plugs off the ends of cables is a tad extreme.

  4. illiad

    I have seen worse on "Gordon Ramsay's kitchen nightmares"..!!

    HOW a Greek restaurant can survive for years with bad food, badly stored etc ... ??

    I have seen similar badly handled IT stuff.. The boss has NO CLUE, thinks it is all 'magic' ... IT guys work long days and nights to get the system going right, and WHO get the praise, and a promotion??? NOT the IT staff, the BOSS!! >:(

    and a few month later, the boss needs to 'save money' and sacks the IT guys!! Oh and the website, that was being maintained **for free** in house??? the boss decides to get 'a professional company' to do it for £5000 a month!!! :O :O

    1. Anonymous Coward
      Anonymous Coward

      "I have seen similar badly handled IT stuff.. The IT manager has NO CLUE, thinks it is all 'magic'... IT guys work long days and nights to get the system going right, and WHO get the praise, and a promotion??? NOT the IT staff, the IT manager!! >:("

      Fixed.

      1. K

        So your not bitter? get over it

        1. Jamie Jones Silver badge
          Headmaster

          "So your not bitter? get over it"

          His not bitter what?

          1. Pascal Monett Silver badge

            He's obviously fizzy, though.

            1. Destroy All Monsters Silver badge
              Trollface

              "Mine not bitter, your not bitter ... better get the case on twitter!"

          2. Bucky 2

            bitter what?

            Bit him, too.

    2. This post has been deleted by its author

      1. Destroy All Monsters Silver badge
        Trollface

        HOW a Greek restaurant can survive for years with bad food, badly stored etc ... ??

        Just waiting for a bailout, obviously.

        1. illiad

          yeah well, how much does it cost, site rental, gas, electric, water, etc, etc.... must be a few 1000's of credit, or re-mortgage, that will take a long time to pay off... :(

      2. Anonymous Coward
        Anonymous Coward

        Yep, I'm sitting on my hands waiting for them to come back begging at this very moment.

        I think the real costs of reversing the insourcing may just be being realised.

        The CFO who decided this route has, of course, already moved on. I was the last of his reports in place on his arrival to be made redundant/retire. Except for his PA, of course, without whom his incompetence would have been more visible sooner.

  5. x 7

    what I don't understand is......why the IT Managers were allowed to remain on site after being sacked. Any normal company has you out the door immediately.

  6. John Brown (no body) Silver badge

    The funniest I've seen...

    ...is two occasions of failed RAIDs within days at two unrelated small orgs.

    First one was a call in to say the server wasn't working. On arrival I find two failed HDDs in a 3 disk array. Pointed out the flashing red LEDs to the client who said "Ah,I thought that might be important. There only used to be one flashing light". How long has the other LED been flashing? "Oh, I don't know. It was doing that when I started here last year".

    A few days later, an urgent call to a site. Their HQ IT had identified a failed disk in a branch office 3 disk RAID. Courier will meet me there about 6pm to replace. I get there at 6pm. Only the security guy there. We wait there 'till 6:30pm, no sign of courier. HQ IT says to go in and have a look anyway please since the server appears to no longer be responding. We walk through and find someone has helpfully pulled the wrong drive out. HQ IT says go home, they fucked it. We'll deal with it during working hours and make them suffer a while. The courier still had not turned up by 7pm even with a 6pm guaranteed delivery.

    1. This post has been deleted by its author

    2. Cpt Blue Bear

      Re: The funniest I've seen...

      "...is two occasions of failed RAIDs within days at two unrelated small orgs."

      i've seen three in the space of 5 hours but they weren't unrelated. Two sites had a shared boundary and mainns feed with a football oval where they had recently installed light towers. The third was in the club offices. Cue Thunderstorm...

      "First one was a call in to say the server wasn't working. On arrival I find two failed HDDs in a 3 disk array. Pointed out the flashing red LEDs to the client who said "Ah,I thought that might be important. There only used to be one flashing light". How long has the other LED been flashing? "Oh, I don't know. It was doing that when I started here last year"."

      Many years ago I did support for an art school. Something to do with sins in past life, I assume*. I used to do a monthly sweep fixing the minor problems that never got reported, generally because they didn't realise they were problems. As soon as I walk into the front office I can hear a piercing BEEEEP every coule of seconds. I could hear it two rooms away but the staff in the office seemed oblivious. 'when questioned they said it had been going on for days, at least one had wondered what it was, but none had thought to follow the noise and find out.

      I tracked it two the server's UPS complaining about batteries that would no longer hold a charge.

      * Actually it was kind of cool 'cause there there were lot's of hot "arty" girls.

      1. Robert Helpmann??
        Childcatcher

        Re: The funniest I've seen...

        @Cpt Blue Bear, art school support is not too bad other than the lack of pay. My first sysadmin job was for one in Santa Fe, NM. At some point, they had hired someone to trench between (rented) buildings for a coax cable run between the two. Unfortunately, the trench was back-filled with the same stuff that came out of the hole, gravel and dirt, instead of packing sand around the PVC conduit that the cable was pulled through. Eventually, the driveway was used by something heavy enough to push a rock through the wall of the pipe which allowed it to fill with water. One of the two buildings had been the campus radio station. The mast was right next to where the trench had been run.

        Santa Fe might not get much rain, but it gets plenty of lightning. I put in reqs to have the trench re-worked or to pull fibre through the existing conduit, but the administration wouldn't pay for it (we had just been through a round of layoffs which had eliminated the other 75% of my department). I kept all the burned out boards due to lightning strikes and tacked them to the wall behind my desk. Eventually, an admin walked into my office and asked me what they were. I got approval to fix the problem after that.

        1. harmjschoonhoven
          Facepalm

          Re: The funniest I've seen...

          @Robert Helpmann??

          The bid to lay the cable ducts for the broadband fiber network in Belgium was initially awarded to to the lowest bidder. The snag was they thought to speed things up by not fitting watertight sockets between the sections of the cable ducts.

    3. Alan Brown Silver badge

      Re: The funniest I've seen...

      "A few days later, an urgent call to a site. Their HQ IT had identified a failed disk in a branch office 3 disk RAID. "

      No raid setup should _ever_ be so badly mangled that a single disk failure is grounds for an urgent callout.

      Hot standby?

    4. Bob H

      Re: The funniest I've seen...

      I did a placement at a local TV station, they had failure on a piece of kit and a bike courier was ordered with a replacement card for a "support within x hours" delivery. They were cursing when he didn't turn up because it made doing the news difficult and they stopped cursing when they were called to say the motorbike courier had been in an accident.

  7. Tony S

    2 stories.

    I had to fly to a remote site because they had lost connectivity. I had queried when it happened etc. but they assured me nothing had changed. When I got there, one of the network cables from the server had been cut through; and they had just tied the two ends together in an attempt at a reef knot!

    My brother set a company up with a system in the days of floppy disks and gave them several warnings about making copies of the disk so that if there was a failure, it would be possible to restore the data from a backup. This happened some 18 months later, so they asked him to go to site. On arrival, he confirmed the floppy was corrupt and asked for the latest copy. The person in charge opened a filing cabinet and produced a sheaf of A4 paper; on each one was a photocopy of the disk!

    1. Doctor Syntax Silver badge

      "the days of floppy disks"

      Ah, floppies. Back in the days of SCO my Oldham-based client had an offshoot in London. If they lost the HD SCO had to be reinstalled from about half a million floppies - at least that's what it felt like. I quickly learned that if I had to go down to do that I also had to take a spare floppy drive. It was a reprographics office and after a while enough the toner got into the drive to ruin it. Then there was the day when the operator rung up to say the ceiling had just fallen in on his desk....

  8. StephenH

    Not so strange that the revenge was happening in an identical manner

    I've seen lots of staff in companies do stupid things "because that was the way they did it "last year

  9. ecofeco Silver badge

    Employee disatisfication is that bad?

    “'THAT'S THE SECOND BLOODY TIME THIS YEAR!' they exclaimed, and my heart sank.”

    Just how in the hell are they treating their server admins?! That kind of damage is some very serious pissed off. I've know some unstable and outright crazy IT people, but that kind of action says "employer is an ass."

    Ah, I also saw the post that the theft (or repo) may have been from other employees after the fact.

  10. razorfishsl

    I find it difficult to believe, this is criminal damage and theft.

    1. Obitim

      Why?

      Because people don't break the law when they leave their job?

    2. ChaoticMike

      @razorfishsl... You aren't really telling me that you don't think that bad people do bad things, are you?

  11. Stoneshop
    Coat

    "Pauli says there were no lights down in the basement, "

    No stairs, and a cupboard in a locked lavatory with a sign on the door saying "Beware of the leopard".

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like