back to article Tech support made the news after bomb squad and police showed up to 'defuse' leaky UPS

By golly it’s Friday again! Which means a spot of R&R isn’t far off, once we get through On-Call, The Register’s weekly column recounting readers’ tales of terrible times in tech support. This week meet “Len” who told us he works “for a small but long-established IT company” n a major Scottish city. “We do a range of work …

Page:

  1. jake Silver badge

    "Why the massive emergency services response?"

    Obvious answer: They was using up all kinds of cop equipment that they had hanging around the police officer's station.

    1. Anonymous Coward
      Anonymous Coward

      Re: "Why the massive emergency services response?"

      > They was using up all kinds of cop equipment that they had hanging around the police officer's station.

      Without the colour 8x10 glossy photographs with circles and arrows and a paragraph on the back of each one, I won't believe it.

      1. Solo Owl

        Re: "Why the massive emergency services response?"

        You forgot to credit Arlo Guthrie....

        1. Jeffrey Nonken

          Re: "Why the massive emergency services response?"

          "You forgot to credit Arlo Guthrie...."

          He didn't have to. Sixty three of us have done it for him.

    2. Antron Argaiv Silver badge
      Thumb Up

      Re: "Why the massive emergency services response?"

      As a teen of the 70s, a resident of Massachusetts, and a proud graduate of UMass Amherst, I salute you, sir!

      // one big pile is better than two little piles

      1. werdsmith Silver badge

        Re: "Why the massive emergency services response?"

        Hey, I have UMass Amherst on my educational CV.

        What big pile? The library tower?

      2. Jeffrey Nonken

        Re: "Why the massive emergency services response?"

        I have only the "teen in the 70s" part going for me, but yeah. Also at one employer we would play the record every Thanksgiving.

    3. FBee

      Re: "Why the massive emergency services response?"

      They also took 27 8X10 colour glossy photographs with circles and arrows and a paragraph on the back of each one explainin' what each one was.

  2. Anonymous Coward
    Anonymous Coward

    Obligatory

    /r/techsupportgore

    But yes, have seen at least one UPS battery go bad like this, the battery essentially split open like the Hulk's shirt.

    Also had one E-bike battery do this as well during a charge attempt after not being used for a long time.

    "He is leaking!"

    1. Stevie

      Re: Obligatory

      I've had many gel lead/acid batteries swell and split. No liquid ones though. My guess is that an internal short forms and heats the gel. A liquid battery "just" boils off the acid until it is gone, which is why you want unsealed ones.

      A colleague had a car battery explode when he did the wrench-short across the terminals thing while working on his car. Wrench welded itself to the terminals in sparrow's fart time, and the battery electrolyte boiled merrily until the casing burst/slagged (not sure which) and the various steamy vapours touched the white-hot wrench and ignited.

      He walked around with a haunted expression for years afterward.

      I also worked with someone some years back who had a mini like mine with the battery under the back seat. His son did some work on it in order to earn car-borrowing privs, and unfortunately left the spanner on one of the battery terminal tightening bolts before closing the seat. The spanner slowly worked its way across both terminals and provided much excitement a few days later as my friend drove to work. Luckily it didn't do the welding-itself-to-the-posts thing, how we never figured out, but the sparking, burning and driver-screaming-like-a-litlle-girl were still impressive, I was told.

      Years later this same guy lifted the engine out of the mini and discovered a socket wrench that had been sitting on the front sub-frame for years, also down to the kid.

      1. ds6 Silver badge

        Re: Obligatory

        Good tales—but I must point out the usage of a sparrow's fart as a unit of measurement. That can't be an approved SI unit.

        1. albegadeep

          Re: Obligatory (Units of measurement)

          There is a long-standing tradition of unusual, but Vulture-approved, units of measurements:

          https://www.theregister.co.uk/2007/08/24/vulture_central_standards/

          Oddly, there's no entries for time. "Sparrow's fart" might well be a good standard for short timespans - any suggestions for longer ones? Paint drying, bad speeches, Shelby light bulb?

          1. onefang

            Re: Obligatory (Units of measurement)

            '"Sparrow's fart" might well be a good standard for short timespans - any suggestions for longer ones?'

            The amount of time it takes a government to make a good decision. Though that tends to a very long time, or never. Or if you want something shorter, frozen hells. So the average length of time taken for an average government to make a decision that makes the world a better place is 6.4 frozen hells, +/- 0.1.

  3. Dave K

    Haven't had a battery split like that, however I've come across numerous situations where the battery has failed in a UPS, the site has been too lazy to do anything with it for months on end, then when we finally turn up to look at it, the ancient battery has started to swell - such that it's no longer possible to remove it from the UPS. Most sites aren't impressed at being told that the entire UPS now needs replacing due to their laziness...

    1. akoepke

      Why is there an incessant beeping noise coming from the room with the IT stuff in it? Who cares, it is still working.

      1. Rich 11

        Back in the day...

        *ring* *ring*

        "Hi, Rich 11, it's X over at Y! Can you come and change the bulb on one of our servers, please? It's been flashing bright red all week and annoying all the people on this side of the office. Have you got a dimmer bulb that will flash more slowly?"

        Yes, pal, I could certainly identify a dimmer bulb in the vicinity.

        1. JimboSmith Silver badge

          Re: Back in the day...

          Years ago had a test rack in the office which we were using to evaluate some equipment. There was one piece of kit with an extremely nice blue light on it. People who passed the rack both technically minded and not commented on how much they liked the light. So when the MD of the firm came in to have a chat about the equipment the first question was where they sourced the light from (it was an LED but a very tasteful blue). He said he had no idea but would check and let us know. He was surprised that that was the first thing mentioned and not the functionality of their kit.

      2. Andytug

        That beeping is....

        at 7am trying to power up after an electrical inspection, the UPS trying to tell you that there's no mains power to the cabinet. Which if you ignore it results in all the servers gracefully powering off about 10 mins after startup, as the UPS tells them to as its battery is nearly empty........

        1. swm

          Re: That beeping is....

          I was working on my home computer and the UPS started beeping like there was no input power. But the lights in the room were working so I was puzzled until I found the UPS had blown a fuse on the input and my computers were running on battery.

      3. Bluebottle

        What alarms ?

        Our local shiny new multi-billion dollar teaching hospital had its first generator test under full load but low fuel levels meant that operating theatres lost power during operations (one surgeon was the head of the Australian Medical Association) and people got stuck in lifts.

        The inquiry found that maintenance staff had ignored the low fuel warning alarms for several days, and anyway the contractor hadn't told the hospital that the test was going to take place.

    2. Steve Cooper

      Had similar with a load of old racks in the comms room had 10+ year old APC units in the bottom of them where the battery had less capacity than a potato and were just being basically used as multiway plugs, with a building-wide UPS having been installed years ago. Cue a black building power down test. Turn it back on again and not a single 10+ year old UPS turned back on (how dare they). Cue running down to the local Robert Dyas/Maplin to buy a load of 4/6/10 way adapters and kettle leads.

  4. Anonymous Coward
    Anonymous Coward

    UPS' can be nasty

    we had of of the big boys (think 32u rack size) explode on us and chuck it's crap across the room.

    Luckily it was in a basement, but the clean up was not fun and took the site out for almost a week.

  5. This post has been deleted by its author

  6. Dodgy Geezer Silver badge

    You were lucky...

    duplicate - removed

    1. jake Silver badge

      Re: You were lucky...

      duplicate reply - removed

      1. TRT Silver badge

        Re: You were lucky...

        Lucky, lucky, lucky.

        1. John G Imrie

          Re: You were lucky...

          Hay Kiley is that you?

        2. Flakk
          Joke

          Re: You were lucky...

          Lucky, lucky, lucky.

          You don't know what a rotten borough is, do you?

          1. Stevie
            Pint

            Re: You were lucky...

            *I* do Flakk.

            E-beer for you.

          2. Midnight

            Re: You were lucky...

            Err... so... what is a rubber button?

  7. wyatt

    One of our engineers was gassed by a leaking UPS battery pack, had to be extracted from the comms room by the Fire Brigade and spent 6 weeks in hospital

    Brand new UPS in an Army vehicle only provided about 30 seconds of power during a generator change. On inspection, most the batteries weren't connected.

    UPS in my last job had a firmware issue which caused an outage for a few days. Was very strange when I noticed and went to check and everything was dark in the comms room.

    Army vehicles again, had some very awkward battery storage locations. Many a time there have been sparks flying when trying to maintain them. Never did get use to it, hate the things.

    1. Lee D Silver badge

      I do think people underestimate the risk of such batteries.

      Such, the PROBABILITY of such an accident is incredibly low, almost immeasurable if they are properly maintained.

      But the IMPACT of such a thing can be incredibly dangerous, more so than people think because they are so used to them "just working".

      The energy density, however, is very high... a small box can power lots of heavy servers for quite a while, considering. People really underestimate the power these things hold. In the normal course of things, if all safety measures are working, they are pretty benign. But if that energy is released all at once, plus the chemicals in use, etc. then they can be bombs.

      My dad tells a story from pre-H&S days, when he worked in a huge warehouse maintaining goods lorries. One day, they had a fork lift that was being retired on a site they were demolishing the next week anyway. They decided to have some fun with batteries... everything from shorting out a lorry battery with a spanner, to the same on the forklift battery (which was just multiple of the same in parallel).

      Generally speaking, the spanner turned red-hot, then white-hot and then shattered explosively into two from the lorry battery, observed from the comfort of a makeshift bunker, which is scary enough. So they moved on to the forklift. Apparently, the resulting explosion (conducted via a large metal spanner and a long piece of rope) completely obliterated the fork-lift, nearly deafened them, and sprayed battery acid on all four walls and the ceiling of a huge empty truck warehouse. It took them two days to wash it down, and they kept finding parts of the fork-lift in odd places, and lodged in the walls.

      Sure, that's the EXTREME end, but that's just ordinary lead acid batteries in tandem. The lorry battery thing is scary enough, that it can melt/bend/explode a fitter's spanner in seconds. That kind of power discharging into, say, water, metal shelving, etc. is a scary thought.

      I've always been wary of large batteries because of such stories, I've never had trouble myself but I don't want to have. The nearest I got was a Macbook with a battery that bulged so much that it destroyed the casing and kept visibly expanding once released. We dropped it into an empty wheelie bin a long way from anything and it ended up going to the skip months later when it had stabilised and calcified and leaked all over the inside of the bin (it was suggested to pour water on it, but that was hastily rebutted, being lithium).

      Especially UPS batteries - the APC one I have had a large battery tray that one man can barely lift and - taking it apart to look at the individual battery modules (which is just a bunch of RPC6's wired together to give 48V and more capacity), there is no fuse on the battery itself. The fuse is in the cable on the UPS side that it connects to, not the battery. So when changing those things out, it's quite possible that you could drop something into the casing (e.g. a screwdriver) and short that battery out with no safeties.

      Anything with energy density like that is a dangerous thing. I've seen 9v NiMH batteries explode when charging and cover a primary school classroom in acid (fortunately, no children in the room at that point). I've shorted out NiCd AA batteries using a basic science kit (intended for alkaline) as a kid and literally set the battery casing and wires on fire in seconds. Sure, you have to do something stupid, but it's relatively easy to do something stupid by accident. And then you're into "fire hazard" as a minimum and, with large lead-acids, potential bangs that can fire metal shards around a room.

      1. Jonathon Green

        Large batteries are Not To Be Taken Lightly. I have a very large spanner which looks like an animal took a bite out of it to remind me of this and that was just a diddy motorcycle battery.

        I try not to think about this too much when I’m barrelling down the A120 at 85MPH on top of 30kwh of Lithium batteries, but I suppose in energy density (and ease of ignition) terms it’s probably less of an issue than half-a-tank[1] of petrol...

        [1] To give plenty of space for a nice juicy mixture of air and petrol vapour.

        1. Lee D Silver badge

          Generally petrol won't explode entirely in a single few seconds. Though you can get a bang, it has to be aerated and dispersed to make a single fireball.

          In general, it will burn fiercely and slowly for several minutes with maybe a bang every now and then. Sure, your car will be totally burnt but generally you have time to get out.

          A battery, especially Li-Po etc., can disperse its entire energy and hit runaway temperatures in seconds. There are videos of laptop batteries catching fire and going bang on people's laps - you'd think that would be the kind of thing you could feel coming. And a guy was just killed by a vape thing exploding: https://soylentnews.org/article.pl?sid=18/05/17/1242259

          Yes, the energy density of both is high, and petrol is nearly twice as much (I believe), but it's how it burns. You need something catastrophic to set fire to your fuel lines, and even worse to the tank. But you just need a stray bit of metal in the wrong place, or for some kind of contact with the chassis, with a battery.

          Plus, lead-acids generally release hydrogen when charging, and lithiums can't contact water, which doesn't help.

          P.S. So many people don't follow the right procedure for jump-starting a car... sure, they get away with it an awful lot. One day they won't.

        2. Anonymous Coward
          Anonymous Coward

          > Large batteries are Not To Be Taken Lightly.

          We knew a guy in the village who worked for a big battery company. We called him Barry Nine-fingered.... (In this case, a huge glass battery bank for a lighthouse. Batteries were taken up first, then the acid. Barry made a teensy tiny little mistake connecting them up.)

          Oddly, my father had a similar experience, welding his wedding ring onto his skin when using a spanner long enough to short the terminals on a battery bank. So when I work on our off-grid power supply, over 1000AHrs at 24v, I tend to be rather cautious, using short, rubber-clad spanners, and lots of adrenalin.

          1. Anonymous Coward
            Anonymous Coward

            automotive batteries

            " spanner long enough to short the terminals on a battery bank"

            Another area to beware of is the starter lead on ca. 1980 GM full size sedans (Regal, Cutlass Supreme, etc.). If you're fishing a new wire down to the starter while building a race car, there's a wedding ring sized gap between the starter stud and a nice big ground (exhaust pipe, IIRC). Somehow I avoided going arcy sparky.

            1. Fruit and Nutcase Silver badge
              Joke

              Re: automotive batteries

              @AC

              If you're fishing a new wire down to the starter while building a race car, there's a wedding ring sized gap between the starter...

              The moral of this story is that pursuing such a hobby is not always compatible with being married

        3. Anonymous Coward
          Anonymous Coward

          " barrelling down the A120 at 85MPH"

          Are you sure about that? Only asking, because you just admitted on a public forum that you were travelling fast enough to get a driving ban....

          1. Stratman

            AC wrote "you've just confessed to a heinous crime"

            With an uncalibrated uncertified speedometer...

          2. Anonymous Coward
            Anonymous Coward

            " barrelling down the A120 at 85MPH"

            Are you sure about that? Only asking, because you just admitted on a public forum that you were travelling fast enough to get a driving ban....

            if he was late picking the missus up from Stansted, then I would be doing a little more than 85,,, risk driving ban or the wrath of the missus pissed off at standing waiting for collection after a long flight?

            1. Anonymous Coward
              Anonymous Coward

              Barreling down the A120

              Dunno about the A120, but last time I was in the UK I was regularly exceeding 90 in the right lane on various M roads - basically just matching the lane's speed of traffic. I don't see what the big deal is with someone admitting to it, given that it isn't at all an uncommon speed of travel.

              If the A120 is one of those winding country roads with two lanes barely as wide as 1 1/2 lanes would be in the US, then I'd agree that's a little crazy...

          3. IanRS

            It was obviously a mis-typed KMPH, of course!

          4. Fruit and Nutcase Silver badge
            Alert

            "barrelling down the A120 at 85MPH"

            This will get around the issue of admitting a traffic offense in that statement...

            "Bombing along the Queen's Highway"

            ...but may still attract undue attention for other reasons

          5. Anonymous Coward
            Anonymous Coward

            Assuming the A120 is a national speed limit dual carriageway, 85 isn't enough for a driving ban. I drive at 95 on motorways regularly, safely below the cut-off for a driving ban.

      2. THMONSTER
        Mushroom

        Boom!

        Also have had experience of truck and forklift batteries exploding (forklift was a diesel model so not too bad, just a large 12v) in my days as a truck mechanic before I ended up slumped behind a keyboard and wall of monitors.

        They certainly do go with quite a bang, one of the truck batteries went whilst jump starting it with another truck and caused a gas cloud to pour from the battery in question.

        Other activities were far more dangerous at that job though, spare time and high pressure air from the compressor makes for crazy dangerous inventions.

        Half inch thick bolts launched hundreds of yards were a regular occurrence!

        The devil makes work for idle hands!

        1. Ben Tasker

          Re: Boom!

          > Other activities were far more dangerous at that job though

          If you want to see something equally scary, try googling for how to de-sulphate a lead acid battery.

          I guarantee you'll find more than a few people recommending that you do the following

          - Put the battery on a bench

          - Get your arc welder and clamp the cathode onto the positive terminal

          - Turn the welder on

          - Repeatedly tap the anode against the negative terminal

          The theory being that the (high) charge going through should shake the suplhate back off the plates.

          Of course, that'll lead to a release of hydrogen, and it's not as if the primary fucking task of an arc welder is to create an arc/spark hot enough to melt metal (let alone ignite hydrogen)....

          1. Anonymous Coward
            Anonymous Coward

            Re: Boom!

            To quote Mark Watney: "in the history of humanity, nothing bad has ever happened from lighting hydrogen on fire"

          2. Anonymous Coward
            Anonymous Coward

            Re: Boom!

            I recently googled how to rescue a set of NI-CD drill battery and saw a lot of people suggesting a similar thing. I thought "that sounds like a good plan with no danger of fiery death".

        2. dmacleo

          Re: Boom!

          big bang very quickly with 4 6volts wired series and someone makes one wire mistake.

          was supposed to be 2 "banks" of 2 6volts in series and guy decided to hook cable from another truck across last 2 terminals instead of the truck (ironically fire truck) + / - leads.

          luckily we had MANY fire extinguishers and axes that would cut cable very handy LOL

        3. Mark 85

          Re: Boom!

          Indeed... very much will they go bang and with many times disastrous results. Back in the day, I did some drag racing. I wasn't there one particular Sunday but heard this from a friend who was. Many of the guys would put 4 12V batteries in the truck wired in parallel. What this did was give them "legal" ballast and also high amps for starting high compression engines. Well, apparently one lad who knew it all had a starting problem. Took his batteries out and put them on a charger. When he put them back, he was connecting them as he installed them while having a beer (stupid thing to do in the pits, IMO, but I digress). The last batter was set into place backwards (reversed). From what I heard the "boom" as he connected the last cable was loud and the guy spent several months in the hospital from burns. Luckily no one else was hurt but there was shrapnel in the cars on either side of his in the pit area. The only thing we could think of is that he had the power turned on in the cockpit while testing and forgot to turn it off before working on the batteries.

          Lesson.. do fool around with batteries unless you can give the job your full attention.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like