back to article Noise from blast of gas destroys Digiplex data depot disk drives

It's a bit rich when a fire prevention system designed to protect a data centre kills its servers, but that's what happened when inert gas was injected into a Nasdaq Digiplex bit barn in Sweden. The gas release – on 18 April – caused a loud, high-pitched whistle, which resulted in vibrations destroying server disk drives, the …

  1. Anonymous Coward
    Anonymous Coward

    Meh....

    ...we used to had to replace a Halon system, not because you would be killed by the gas, but by the floor tiles being thrown several metres in the air at high speed.

    1. Anonymous Coward
      Anonymous Coward

      Re: Meh....

      I assume the replacement came after the bulk of the management had been addressed and a lower capacity re-training system was more appropriate?

  2. Anonymous Coward
    Anonymous Coward

    Safe for personnel?

    It doesn't matter what mix of inert gasses you have in your fire suppression system, anyone left in there will die if the oxygen level gets too low.

    They really ought to use inert gas mixtures in US gas chambers, they're quick, totally painless and kill without any sense of suffocation or fighting for breath.

    1. kryptylomese

      Re: Safe for personnel?

      "They really ought to use inert gas mixtures in US gas chambers, they're quick, totally painless and kill without any sense of suffocation or fighting for breath."

      Or just completely abolish the death penalty like the rest of the humane world did decades ago!

      1. Anonymous Coward
        Anonymous Coward

        Re: Safe for personnel?

        I don't disagree about abolishing the death penalty (if only because once you've offed someone it's too late to say sorry if you then find they were innocent after all), but if you're going to do it then at least do it without additional suffering.

        1. Charles 9

          Re: Safe for personnel?

          I'm also conditionally for the death penalty, but ONLY for the very worst of the worst. I mean, what else can you do to someone who's an avowed and repeat killer and escape risk (say someone with a reputation like El Chapo's), what options do you have left being as he's a continual threat as long as he's alive?

        2. Gene Cash Silver badge

          Re: Safe for personnel?

          > do it without additional suffering

          Why? They sure as hell didn't extend that courtesy to their victims.

          My argument against capital punishment is that the state couldn't properly arrange a piss-up in a brewery, so they can't be trusted to do it right.

          1. handleoclast

            Re: Safe for personnel?

            My argument against capital punishment is that the state couldn't properly arrange a piss-up in a brewery, so they can't be trusted to do it right.

            One of my arguments too, after I've read about the number of botched executions in the US.

            Another argument is the number of false convictions (particularly in the US) where decades later evidence surfaces to prove the convicted person is totally innocent. You cannot fully compensate somebody for spending a couple of decades in jail that s/he shouldn't have but you cannot compensate somebody you wrongly executed at all.

            My main argument is that the death sentence is less of a deterrent than a life sentence for religious people. Statistics show that crime rates of all kinds are higher in countries (or US states) with high religiosity. My suspicion is that the religious see it as "If I kill him he'll end up in the appropriate place in the afterlife, and he'd have ended up in the same place eventually anyway so it doesn't really matter that I sent him off the pitch early. And anyway I can repent, say sorry to Jesus and go to heaven anyway." They don't seem so sanguine about it when they end up on death row, but as a deterrent it's less effective.

            My number two argument is that I'm an atheist and think death is final. Even a few decades in prison is probably preferable to oblivion. Although I suspect a few encounters in a prison shower might change my mind on that.

            1. aks

              Re: Safe for personnel?

              Monsieur Guillotin invented such a device.

              "The machine was successful because it was considered a humane form of execution, contrasting with the methods used in pre-revolutionary Ancien Régime."

              https://en.wikipedia.org/wiki/Guillotine#Introduction_in_France

              1. Anonymous Coward
                Anonymous Coward

                Re: Safe for personnel?

                Except after a few live demonstrations, it was found both the head and the body could keep moving and reacting after the decapitation, raising the distinct possibility either one could still feel after the fact, especially given our more-advanced knowledge of the processes that go on in an actual death.

          2. Jason Bloomberg Silver badge

            Re: Safe for personnel?

            > do it without additional suffering

            Why? They sure as hell didn't extend that courtesy to their victims.

            Because we are better than they are. It's not meant to be a race to the bottom. It's not meant to be an opportunity for revenge or venting desires to inflict suffering on others.

            1. Anonymous Coward
              Anonymous Coward

              Re: Safe for personnel?

              If the sound was able to damage hard drives, what would it do for a person's hearing? I wouldn't consider something "safe for personnel" if left them alive, but deaf.

      2. Anonymous Coward
        Anonymous Coward

        Re: Safe for personnel?

        Or just completely abolish the death penalty like the rest of the humane world did decades ago!

        Not really wanting to get into a debate on the death penalty, but some people who cant be rehabilitated and whose actions were so abhorrent they don't deserve to share in our oxygen supply.

        considering the cost of keeping said sub-humans locked up for the rest of their life, terminate them.

        1. allthecoolshortnamesweretaken

          Re: Safe for personnel?

          "considering the cost of keeping said sub-humans locked up for the rest of their life, terminate them"

          Exactly the "reasoning" behind this.

      3. Gene Cash Silver badge

        Re: Safe for personnel?

        > ought to use inert gas mixtures in US gas chambers

        That's the case in Oklahoma now.

    2. Wily Veteran
      Meh

      Re: Safe for personnel?

      Supposedly, Halon was heavier than air and would not build to a dangerous concentration at the height of a human's head until enough time had elapsed to allow the operators to initiate an orderly shutdown before they evacuated. Supposedly. That's what we were told in the 70's anyway.

    3. MonkeyBob

      Re: Safe for personnel?

      They really ought to use inert gas mixtures in US gas chambers, they're quick, totally painless and kill without any sense of suffocation or fighting for breath.

      Micheal Portillo did a BBc documentary on this a few years ago. When he asked US people if they preffered the gas method most didn't as it no longer seemed like a punishment.

      http://news.bbc.co.uk/1/hi/magazine/7183957.stm

    4. Anonymous Coward
      Anonymous Coward

      Re: Safe for personnel?

      Thoguht,

      Are you so sure about those assertions ?

      I am sure the 'lack of oxygen' would be noticed by your body and appropriate 'actions' would be taken.

      These actions are not painless or lacking in any sense of suffocation.

      To quote text on 'Inert gas asphyxiation' :

      Some individuals experience headache, dizziness, fatigue, nausea, euphoria and some become unconscious without warning.

      Loss of consciousness may be accompanied by convulsions and is followed by cyanosis and cardiac arrest.

      The painful part(s) are the 'Convulsions' & 'Cardiac Arrest', at least.

      Symptoms can include: blackouts, confusion, drooling, loss of bowel/bladder control, sudden shaking of entire body, uncontrollable muscle spasms, temporary cessation of breathing.

      All in all not as 'gentle' a way to 'go' as it would seem !!!

      [Only if you are 'lucky' will you fall unconscious and your heart stop, otherwise not so nice !!!]

      Although, in line with the vengeful attitude towards killing Criminals that exists in the US of A, it is probably 'Good enough' .......... unless you have convicted the 'Wrong person' !!!!

      Don't worry though, that never ever happens does it ???

      1. Gene Cash Silver badge

        Re: Safe for personnel?

        > I am sure the 'lack of oxygen' would be noticed by your body and appropriate 'actions' would be taken.

        Actually, no, it's not. For some reason only dogs notice the lack of oxygen... other mammals rely on CO2 accumulation. The nitrogen keeps the CO2 down, so you don't feel anything.

        I've seriously looked into this, as it's my "treatment of choice" for Alzheimer's.

        It first came to my attention when 3 Shuttle workers died in the aft compartment because it was nitrogen purged but not signed. Rescuers almost "fell asleep" themselves.

        1. Anonymous Coward
          Anonymous Coward

          Re: Safe for personnel?

          Gene Cash,

          I would research a little deeper.

          Inert gases are not your answer.

          There are chemical compounds that do what you require and quickly but they are lethal to *everyone* and residual pockets accidentally left behind in the Death Chamber are too dangerous for the use to be sanctioned. (Not to mention the 'Safe Handling' risks !!!)

          I will not name them BUT anyone with a little knowledge in Chemistry could find them.

          BTW:

          You will not be able to order them from Gallenkamp* (Fisons as they are now known) or your favourite Chemical Supplier either :)

          *Gallenkamp gives away when I worked in Chemistry :)

      2. handleoclast

        Re: Safe for personnel?

        I am sure the 'lack of oxygen' would be noticed by your body and appropriate 'actions' would be taken.

        Nope. Not for most people. In most people the breathing reflex is triggered by build-up of carbon dioxide, not lack of oxygen. There are some medical conditions that can result in oxygen depletion being detected instead, but they take a long period of bodily adjustment (we're talking months, not minutes). Confirmation of this can be seen in simulations of aircraft decompression where people pass out without realizing there's anything wrong, particularly if the change in pressure is gradual rather than fast.

        To quote text on 'Inert gas asphyxiation' :

        Largely depends on the "inert" gas. Nitrogen and carbon dioxide, although it takes a lot to get them to react, do have nasty biological effects (with nitrogen you need a lot higher partial pressure than in the atmosphere, but then you get nitrogen narcosis). As do several other "inert" gases, including xenon. Xenon (despite being a noble gas) is, bizarrely, an anaesthetic (so a good choice for killing people painlessly).

        Lack of oxygen alone isn't going to have any nasty side-effects. Unless, for example, you suffer from chronic respiratory acidosis, in which case your breathing reflex will be triggered by low oxygen rather than high CO2 and then you'll know there's something wrong.

        1. Anonymous Coward
          Anonymous Coward

          Re: Safe for personnel?

          Gas, injections, and electrocution all seem cruel compared to traditional methods like shooting, hanging, and guillotining. Not as cruel as "Oops, we wrongfully convicted you 30 years ago, you're free to go now, sorry about the sodomy and AIDS" however!

    5. An0n C0w4rd

      Re: Safe for personnel?

      When I worked in a facility that had a gas discharge fire system (FM200 from memory) we were told to get out of the room ASAP when the alarm went off as being in there when the discharge went off would likely blow out your ear drums and possible cause other problems also. Even people near the room weren't safe as for some reason they put windows in one of the exterior walls to a corridor so they could show off the contents of the DC. The glass wouldn't survive the discharge

      1. Alan Brown Silver badge

        Re: Safe for personnel?

        "The glass wouldn't survive the discharge"

        In that case I hope the volume calculations made sure to include the corridor, else there's a good chance the room fire wouldn't get put out.

    6. Uberior

      Re: Safe for personnel?

      It was the suicide method of choice for a former colleague.

      He blocked the door to the server room from the inside, banged on the windows whilst holding up a sign detailing that our boss was a "fud" (check the Scottish slang).

      Then activated the extinguishers. They extinguished.

      It was awful. Obviously our boss was promoted out of the department...

  3. Davegoody

    After a particularly long liquid and curry lunch.....

    I too have destroyed HDDs with a rapid expulsion of Noxious gasses...... I'll get my coat....

    1. ICPurvis47
      Mushroom

      Re: After a particularly long liquid and curry lunch.....

      Didn't you fit a silencer, then?

  4. Nate Amsden

    why replace servers

    Seems that simply replacing the disks would be much quicker. Though perhaps their servers don't have easily swappable drives.

    1. Stoneshop

      Re: why replace servers

      The 'servers' apparently comes from a quote by the Nasdaq Nordic CEO, who probably understood the technical explanation as "thingies in compjooter room went boom". And what's likely being replaced are the disk shelves, or maybe the entire SAN racks. I can think of a few reasons to do it that way instead of just the disks themselves, like the old disks not having a direct replacement any more and the current generation being 2.5" instead of 3.5", or shelf controllers not being able to handle larger capacity drives with again the correct-sized replacements having been phased out.

  5. Anonymous Coward
    Anonymous Coward

    " fitted with a drilled orifice"

    fnarr

  6. Daedalus

    It's a gas, man

    Lest anybody think that 200 bar gas cylinders are out of the ordinary, they are pretty much standard for any application that uses gases such as nitrogen. They're the 5ft high, 10 in wide ones in the corner. Or if you like, the one rocketing along the street in the stupid prank video you downloaded.

    Oh and if you have a CO2 cylinder, there's this great trick for making dry ice......

    1. Stoneshop

      Re: It's a gas, man

      They're the 5ft high, 10 in wide ones in the corner.

      The ones I come across are usually quite a bit wider than that, more like 16..20 inch.

    2. DNTP

      Re: It's a gas, man

      My personal gas cylinder secured next to my lab bench is argon, analytic grade, 1500psi/100bar at full.

      On an "off day" I gave my sequencing tech the full and empty tank masses, the speed of sound in argon gas, and challenged us to calculate the ideal delta-v of that tank as a mathematical exercise.

      It was around 80mph/130kph. For a five foot tall steel pressure tank. We did not verify the calculation experimentally because we would rather have two jobs and zero new round windows.

    3. Alister

      Re: It's a gas, man

      I once saw the aftermath of dropping a full "F" size Medical Oxygen cylinder (4 foot tall and 8 inch diameter).

      The valve neck snapped, and it departed through the side of an ambulance, went through two brick walls, and disappeared into the woods at the side of the ambulance station. We never did find it...

  7. Stoneshop
    Boffin

    It's hard to not be extremely loud

    when emptying a bunch of high pressure gas cylinders as fast as required so as to suppress a starting fire. Maybe you can keep it down at just bloody loud instead of fscking loud, but loud it's going to be. It's to do with physics.

    1. Pascal Monett Silver badge

      Re: It's hard to not be extremely loud

      It's not the loud that is the problem, apparently, it's the whistling.

      I'd suggest adding another nozzle size, say 1.43 linguine, or 7.9 inches. Compared to the 0.9, 0.13, 0.18 and 0.27 linguine sizes, that should allow for rapid gas egress without whistling, I would hope.

      1. John Brown (no body) Silver badge

        Re: It's hard to not be extremely loud

        ..or even just put a pressure reducing valve in. The claim in the article is that the system can pout a fire out in 60 seconds. Not really that helpful if it then destroys all the HDDs in the room. Maybe a a lower exhaust pressure taking 120 seconds might be more economical and just as safe.

        1. anothercynic Silver badge

          Re: It's hard to not be extremely loud

          That depends on whether you're looking at an inferno that spirals out of control in 120 seconds or not.

          But yes... high-pitch whistling breaking disk drives is... not helpful.

          1. John Brown (no body) Silver badge

            Re: It's hard to not be extremely loud

            "That depends on whether you're looking at an inferno that spirals out of control in 120 seconds or not."

            That's a reasonable point but I'd counter that with the point made in the article that the fire suppression system can detect the slightest whiff of combustion particles even before it becomes a fire, eg extreme overheating beforehand. I remember seeing fire detection systems of that sensitivity at Kidde over 25 years ago, so anything that can turn into an inferno in a data centre in under 120 seconds from being detected is going to be a damned serious event.

  8. Norman Nescio Silver badge
  9. Blockchain commentard

    It should have the power supply, including air-con, linked so it powers off any kit before the gas goes off.

    1. Anonymous Coward
      Anonymous Coward

      Fire doesn't care about your graceful shutdown.

    2. Stoneshop

      Power cut

      It should have the power supply, including air-con, linked so it powers off any kit before the gas goes off.

      Congratulations, you've now buggered up that entire datacentre compartment. Transactions halted mid-stream, file systems unclean. Systems that were running 24/7 now having to cold start. And spinning rust doesn't stop and retract the heads immediately once you kill the power, so the danger of sound damage is still there for a moment.

      And all because just one piece of kit was getting rid of its magic smoke, and the fire suppression system going about its business in trying to keep it contained to that one box. Which, in this case, didn't work out so well, never mind that it was a false alarm, but I doubt very much that a power cut would have made for a better outcome. See, for instance, the Heathrow data centre failure a couple of months ago.

      1. jelabarre59

        Re: Power cut

        And all because just one piece of kit was getting rid of its magic smoke, and the fire suppression system going about its business in trying to keep it contained to that one box

        Well, don't have Lucas Electrics build your datacentre then. No more magic smoke evacuation.

  10. myxiplx2

    Yup, seen this before

    About 7 years ago I was working as a support engineer for a storage company when we had a call from a client who'd run into this exact issue.

    About half of the 15k drives in their datacentre had utterly failed after a fire suppression test, affecting a wide variety of manufacturers systems. There was too much data loss for any attempt at recovery and it was the IBM engineering team who were the first to identify the root cause.

    7k and 10k drives were unaffected, it seemed the particular frequency of the nozzles at this site managed to hit the resonant frequency of 15k drives, causing the heads to physically impact the platters, utterly destroying the drives and data beyond any reasonable hope of recovery.

    Cue one DR program invitation, and a very sizeable insurance claim!

  11. L'Ecossais

    Time to invest in SSD's?

    It's not as if it hasn't happened before - https://www.theregister.co.uk/2015/12/17/glasgow_council_it_meltdown/

  12. Anonymous Coward
    Anonymous Coward

    Happens more often than you might think...

    Failure rate should be less than 50%. In a modern server environment, with local redundancy, RAID storage arrays and geographic replication, you’d have to be a tad unlucky to lose any data. Bloody expensive to fix though - getting the gas canisters removed and refilled costs a small fortune.

    1. Anonymous Coward
      Anonymous Coward

      Re: Happens more often than you might think...

      I hope you understand that just because you lost say 45% of disks and use mirroring, that doesn't imply there is no data loss - because both mirrors will fail in many cases. I'm sure someone good at calculating odds could figure out if you had say 1000 disks and lost 5%, what the odds are that you lost both ends up a mirror. If you use 6D/2P double parity RAID odds are good you would survive losing 5% of drives, but maybe not 10%.

      A stock exchange really should have a remote replica - probably not geographic because latency is going to be a problem, but even a few miles away would insulate it from issues like this one, a power outage that affected both is unlikely to be an issue since that will protect you against one-off issues with a UPS or generator transfer, etc.

  13. This post has been deleted by its author

  14. Anonymous Coward
    Gimp

    A Banshee scream decimates hard drives

    Crikey, and a baffle would only fix some of the problems.

    Probably should consider a shutdown though, to limit the damage. The old shutdown versus crash down issue had a solution, It was in the days of DOS, users would Ctrl + Alt + Del and reboot the system and the system would be fine, but if they hit the off switch or reset button it could kill the hard drive and data. The keyboard sent a signal to the hard drive that a reboot was imminent and to write out cached data, whereas the button press didn't do the same. but of course it could always have been designed to do so, if they bothered a tiny bit.

  15. DirtyJoeX

    "All we are is dust in the wind"

    https://www.youtube.com/watch?v=tH2w6Oxx0kQ

  16. Anonymous Coward
    Anonymous Coward

    We've got a few servers there and they suffered some disk failures. It took a while to get some info as to what was going on and figure out why some of our servers had failures and some didn't. As soon as we heard that the gas-based fire-supression system had kicked in then it all became clear (thanks to El Reg as I'm pretty sure I'd first heard about this phenomenon in the comments of some other story).

  17. Anonymous Coward
    Anonymous Coward

    Aha,

    The hold harmonic vibration phenomenon.

    I must say when I looked after a Halon equipped room, the noise of discharge was never alerted to me as a potential area of concern.

    I did however expect it to be loud, although the mandatory "pressure test" of the room seemed amateur and involve a lot of plastic sheets and a fan.

    I would have thought though that any resolution would simply involve reshaped nozzles appropriate to the density of the gas of choice would be the fix, and not beyond the wit of the manufacturers to come up with some that do not match the likely environment they are to be deployed in.

    Part of the point of the system is to protect the equipment from damage, not make a problem worse by ensuring most of the kit is destroyed regardless of fire or mildly smoking power supplies...

  18. Amos1

    How do they know it was the sound and not the smell?

    Perhaps it a former co-worker now works there. He could stop anything in its tracks with his gas discharge.

  19. allthecoolshortnamesweretaken

    I wonder whether the acoustics of the room itself play a part in this.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon