back to article Sensor failure led to Soyuz launch failure, says Roscosmos

A crew crisis at the International Space Station could be averted, with Russia's Roscosmos saying this month's Soyuz launch incident was caused by a sensor failure. According to Sputnik News, Roscosmos executive director Sergei Krikalev said the malfunction was in a sensor that detects separation of the booster's first and …

  1. Anonymous Coward
    Anonymous Coward

    spacecraft design

    Ok, we've ripped on the Soyuz and the Russian space program plenty around here, but consider how this launch unfolded: The first stage bumped into the second stage, which blew up said second stage, and the end result was the crew got shaken up a bit and ended up spending the next evening doing interviews from a couch (versus being vaporized over the tundra).

    Looking for the vodka icon to raise to the parties involved.

    1. Bubba Von Braun

      Re: spacecraft design

      Yes, its one the the arguments NASA is using the slow down crewed Dragon, can the fault detection system work fast enough. There was some doubt it would on Apollo, though it never got tested fortunately.

      This is the second time for the Soyuz, certainly worked well a pretty wild ride.

      BvB

      1. John Smith 19 Gold badge

        "can the fault detection system work fast enough. "

        Well give it was mostly hard wired but IIRC there was a very slot processor in the loop (sub 1 MHz) I'd guess a 200MHz ARM would be substantially faster, or a discrete ECL SSI logic chip design at 1GHz (it'll guzzle electricity during the minutes of launch with the escape system is usable, then shut down).

        1. Peter2 Silver badge

          Re: "can the fault detection system work fast enough. "

          Lest we forget, a 1MHz processor is running at 1,000,000 cycles per second, which even assuming an inefficient architecture that needs 10 cycles to process an instruction means that it's still capable of doing 100,000 calculations per second. That's plenty fast enough for doing pretty much anything when your programming in assembler.

          We now need the 200MHz processor to try and keep up with that 1MHz processor because the useful work has to pass through a hundred abstraction layers to get to the hardware.

          1. DCFusor
            Thumb Up

            Re: "can the fault detection system work fast enough. "

            @Peter - clearly most HAVE forgotten...

            It's nice to have the new fast stuff as you can save programmer time and basically use dumber developers since the tools that come with those abstractions are much more helpful and require a lot less ingenuity to get an issue debugged. But not only is all that unnecessary, it's bad.

            You'd much rather have a more experienced and competent developer who can do without all that, and who has the ingenuity to not only track down issues - building their own tools for those cases as required (down to toggle this wire if I got here with x data), and who can anticipate a bunch of corner cases the less advanced guy didn't think of up front when they are best handled.

            You can't really test those 9's in past the first 2 or so. They have to be there by design. Once there is a solid design, coding is the simple part, even in the more difficult to use languages and simpler homebrew opsys' on minimal hardware.

            1. Dave 32
              Flame

              Re: "can the fault detection system work fast enough. "

              Ah, yes, shades of the Arianne 501 launch failure. Numeric overflow->diagnostic/test/calibration code->turn rocket sideways->structural failure->whoopsie.

              Dave

          2. Dave 32

            Re: "can the fault detection system work fast enough. "

            One should remember the Saturn V Launch Vehicle Digital Computer (LVDC):

            "The LVDC was capable of executing 12190 instructions per second."

            https://en.wikipedia.org/wiki/Saturn_Launch_Vehicle_Digital_Computer#Hardware

            Dave

            P.S. I worked with a guy who's first job was programming the LVDC. When he retired, 40 years later, I printed out a copy of the LVDC manual and presented it to him.

            1. John Smith 19 Gold badge

              "The LVDC was capable of executing 12190 instructions per second."

              People often over estimate how much performance is needed to run LV's and their engines in steady state, with limited monitoring

              OTOH if you're doing turbine monitoring using FFT's of the vibration spectrum of the drive turbines to decide if you need to shut down rather than get away with a throttle down (developed, but not AFAIK flown, for the Shuttle) and do fault diagnosis on the sensors (to decide if you can trust their readings) you need something with a bit more "oomph" (In Shuttles a whole separate processor board in the box).

        2. Anonymous Coward
          Anonymous Coward

          Re: "can the fault detection system work fast enough. "

          "or a discrete ECL SSI logic chip design at 1GHz (it'll guzzle electricity during the minutes of launch with the escape system is usable, then shut down)"

          If it doesn't need to be rad hard that might be overkill. But yes, I once did an outline design for an ECL subsystem (bitslice) which would have run at about 200MHz, back in the days when 4MHz was a fast CPU. It needed to run for about 20 seconds max, after which it was redundant because either the incoming thing had been taken out, or you were dead and didn't care. And it was intended to run on dry batteries.

        3. Destroy All Monsters Silver badge

          Re: "can the fault detection system work fast enough. "

          What are you guys even talking about? Do you think a fault detection system has to run tensorflow computations?

          1. theblackhand

            Re: "can the fault detection system work fast enough. "

            "What are you guys even talking about? Do you think a fault detection system has to run tensorflow computations?"

            No....mine bitcoin.

            These rocket launches aren't cheap....

      2. John Robson Silver badge

        Re: spacecraft design

        The launch escape for Apollo got tested...

        " There was some doubt it would on Apollo, though it never got tested fortunately."

        Of course the test flight was a technical failure, in that the booster used failed before reaching the appropriate altitude... The abort launch escape system however performed flawlessly, detecting the breakup and pulling the command module away to a safe landing.

        Surely the absolutely best flight to have a booster failure on.

        1. John Robson Silver badge

          Re: spacecraft design

          Forgot to add the link:

          https://www.youtube.com/watch?v=AqeJzItldSQ

          The launch escape for Apollo got tested...

          " There was some doubt it would on Apollo, though it never got tested fortunately."

          Of course the test flight was a technical failure, in that the booster used failed before reaching the appropriate altitude... The abort launch escape system however performed flawlessly, detecting the breakup and pulling the command module away to a safe landing.

          Surely the absolutely best flight to have a booster failure on.

    2. ridley

      Re: spacecraft design

      That why it is best to put the crew in a spacecraft on top of the stack, not bolted to its side.

      NASA didn't want to put the shuttle on the side but it was a compromise in order to be able to meet the requirements set.

      1. Baldrickk

        Re: spacecraft design

        You have to consider the fuel tank and the shuttle itself as one item at launch. Given that you want the expensive engines back, they need to be connected to the lander. You also want the sources of drag (e.g. the wings and tail) as far back as possible.

        Given that the engines are used in the launch, and the whole point is to make as much of it usable as possible, you don't want the entire thing on top of a first stage that is disposed of each time you launch.

        The only viable place to put any booster rockets is on the side, like the side boosters of the falcon heavy or any other boosters used on any rocket in the past.

        The only alternative design I can really think of would be to mount the fuel tank right on top of the shuttle - in front of the nose.

        Of course that would mean that you would require long pipes to get the fuel to the engines, instead of a direct connection.

        I don't think there is really another alternative layout for a reusable rocket launched winged craft, unless you were to put a smaller winged pod on top of something like the Falcon Heavy or the BFR where you can land the first stage as well.

        At this point though, a good capsule like the Dragon is going to work out cheaper and lighter, which is why of course that's the route SpaceX is taking.

        The Shuttle was, as much as anything a product of the technical limitations of the time, as much as it was the technical abilities and the political wrangling that went on.

        1. IsJustabloke
          Trollface

          Re: spacecraft design

          "I don't think there is really another alternative layout for a reusable rocket launched winged craft"

          Well Gerry Anderson would disagree!

    3. Anonymous Coward
      Anonymous Coward

      Re: spacecraft design

      "The first stage bumped into the second stage, which blew up said second stage..."

      Not quite - the R7 series of launchers have a central 'core' section, that forms the 'first' stage, to which are attached four discrete boosters. Thrust from each of the four boosters is transferred to the 1st stage core by a 'ball' at the tip of each booster, which sits inside a 'cup' that is built in to the side of the upper section of the core. The base of each booster has a linkage to the core, to retain the booster and keep it in place alongside the core, but which transfers none of the thrust. This arrangement, that of transferring the thrust from the boosters to the core at the core's upper section and not at the base, means that the lower section of the core need only be strong enough to cope with the thrust from its own engines and not the entire thrust of the launcher.

      When the boosters have done their work and are to be jettisoned the motors are first shut down and then the lower links are relaxed, allowing the booster to swing out at its base and drop back a little and this allows the thrust transfer 'ball' to drop out of the 'cup' on the side of the upper section of the core. After the ball has dropped out of the cup, small thrusters built in to the tips of each of the boosters are then fired to swing the tips of the boosters out and away from the core, the boosters rotating around their base linkages as the tips swing out. Once the boosters have rotated far enough they disengage from their base linkages and fall away.

      The problem seems to have occurred when the booster tip thrusters were fired, with one of them failing to operate correctly and resulting in the tip of that booster re-contacting the side of the core, which at that point still had fuel and which was still running.

      The second stage sat on top of the core module of the first stage, with the Soyuz sitting on top of that.

      1. DCFusor
        Thumb Up

        Re: spacecraft design

        @Lee -+1 informative, thanks!

      2. Anonymous Coward
        Anonymous Coward

        Re: spacecraft design

        I need to reply to my own post here to add some corrections to the booster jettison sequence:

        The first stage of jettison is not the shutdown of the booster motors, as I previously stated, but the release of the lower linkages. Because the booster motors are still running this causes the base of the booster to swing out and away from the core - then the booster motors are shutdown, and then the tip thrusters are activated to swing the tips of the boosters clear of the core.

        It might also be considered a misnomer to describe the tip thrusters as such because the thrust is not provided by supplemental motors but by venting surplus oxidizer (oxygen) carried for this particular purpose.

        My bad.

        I can't down-vote myself so I'll have to leave that to others.

  2. ilmari

    Iirc, Apollo era fault detection, and probably Soyuz too, consists of a long piece of wire that runs up and down the length of the rocket. If the rocket goes boom, the wire is broken, which triggers abort.

    Then it's a question of how fast explosive bolts and the abort motor light up after being lit up. Hopefully fast enough that the fireball shockwave still hasn't reached the crew.

    1. Spazturtle Silver badge

      "Iirc, Apollo era fault detection, and probably Soyuz too, consists of a long piece of wire that runs up and down the length of the rocket. If the rocket goes boom, the wire is broken, which triggers abort."

      It uses multiple wires, and if more then N are severed the system activates. This prevents a single wire breaking erroneously from aborting an otherwise normal launch.

      I assume modern manned rockets will use a similar system, it is a reliable system that works at wire speed.

      1. Anonymous Coward
        Anonymous Coward

        so a program written in 10 minutes and loaded onto an arduino could handle it

        1. Anonymous Coward
          Anonymous Coward

          "an arduino could handle it"

          Much, much more powerful than anything that flew on Apollo.

    2. Baldrickk

      Then it's a question of how fast explosive bolts and the abort motor light up after being lit up. Hopefully fast enough that the fireball shockwave still hasn't reached the crew.

      They're designed to be fast and get the crew away, unsurprisingly.

      https://youtu.be/AqeJzItldSQ?t=100 apollo capsule escape system firing

      https://youtu.be/1_FXVjf46T8?t=9 spaceX dragon escape system test

      https://youtu.be/ESc_0MgmqOA?t=51 Blue Origin New shephard escape system test

    3. John Smith 19 Gold badge

      "Iirc, Apollo era fault detection, and probably Soyuz too, consists of a long piece of wire"

      Hilarious.

      Actually it was a bit more complex, as described here

      The trick is to not identify fault patterns, then install sensors to find them (which need wiring and processing, and might themselves be faulty), but find the signatures they cause

      Mostly viciously aggressive acceleration. The cause don't matter, just the results.

      Apollo also put the 'nauts in the loop to track slower developing possible problems (which might get bad enough to go to a full abort, but might just as easily decay back to nothing).

      It would seem a large part of the EDS was in fact hard wired, using diodes and (gasp) relay logic.

      Note that the basic trajectory to orbit (and how fast it's traversed) hasn't actually changed in the 5 decades since Apollo.

  3. Dave Bell

    I am a little bit unsure about just what the first stage and second stage are on the Soyuz. The basic design goes all the way back to the first Sputnik launch, which five units all firing together at launch: the four boosters and the core. When the boosters have burnt all their fuel they detach, leaving the core still burning.

    The pictures show one of the four boosters having a bad separation.

    If one control module failed, out of four, it all makes a lot on sense, if what I am thinking of as booster separation is what is been called first stage to second stage.

    1. Anonymous Coward
      Anonymous Coward

      Diagrams online such as this one mostly indeed seem to show the four boosters as "Stage 1".

      Yes, I know I picked a diagram of a three stage variant, but it makes the point about the first stage more clearly than most.

  4. Voland's right hand Silver badge

    They just changed it this morning

    Up to yesterday, it was sensor malfunction which was described in more detail as a snag of the sensor rod on the 2nd stage preventing separation and resulting in the first stage pivoting onto the 2nd stage fuel tank (with a following rupture).

    As of this morning, it is lid failure on the separation engine. The first stage has small engines which guide them into the Korolev's cross positions after the explosive bolts have performed the initial separation.

    That sounds more plausible than sensor failure - I am having some difficulty to imagine a sensor rod thick enough to snag something the size of the first stage against the forces exerted on it upon separation.

    So actually, el reg info is a bit out of date (same as most other media).

    1. phuzz Silver badge
      Meh

      Re: They just changed it this morning

      I'll bow to the knowledge of someone who actually reads Russian, but according to Google translate, your link says:

      “We are talking about damage to the end sensor as a result of unintended, but erroneous negligent actions of one or several Progress RCC specialists in the assembly and test building at the 112th site of the cosmodrome when assembling a package consisting of side blocks of the first stage and the central unit of the second stage. there could be a one-time single violation of the technological process, "said the source.

      As a result, when this side unit was detached, the jet nozzle cover jammed to remove the side unit from the central unit.

      ie the sensor failed, so the signal that the booster was detached was never sent, so the 'jet' (not an engine, it's a vent at the top of the O2 tank) that pushes the nose of the booster away from the core never operated.

      The sensor didn't "snag" the first stage. It failed to operate because it was bent, so the necessary signal was not sent.

      Of course, that all depends on your translation of that second and third paragraphs in the linked article, I'm not sure how many elReg commentators are bilingual in Russian?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like