nav search
Data Centre Software Security DevOps Business Personal Tech Science Emergent Tech Bootnotes
BOFH
Lectures

back to article
Roscosmos: An assembly error doomed our Soyuz, but we promise it won't happen again

Aladdin Sane Silver badge

I can't get the sensor to fit

Hit it until it does.

Big John Silver badge

Re: I can't get the sensor to fit

The builders must know (don't they?) that this sensor is absolutely critical, meaning a failure of any one of them means a catastrophe. Yet this one went in "bent." Clearly the Russians have a systematic quality assurance problem, and those ain't cheap.

Voyna i Mor Silver badge

Re: I can't get the sensor to fit

I'm certainly not going to disagree that by 2018 standards Soyuz QA clearly sucks equine testicles. (In 40 000 miles my car has suffered zero faults, zero advisories, and the Toyota service manager says "we expect that".)

But it is worth remembering that the fire which killed three Apollo astronauts was caused by a dreadful error chain, starting with a contractor stuffing wires through ducts so badly that insulation was stripped in places (IIRC). How did anybody, knowing that people were going to travel on that thing, do that?

I recall being told that one reason so few existing car industry workers were taken on by the Japanese companies in the UK was that years of bad management attitude had meant that they simply didn't have the mindset for defect free assembly.

tl;dr: perhaps the space industry has to go the way the car industry did with the new entrants setting new standards, because once people have been kicked long enough to just get stuff out of the door, untraining them is impossibly long and expensive.

lglethal Silver badge

Re: I can't get the sensor to fit

It's all well and good to say that but if you do any job a hundred times it stops being a special job, it just becomes step 7b of job 256. And you've got 4 more jobs to do before the end of the day or the boss is going to get pissy with you. And this bloody bracket, which you've complained about half a dozen times but no one listens, is not going in easily. So you just take your hammer and get it to fit easy as pie. And move on to step 7c.

This is how cock ups like this happen. Because people are more worried about the pressure they'll get from managers for missing a deadline. Their manager is a more immediate problem then some future potential problem for some astronauts they'll never meet.

To avoid this, you really need a culture in the workplace where problems and delays are dealt with properly and no blame is put on the worker when they occur. It sounds very much like Roscosmos does not currently have that culture...

Anonymous Coward
Anonymous Coward

Re: I can't get the sensor to fit

@Big John

You seem to have a naive expectancy that any assembly line worker would really think about the criticality of any component, or take more care because of that.

It might be the case that the sensor was simply dropped whilst being handled, and then put on anyway, because the alternative is for the worker to own up to damaging it, and suffer the consequences - which might be punitive.

I wonder what you would do in the circumstances?

Anonymous Coward
Anonymous Coward

Re: I can't get the sensor to fit

"It might be the case that the sensor was simply dropped whilst being handled, and then put on anyway, because the alternative is for the worker to own up to damaging it, and suffer the consequences - which might be punitive."

Exactly. It's cultural.

In Toyota or Sony, to name just two I know a bit about, the worker would admit to having dropped it, the sensor would be replaced properly, and the quality circle would address itself to ways of preventing sensors from being dropped and damaged.

An example: wall hung toilet bowls. These are prone to damage by being dropped. Unless you buy them from a certain German manufacturer who supplies them with the foam packing formed into an assembly jig which holds them secure and presents them to the wall at the right height.

Then there's the marine engine builder who supplies each of their engines with a beautifully shaped jig which simultaneously compresses the piston rings and slides the piston down the cylinder in a straight line, preventing ring damage. When you see that compared to the crudity of people trying to get ring compressors to line up over a piston, you appreciate real quality engineering.

I've worked for a company where the production director had to be fired to start developing a blame-free culture to get towards zero defect assembly.

Stuart Halliday

Re: I can't get the sensor to fit

When I worked for Ferranti, we had the usual Military and Space standards.

Amazing how many assemblers would botch their job to get a time bonus.

We as the prefinal testers would regularly see amazing errors that people would do.

The next stage from us was environmental testing. Then it was fitted into the Space craft or Fighter Jet.

I was in disbelief at one PCB that had a dozen 2" long component leads still attached to the board so that you couldn't close the carry case it came in.

The Quality Inspector who was supposed to check assembly before me said, "anyone can make a mistake". Yeah right a little mistake caused by the Assembler and you!

If the British Air force pilots or Ariane knew how many errors were spotted and fixed at the last moment, I doubt the Pilots would switch their planes/Rockets on!

boltar Silver badge

Re: I can't get the sensor to fit

"simply didn't have the mindset"

ITYM "lazy bolshevic bastards who wouldnt recognise a good days work if it whacked them in the balls with a copy of the Morning Star"

FTFY

asdf Silver badge

Re: I can't get the sensor to fit

>Exactly. It's cultural.

If only the Russians had a quality first mindset like the US then they too could have the F35 program as a shining example. Not a russian troll just disgusted with how greed turns everything to shit in the US and causes us to lose the ability to even engage in space travel. Think positive. Looking forward to the JWST launching in 2025.

Stuart Halliday

Re: I can't get the sensor to fit

I can only imagine what the ground crew of the recent Helicopter crash are going through.

Most crashes are caused by the very complex maintenance required on the main rotar motor not being carried out correctly.

ridley

Re: I can't get the sensor to fit

I am surprised that with a mission critical system there is only one sensor system.

Florida1920 Silver badge

Re: I can't get the sensor to fit

We as the prefinal testers would regularly see amazing errors that people would do.
My first new car was a 1969 Pontiac, which I kept for many years until it was destroyed by another car striking it while it was stopped. I was still driving it when the state I lived in instituted emissions tests. It failed because of excess hydrocarbon emissions. Compression test looked okay so let's look at the carburetor. Removed top cap on carburetor, then the dried-out gasket. In the top rim of the main housing was a _serious_ dent, where something pretty heavy had fallen on it, or the carb had been dropped before assembly. That carb never should have found its way into a car. The gasket had sealed the dent for many years, until it dried out. Put a little dab of gasket sealer on the dent and let it cure, then replaced only the gasket, and the car passed its emissions test well below the PPM limits for that year. I forget which company built that carb, but that wasn't the only quality problem with the car. The first month I had it a rear-window regulator let go and the glass disappeared inside the body. Then the latch that held the seat in place front-back let go. Buy American? Call me a traitor, but I don't need the drama.

Space vehicles undergo rather more stress than that carburetor, so even if a problem isn't immediately apparent, poor QA can result in catastrophe later on. As John Glenn famously noted, every part of an American space vehicle was built by the lowest bidder. The problem is, once the thing is assembled, it's impossible to know if something was damaged or mis-assembled. As someone else noted, you have to trust in the quality culture of the organizations responsible for making the beast.

Anonymous Coward
Anonymous Coward

Re: I can't get the sensor to fit

@Big John

"Clearly the Russians have a systematic quality assurance problem, and those ain't cheap"

Before this latest problem, the last time a Soyuz launcher needed to use its escape system was 1983.

<http://www.russianspaceweb.com/soyuz_sas.html>

A 35 year run of manned launch successes implies that the problem which occurred isn't systematic.

Clearly, there is some sort of quality management problem which needs to be dealt with. But given the long-term reliability of the Soyuz system, whatever the problem is, it isn't deeply embedded into the engineering or operational culture so isn't likely to be particularly difficult or expensive to deal with.

I do wonder about this point:

<http://www.russianspaceweb.com/soyuz-fg.html>

"As of September 2017, the two final Soyuz-FG rockets had been scheduled to fly in April and September 2020. However the Roskosmos leadership was pressing the industry to retire the FG variant as early as 2019 to cut costs and avoid problems with Russian security services, which put serious obstacles in obtaining avionics and associated technical assistance from Ukraine's Kommunar plant. As a result, at least one of previously planned launches of Soyuz-FG in 2020 could be performed in 2019, industry sources said."

- one might speculate that - just maybe - management has perhaps been pushing workers to get things done faster than usual, which could have contributed to the problem in this case. I don't suppose we'll find out about that one way or another.

Voland's right hand Silver badge

Re: I can't get the sensor to fit

Clearly the Russians have a systematic quality assurance

I would have given the poster of this the benefit of the doubt if not for:

1. It being posted by a known AI-synthesized Trump Clone Bot

2. It having a proper statistical basis in the actual data from the field. Do we like it or not, but based on actual launch statistics the old village bus beats any of the other manned flight programs by a very large margin. Shuttle - 135 launches, 14 dead bodies, zero successful landings under major failure conditions. Soyuz - 139 crewed launches, 4 dead bodies, 3+ successful landings under various failure and/or abort conditions with crew alive in all cases (not something the shuttle can claim).

martinusher Silver badge

Re: I can't get the sensor to fit

>When you see that compared to the crudity of people trying to get ring compressors to line up over a piston, you appreciate real quality engineering. The Japanese bikes were never that much better than the British ones, especially when they were both relatively new, but the attention to detail in the assembly is much more obvious with the Japanese one. (They Japanese also seemed to put 'working' as a key criteria for outsourced components such as electrical parts -- they like stuff to be cheap but they regard working as somewhat more important.)

When you compare the service manual for my old Triumph with that for a generic Japanese motorcycle then you immediately understand why the old bike leaks and has had periodic reliability problems (up to and including a broken connecting rod) while the Japanese bike(s) just run. Its not really the line workers that are to blame but the company culture, it sets the tone.

The good news with Soyuz is that the emergency escape system got a good workout and seems to be functioning. Its also impressive that they could find the problem by looking at the pile of bits of crushed and twisted metal (and the telemetry).

Voyna i Mor Silver badge

Re: I can't get the sensor to fit

"The Japanese bikes were never that much better than the British ones, especially when they were both relatively new, but the attention to detail in the assembly is much more obvious with the Japanese one."

I like to watch the old videos of engine assembly on Youtube, and one thing I've noticed on both a British and a Harley video is the con rods being allowed to knock against the crankcase mouth, something the guy who taught me how to do that stuff (PhD mechanical engineer) would have regarded with utter horror. You do not risk making even the smallest scratch or dent in a stressed part!

Video of assembly of small Honda engine - nothing is ever allowed to bump into anything else. And the workers are wearing white gloves (skin grease can corrode metal).

Voyna i Mor Silver badge

Re: I can't get the sensor to fit

"If only the Russians had a quality first mindset like the US"

I am not an American, though I have worked for US companies in several capacities.

Some US companies I've come across are as good as any in the world.

Some are crap. Some are in between.

But the problems of the F-35, as so often with military programs, are political.

My friendly consultant to the US government says that the pork barrel basically prevents things from happening but is essential, because it's the US equivalent of the welfare state - providing jobs in depressed parts of the country, either in the armed forces or in making arms. Every politician wants their share.

So unlike our own dear country where Labour has to support Trident because of where the jobs are.

Big John Silver badge

Re: I can't get the sensor to fit

> "I would have given the poster of this the benefit of the doubt if not for..."

I love you too, VRH. ;-/

Down not across

Re: I can't get the sensor to fit

I forget which company built that carb, but that wasn't the only quality problem with the car.

I seem to recall GM using Rochester carbs a lot. Of course many people replaced them with Holleys.

Sgt_Oddball Silver badge

Re: I can't get the sensor to fit

British cars were worse. My grandad once owned an Austin Princess. Upon having owned the car a number of years found himself needing to use the spare tire. Not so unusual....

When said tire was removed he noticed an odd lump. After poking it he found a nearly full pack of cigarettes under the spay paint. Bollocks to picking the things up. Just spray paint over ir

Sorry that handle is already taken. Silver badge
FAIL

Re: I can't get the sensor to fit

Clearly the Russians have a systematic quality assurance problem

So systematic that it caused a single failure.

Now if you'd mentioned the problem they had with defective gas generators being installed in RD-021x engines during 2015/16, which ultimately killed off the Proton rocket, you might have had a point. However, that was due to attempted cost-cutting, which is the kind of thing you expect to see in the west...

Rupert Fiennes

Re: I can't get the sensor to fit

Actually, if you look at the stats differently, you might say the US and USSR/Russia are much the same. Both had two flights kill all the astronauts on board. And the Shuttle had flights like STS27 where there was major heat shield damage that the vehicle survived. The Shuttle both carried cargo and astronauts into orbit, and was much more complicated than Soyuz, so it might be reasonable to say the US has the better safety culture.

pavel.petrman

Re: I can't get the sensor to fit

The last hundred years conditioned the whole country to be a QA nightmare, and it can't easily be conditioned away. I'm writing from a country with only about 40 years of "peace and socialism" under its belt, which was done away with thirty ears ago, and the ripples are still to be seen and dealt with today.

asdf Silver badge

Re: I can't get the sensor to fit

>But the problems of the F-35, as so often with military programs, are political.

Same with our space program which is why Russian safety culture is even the topic at hand.

Alan Brown Silver badge

Re: I can't get the sensor to fit

"It might be the case that the sensor was simply dropped whilst being handled, and then put on anyway, because the alternative is for the worker to own up to damaging it, and suffer the consequences - which might be punitive."

Quite likely. When you look into reports of soviet nuclear accidents it's clear that more than a few fatal doses of radiation poisoning were down to this kind of incident. The country might change name and politics but ingrained cultural training takes much longer to alter.

Alan Brown Silver badge

Re: I can't get the sensor to fit

> The Quality Inspector who was supposed to check assembly before me said, "anyone can make a mistake".

Quality Bru^Hitsh workmanship - and why "Made in the UK" became a warning label across the Commonwealth.

In the fields I worked, even in the late 1980s anything which came out of the UK was regarded with deep suspicion - and this was despite Marconi putting out some quite decent kit.

As for cars - I think the lesson might have finally sunk in when GM tried to reintroduce Vauxhall as a brand in New Zealand in the 1990s and only sold about 3 vehicles until they gave up and rebadged them as Opels. (When Japanese cars went to the same import duty as British cars in 1973, British cars went from 40% of the market to 3% in less than 12 months.)

"If the British Air force pilots or Ariane knew how many errors were spotted and fixed at the last moment"

I'd be far more worried about the botches that _didn't get spotted. One of the reasons for the "deep suspicion" mentioned above was the British love of sending pristine demonstrators, but shoddy sales items which broke down a lot and then stiffing the buyer on warranty support. Wining and dining the bosses might win orders for a while, but Ferranti and a few others lost major orders when staff threatened to strike if they had to work on any more substandard kit - it helped that the accountants pointed out that the "more expensive" Japanese or American kit had a total operating cost far lower than the UK version and generally worked first time (and on budget) instead of needing 2 years of tweaking before being ready.

It wasn't just poor workmanship either: Who on earth designs pieces of equipment that are meant to be opened up to be serviced but doesn't have a wiring harness allowing this to happen? You can guarantee that having to desolder 50+ joints to get the device apart and resolder them afterwards is going to shorten its life. It was obvious that in most cases someone had just taken a prototype and not bothered to make it ready for production/service life before firing it down the assembly line and we ended up making our own replacement looms for a lot of kit in these kinds of cases as it was cheaper than the wasted time dealing with the craptacular originals.

"When I was a lad" - people used to lambast us colonials for a "she'll be right" attitude, but it was pretty obvious that it was nothing to the attitudes in British manufacturing (mainly at management level). Britain joining the EU and dumping all its cozy trading arrangements with former colonies was one of the best favours it could have done those colonies in the long term. They've moved on, but it's clear that the Brexit crowd haven't and they're still convinced that Britain rules all, instead of being a minor northern European country with a far greater interest in getting on extremely well with its neighbours than cosying up to the rich kids over the Atlantic who've repeatedly shown that the "special relationship" is only special when it suits them.

(As for Russian stuff: Surprisingly, their high tech works well, their low tech is bullet proof and everything in between tends to be crap)

Nigel Campbell

Re: I can't get the sensor to fit

The underlying problem is a culture that encourages middle management to treat line workers like that and not escalate issues that need fixing. Organisations will get politicised to the point that people start behaving that way if you don't keep an eye on the culture and values. The hard part is to avoid letting carpetbaggers into your management team. Once you get one manager with that mentality the toxicity will spread to everyone that person is in a position to put pressure on. It affects culture - by allowing a culture of undue pressure on workers - and hiring practices - A's hire A's, B's hire C's.

Keeping a culture alive where people are able to work on complex tasks in a zero-defect manner is not a trivial undertaking, and the vast majority of management teams are not capable of doing it. Certainly, British hiring practices for managers favour politically savvy 'people manager' types who aren't inclined to rock the boat, and the trend in management culture seems to be moving more in that direction in pretty much all industrialised countries. It happened at NASA with the O-ring fiasco, it's happened at Toyota (coming out with brakes IIRC) and now it's happened at Roscosmos.

Tom Paine Silver badge

Re: I can't get the sensor to fit

Oooh, my cue to post the L5 ground loop audio from Eileen Collins's launch. Stay with it for the low LOX level premature SSME cut-off half a sec before the scheduled MECO, which IIRC was due to a big hole in a nozzle cooling circuit. (The auto cut-off is there because apparently cryogenic gas turbopumps tend to explode if suddenly fed vacuum when running.)

"Yikes!"

"Concur."

Oy, this makes the hair on the back of my neck stand up every time I watch it!

https://youtu.be/O9WjCyWq-iA

Context https://waynehale.wordpress.com/2014/10/26/sts-93-we-dont-need-any-more-of-those/

And there's a WP article of course.

Semtex451 Silver badge

"avoid a potential decrewing" sounds better than 'evacuation' I suppose.

My lexicon is constantly embiggened by reading El Reg

petur

rapid unscheduled decrewing ;)

Rameses Niblick the Third Kerplunk Kerplunk Whoops Where's My Thribble? Silver badge

My lexicon is constantly embiggened by reading El Reg

A perfectly cromulent reason to read The Register.

jockmcthingiemibobb

Wasn't The Morning Star the Crommulist rag?

boltar Silver badge

"Wasn't The Morning Star the Crommulist rag?"

Still is AFAIK. I can only assume the idiots who modded me down never lived through the 70s and have some sentimental view of salt-o-tha-earth union workers fighting against The Man. Truth was british car workers back in the day were short sighted, cretinous, lazy fools who would use strong arm tactics to get what they wanted no matter what the long term cost - which turned out to be their jobs in a lot of cases. A bit like the current RMT tbh.

Alister Silver badge

@boltar

I modded you down, because of your clear assumption that the behaviour of a Russian, (or Ukranian, or wherever) assembly line worker would bear any resemblance to that of the work-shy British factory workers, who had the privileges and comforts of living in the West.

IceC0ld Bronze badge

My lexicon is constantly embiggened by reading El Reg

===

and today mine has also been 'embiggened' :o)

boltar Silver badge

@alister

I wasn't even refering to russians. Perhaps learn to read first, then when you've done that look up "bolshevic". Unless you're suggesting all russians are communist extremists.

Tom 7 Silver badge

@boltar and unions

Unions could be shit in the 70s. Some of those strikes were almost as damaging as the British Management,

Brilliant idea by thatcher to stop the unions damaging business by closing those businesses.

strum Silver badge

>Truth was british car workers back in the day were short sighted, cretinous, lazy fools who would use strong arm tactics to get what they wanted no matter what the long term cost

I lived through the 70s, and recognise that propaganda from the right-wing press of the time. The real truth was that British management was crap - as we discovered when foreign management managed to produce decent cars - with British workers.

As Geoffrey Goodman later observed - you could always predict when there was going to be a wildcat strike; when the storage car parks filled up with unsold cars, some issue would be created, to annoy the workers enough to walk out (and allow some cars to be sold, to clear the storage parks).

Yet Another Anonymous coward Silver badge

The same people

That hammered a sensor into place because it wouldn't fit - because it had been carefully designed with locating pins stopping it being installed upside down ?

proton failure

Semtex451 Silver badge

Re: The same people

Reminds me of the time I watched in horror as a 'colleague' hammered a PCI card into an ISA slot.

Naturally she was promoted.

Aladdin Sane Silver badge

Re: The same people

Ah, the Dilbert principle.

DropBear Silver badge

Say what you want...

...I much prefer this kind of incident investigation to that other kind that digs itself into a mountain of reports and committees with all flight operations halted for half a decade. Besides, they allegedly do intend to take apart all assembled boosters again to make sure there are no further issues of this kind. It sure beats sitting grounded until you can design, test and certify your new ISO-compliant procedures for establishing management guidelines and decision strategies for drafting quality assurance protocols that will hopefully ensure this exact specific particular fault can never occur again.

I ain't Spartacus Gold badge
Flame

Re: Say what you want...

DropBear,

That's all very well - until it turns out that you've got larger problems than a quick investigation will turn up, and you end up killing your cosmonauts live on TV.

It's not exactly very reassuring to say, "we broke a vital component of our rocket at some point in manufacturing, but we don't know how or when. Still that's OK - because it passed all the checks - or there weren't any checks - or people aren't doing the checks properly. But seeing as we don't know which, we'll just invent some new procedures and hope they work.

Meantime our current Soyuz in orbit has a hole in it, that was subsequently crudely covered up with epoxy (presumably by the guilty employee). And that also got through inspection, assuming there was any) and we also still don't know how that happened, when that happened, or how it got through checking, or if it was checked.

Oh and some of our senior leadership decided to publicly accuse the personnel in orbit of doing it, without any evidence or even credible possibility. Because that's really reassuring that management culture is all about solving this problem, rather than just rushing back to full operations with the minimum of arse-covering neccessary.

Yes a quick report can be an opportunity to get back to operations with maximum efficiency. But it can also be an opportunity for a whitewash that's eventually going to get people killed.

You can't just blithely say, "we have a manufacturing quality control problem of unknown dimensions, but fuck-it we're just going to fly anyway."

If NASA have got any sense, they will run a mile from the next launch. I hope they don't get pressured into it because they'll be accused of cowardice for not flying on a rocket the Russians will. Unless they're getting more indications that this is being taken more seriously than it looks to be on the surface. I'll be delighted to be wrong - but I'm seriously worried.

Aladdin Sane Silver badge

Re: accused of cowardice

Tanya Kirbuk: Tell me, Dr. Floyd, what has happened to American bravery?

Dr. Heywood Floyd: [testy] It's alive and well, thank you. What happened to Russian common sense?

asdf Silver badge

Re: Say what you want...

NASA being unable to meet any kind of deadlines or cost estimates is why we are depending on our fr-enemies to get into space in the first place. As Neil deGrasse Tyson says our space program looks awesome if you reverse time's arrow. Thanks to shitty decisions made for decades our two choices are now no space flight or trust the Russians. Easy to poo poo their culture but the alternative is a jobs program ran by defense contractors with too many heads in the trough that delivers nothing these days but invoices for the taxpayers.

asdf Silver badge

Re: Say what you want...

Oh I forgot that bi-polar stoner Elon Musk is going to save our space program. Yay for free markets. What a complete and predictable charlie foxtrot.

Gene Cash Silver badge

Re: Say what you want...

> Yay for free markets

The American space program is most definitely NOT anything close to a free market. It is closely controlled by Congress and used as a political football.

For example, the current hoorah over launching American spysats on rockets powered by Russian engines. They'd be powered by American engines, except guess what? Congress cancelled the last FIVE of the NASA engine R&D programs. One of which (FASTRAC) was rescued by that bi-polar stoner Elon Musk and resurrected as Merlin

> NASA being unable to meet any kind of deadlines or cost estimates

See above. This is because Congress has cut the COTS crew program funding by 67% and the BEST thing about it is that the senator responsible for the cuts is now bitching it's behind schedule!

And when you have to punt deadlines and jerk contractors around, of course it all gets more expensive... and I end up footing the bill for all this bullshit, as a taxpayer.

Gene Cash Silver badge

Re: Say what you want...

Oh yeah, I forgot:

Those Russian engines launching American spysats?

Yeah, the design of those were licensed to us and supposed to be "made in America" except guess what?

Again, Congress denied the promised funding for that to happen and told ULA to go fuck itself.

asdf Silver badge

Re: Say what you want...

For the record yes Elon Musk is amazing and yes SpaceX is too but was on a rant and have to admit the guy hasn't been a rock of stability lately. Sarcasm aside he may still well be the US's best hope.

Also I am well aware of our recently history of failed political leadership especially in regards to science but then an article like this shows NASA shares some of the blame as well. ISS was also a mistake IMO.

https://spectrum.ieee.org/riskfactor/aerospace/space-flight/gao-warns-of-deteriorating-costs-and-schedules-in-nasas-major-project-portfolio

Tom Paine Silver badge

Re: Say what you want...

Looking around the world today, it appears to be wall-to-wall Mike Charlie Foxtrottery as far as the eye can see.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

The Register - Independent news and views for the tech community. Part of Situation Publishing