26

What precautions are planned to prevent samples returned from Mars crashing and releasing organisms on Earth? and this answer to Why would bringing samples from Mars back to Earth be a “civilization-level changing capability”? cite the crash landing on Earth of the return capsule from the Genesis mission.

Despite the crash the mission was a success in that solar wind particles embedded in special materials were not seriously contaminated and could still be analyzed such that all success criteria were met.

That Wikipedia article links to this archived version of Space.com's Genesis Update which says in part:

Michael G. Ryschkewitsch, who heads a board investigating the mishap, said the planned test would have spotted the fatal flaw and prevented the crash.

"If the test had been done, the problem would have been caught," said Ryschkewitsch, the deputy director of NASA's Goddard Space Flight Center.

Officials at Lockheed Martin do not dispute that conducting the pre-launch test would have uncovered the error that doomed the $264 million missions.

Question: Exactly what was the nature of the test that was skipped? Was it simply a sufficiently long period of acceleration at 3 g to close the contacts or did it rise to the 30 g maximum anticipated? How was the acceleration of sufficient magnitude and duration going to be generated?

"bonus points" for information addressing why there wasn't a visual cue for the proper direction; wasn't there an arrow or something printed right on the G-switch?

Genesis crash site scenery

Wikimedia: "Genesis crash site scenery"

uhoh
  • 148,791
  • 53
  • 476
  • 1,473

1 Answers1

58

The planned test was a centrifuge test. They were going to take the entry vehicle up past the 10-g mark and back down. According to the JPL Mission Manager, who was my boss at the time, the g-switch was supposed to come off the peg at 3 g's, saturate at 10 g's which told the deployment controller to enable the deployment sequence start (i.e., to "arm" it), and when it got back down to 3 g's, "Let'er rip!", starting the deployment sequence that eventually would've fired the pyros to deploy the parafoil. Of course in the test the controller would've had the actual deployment sequence disabled, but we'd have seen the signals from the switch at those g levels, indicating all was working as planned. With the upside-down g-switch, we'd have gotten flatline signals from the g-switch and the problem would've been brought to light.

But the project was under budget pressure (chewing farther into reserves than anticipated) and schedule pressure: the centrifuge schedule was becoming a problem due to a backlog of tasks for the centrifuge facility. Management then said, "This accelerometer/controller system is the same as the one on the Stardust spacecraft, and that system passed its centrifuge test. We're declaring heritage for that system so we'll skip the test."

They didn't tell us that, though all the components of the system were identical, to fit into the Genesis spacecraft they all had to be mounted on a board of different shape so the mounting configuration was different! Had we at JPL known that, the heritage claim would've gone out the window.

I can't verify it, but months after the crash the Genesis Project Mission Assurance lead (the late Robert Axsom) told me that indeed the g-switch was labeled for proper orientation — but the way the blueprints showed it, it was upside-down!

And yes, they are recovering much of the science originally proposed, but the cleaning procedures and analysis procedures are more laborious and expensive.

When the crash occurred, on camera for everyone in the JPL mission control room in Building 230 to see, I was sitting next to the Director of JPL, Charles Elachi, and the NASA Deputy Associate Administrator for Planetary Science. I'm really glad that when the crash occurred they didn't both glare at me!

Tom Spilker
  • 18,306
  • 1
  • 66
  • 83
  • 11
    They were probably already too busy imagining themselves being glared at by some congressional sub-committee, review board, or whatnot. Thank you for the thorough answer! – uhoh Dec 16 '20 at 08:23
  • 17
    I imagine that the management who ordered the test to be skipped got promoted, as per standard US government culture. – Ian Kemp Dec 16 '20 at 12:53
  • 1
    Ah, so it wasn't the contractor who decided to skip it. I was surprised by your comment about that. Most NASA contractors can't make that kind of call. – Organic Marble Dec 16 '20 at 14:59
  • 28
    These first-person experiences and recollections from people who were actually involved with the events surrounding these Space Exploration questions is why I love StackExchange! – Milwrdfan Dec 16 '20 at 15:27
  • 11
    @OrganicMarble Actually, it was the contractor — in the Discovery Program lingo, called a "partner" — who proposed eliminating the test. But the managing partner, JPL, had to approve of it. Not aware that the controller hardware had been reconfigured, that approval was given. – Tom Spilker Dec 16 '20 at 16:31
  • Did these people never think of adding a small pin and a slot so that the module could not be inserted backwards? This IS rocket science, right? With real rocket scientists? – ErikE Dec 17 '20 at 01:42
  • 10
    @ErikE It wasn't a matter of a "module" (actually a mounting board) being inserted upside down; the board was correctly installed. The g-switch, a relatively small, non-descript (from the outside) component with some wires coming out of it, was mounted on the board and connected the way the blueprints showed it being mounted and connected — but the orientation of that component shown on the blueprint was upside down on the board! Whomever initially drew that blueprint diagram goofed — and how! And ... whomever checked that blueprint and OK'd it (if that actually happened) goofed — and how! – Tom Spilker Dec 17 '20 at 04:33
  • 2
    The "component being installed upside-down" problem is, sadly, neither new nor unique to NASA. Russia has experienced it and the most recent European Vega launch that failed was also due to this. – Ian Kemp Dec 17 '20 at 13:53
  • 3
    @TomSpilker The fact that the blueprints were wrong is pretty shocking, but also a good demonstration of why you should never skimp on actual physical testing in such a critical application as this. In conjunction with the recent Starliner "mishap", it makes me think that NASA has not learned from either Challenger or Columbia - yes there are budgetary pressures, but everyone knows that when push comes to shove no Senator is going to be held accountable for withholding funds for testing. – Ian Kemp Dec 17 '20 at 13:58
  • 2
    @TomSpilker Thanks for clarifying! It still seems to me that some kind of engineering is possible that would make incorrect connecting or mounting of components harder to do. – ErikE Dec 17 '20 at 21:08
  • 1
    @ErikE I agree wholeheartedly! But I'm reminded of this joke, originally about software engineering but applicable to just about any technical endeavor: "Modern software engineering is a race between software engineers, that are trying to build better and better idiot-proof software, and the universe, that is trying to build better and better idiots. So far the universe is winning." – Tom Spilker Dec 17 '20 at 21:21
  • @IanKemp re Vega: Is it common and good engineering for a pair of cables to be easily plugged into each other's connectors in modern spacecraft, or... – uhoh Dec 17 '20 at 23:18
  • @ErikE see above – uhoh Dec 17 '20 at 23:20
  • In the news today: NASA to skip repair of Orion electronics unit and NASA will fly Orion deep-space crew capsule ‘as is,’ despite failed component – uhoh Dec 19 '20 at 05:32
  • 1
    @uhoh To plagiarize your tag: uhoh. I'm reminded of a sentence uttered by someone in NASA or Morton Thiokol management, "Let's put our management hats on," — just before the Space Shuttle Challenger disaster. This flight won't involve a crew, but if they get away with it in this flight it might embolden them for future crewed flights. – Tom Spilker Dec 19 '20 at 17:07
  • @TomSpilker potentially of interest in meta: How extensively should we interrogate/pursue post authors about suspected ulterior motives or suspected thoughts or beliefs? general topic is questions and answers about backwards and forwards biological contamination. – uhoh Jul 29 '21 at 01:35
  • This is Murphy's Law in action. I've been hit by it myself. The circuitry for a device I worked on had been wired exactly backwards. Fortunately we found the problem prior to deployment. The solution was to peel off the on/off label and reapply it upside down. Amazingly, that simple solution worked. – David Hammen Jul 29 '21 at 11:24
  • 1
    @DavidHammen Thanks so much for your comment! I think it helps for people working on space flight hardware and its design to see that they can't just assume, "Aww, nobody would do that!" because indeed somebody can "do that". Real-life stories such as these reinforce that idea, and I think hearing them makes better engineers. – Tom Spilker Jul 31 '21 at 18:58
  • @TomSpilker Murphy's Law says somebody will do that. – David Hammen Aug 01 '21 at 02:15
  • And don't forget about software. The only way well-designed thrusters can fail on is if the software mistakenly commands the thrusters to be on. That apparently is exactly what happened with the Nauka that recently docked with the ISS. Trying to be born to be wild, the Nauka fired all its guns at once and almost made the ISS explode into space. – David Hammen Aug 01 '21 at 10:21
  • @DavidHammen Yes indeed! 50 years ago spacecraft avionics were simple enough that you could send the software a chain of "vectors" that covered every possible combination of input values for every piece of software on the spacecraft, and that uncovered bugs not previously found. Now that's not feasible, with the number of possible combinations going up as the factorial of the number of parameters and number of possible values for each. Software engineers have to test their own code, and then a limited test program checks for the interactions among various code packages. Thus: $#!+ happens! – Tom Spilker Aug 04 '21 at 18:14
  • @TomSpilker It sure does happen, and waterfall doesn't work. Agile is our only hope, and I'm not sure that works. Nothing works perfectly. – David Hammen Aug 04 '21 at 18:27