Tuesday, September 8, 2015

Apollo 6 and the wiring fuckup

About two years ago I had the pleasure of visiting the Johnson Space Center and managed to sneak into the Saturn V exhibit when nobody was around.  Now I mean that literally, the exhibit was completely empty.

After spending some time looking at the massive F-1 engines (scale is so difficult to convey, but imagine a tanker big rig, and the width of the engine bell is wider than that.)  The first stage is so massive it defies modern thinking.  We still haven't built something so large since.  It's a marvel sitting on it's side.  One can literally only imagine how it looked for it's all-up testing and launching.  It is a massive monstrosity.  It was also designed in the 60s and had incredibly tight tolerances.

Every system manager was asked to "make it lighter."  Fueled and on the pad, the vehicle weighed almost 3 thousand metric tons.  All of that mass to lift 120 tons to orbit.  Less than 4% seems like such a minuscule amount when it comes down to it.  But in 1960 slide rules and gray-matter brought men to the stars.  The Apollo program was no exception.

After having snuck into the Saturn V exhibit (it wasn't locked, and it was a Sunday afternoon) I found myself standing in front of the second stage, looking into the interstage.  It was massive, and amazing.  At each tier of the rocket, it shrank slightly and the second stage was still seemingly massive.  Inside the interstage were huge copper connectors, oxidized from exposure to the Houston weather for decades.

Connected with small dollops of solder were individual wires.  The wires are unwrapped (you can see them today) and ran along the most economical route in the interstage to their corresponding terminal.  But there are hundreds of them.  One familiar with electrical engineering can see, and easily understand how Apollo 6 happened.  The hundreds of wires individually ran would each require a human connect and isolate each terminal.

The massive undertaking would have to be repeated at every interconnection and then again at every component that needed to be telemetered, or controlled.  Thousands of terminals would need to be connected correctly and one simple mislabeling or miswiring would easily have accounted for Apollo 6's unfortunate Engine.  From teh wiki "Unfortunately, the command signals for engine three were partially cross-wired with engine two, so that the shutdown of engine two caused a liquid oxygen valve for engine three to close, resulting in a shutdown of that engine as well." 

It's easy to see why this was the assumption.  It would have been easy, even under immense scrutiny, to mistakenly wire one engine as another.  



I stood in front of the open interstage in awe, the massive effort, the hundreds of thousands of man hours, and all even then a simple human fuckup nearly caused a flight failure.
Overall, outside the aerospace community most people don't realize that Apollo 6 was an almost complete fuckup.  It didn't have a Hollywood Apollo 13 rendition, or even a discovery channel review of near misses.  It didn't rate like the shuttle repair of the Hubble, or like the moon landings. 

Nobody remembers the near misses that still succeed, as misses.  They're successes.  Engineers will still suffer over the failure, make the system better overall, and attempt to prevent the same thing from happening again.  Which is exactly what engineers do, but we should recognize this as what it was.   A near hit that superb engineering had nonetheless designed around the entirely improbable, but nonetheless very realized, eventuality.

Gazing at this massive racetrack of individualy isolated 20mm wires I still revel in amazement at the complexity of the design.  Next time I'm there I'll see if I can get a picture inside the interstage, it's remarkably unremarkable, but I still think worth documenting for historical purposes.