V. A. Samaranayake
Flight controllers here looking very carefully at the situation. Obviously a major malfunction.â The words of Steve Nesbitt, the public affairs officer at Houston, are etched indelibly in all of us who watched the launch of the space shuttle Challenger on the morning of 28 January 1986. Ironically, the phrase âmajor malfunctionâ aptly describes not only the physical disintegration of the shuttle, but also the managerial and engineering missteps that led to this tragedy, taking the lives of seven astronauts, including the high school teacher Christa McAuliffe. Fingers have been pointed at faulty engineering design, poor communication, conflicting engineering and managerial goals, the culture at NASA, as well as political pressure. But to a statistician, the lack of sound statistical reasoning and the failure to use appropriate data and visual aides to convince a set of sceptical administrators â especially at the crucial juncture of determining launch risk â seem to constitute an equally serious misstep. Engineers at Morton Thiokol, the solid rocket booster (SRB) manufacturer in Utah, had concerns about the field joints on the boosters (also referred to as the solid rocket motors) for many years.1 It is one of these joints that failed during the Challenger launch, setting off a cascading series of events that led to the destruction of the shuttle. To the casual observer, an SRB may seem to be a single metal cylinder running the whole length of the booster, but it consists of four connected segments each of which is more than 25 feet long. Tang-and-clevis connections join the segments (see Figure 1). The connections, referred to as field joints, are then locked using 177 steel pins.2 To prevent hot gasses from escaping through the joints during the ignition of the solid fuel inside the boosters, engineers placed two O-rings made of synthetic rubber in machined grooves on the inside surface of the clevis. Flame-proof zinc chromate putty sitting between the rocket fuel and the O-rings acted as a pressure transmission mechanism during ignition, which helped extrude the O-rings into the grooves they were sitting in, forcing the rubber to seal any gap between the metal surfaces.1 The secondary O-ring was supposed to take over if the primary ring failed. Illustration of the field joint connection. But there was a potential problem. Engineers noted that ignition of the solid rocket caused the shell of the boosters to distort under pressure, triggering what was labelled âjoint rotationâ, creating a momentary gap to open near the O-ring slots. This caused the primary O-ring to extrude into the gap and the secondary O-ring to âbecome completely disengaged from its sealing surface on the tangâ.3 Thus, in practice there was no redundancy in the system, and hence the field joints were reassigned from Criticality 1R (R designating redundancy) to Criticality 1, meaning that the failure of the joint would result in loss of life or vehicle.3 But the continued belief in this redundancy by decision-makers at NASA proved disastrous. There was another critical issue, initially not given much thought. Morton Thiokol engineers banked on the elastic ability of the O-rings to fill any gaps, providing a proper seal. It was common knowledge that low temperatures can make synthetic rubber harder.4 This could limit the O-ringâs ability to deform from the original circular cross-sectional shape into a configuration necessary to provide a proper seal, but the engineers claimed that they had no real data on this.2 The first space shuttle flight, named âSpace Transport System 1â (STS-1) was launched on 12 April 1981. Between then and Challenger, 22 shuttle flights took place without any outwardly troubling incidents, but post-flight inspection of recovered booster rockets revealed 10 field joint O-rings with erosion due to hot gasses.3 In fact, the primary O-ring on a field joint of the second shuttle flight, STS-2, showed a 0.053-inch-deep erosion.3 Note that the cross-section diameter of the O-ring is 0.272 inches.4 It was, however, flight STS-41B in February 1984 that prompted engineers to file a problem report, but they went on to state that recent tests indicated that the secondary O-ring would provide âadequate backupâ.3 This assessment was questioned by Keith Coates at the Marshall Space Flight Center in a memorandum to George Hardy, the deputy director of engineering at Marshall. Hardy eventually played a role in the eleventh-hour teleconference that led to the decision to launch Challenger. To their credit, Morton Thiokol engineers conducted a series of experiments to determine if the O-rings with a 0.095-inch erosion depth (0.005 inches greater than the maximum possible erosion limit of 0.09 inches they arrived at by computations), would still seal properly under various pressures. They found them to do so even at pressures of 3,000 psi and concluded that âthis [O-ring erosion] is not a constraint to future launchesâ (tinyurl.com/uj5x5p5t). Their results led them to believe that joint erosion was an âacceptable riskâ because 0.053 inches is well within their upper limit of 0.09 inches.2 It appears that engineers failed to consider the random nature of erosion and that the probability of it exceeding the 0.09-inch limit may not be zero or negligible. In other words, there is no evidence that the engineers accounted for the inherent variability in the data in a statistically valid fashion when estimating the risk of failure. The Challenger (STS-51-L) flight was scheduled to be the first in 1986, but due to delays associated with many factors and the last-minute scrubbing on 27 January the launch was finally rescheduled for 9:38 a.m. Eastern Time on 28 January.2 This delay placed the launch on a day with a forecast of temperatures in the low 20s Fahrenheit during early morning hours and 26°F at launch time.1,4 Morton Thiokol engineers and others were concerned about the cold-weather launch, especially because of their experience with an earlier space shuttle launch in January 1985. The calculated O-ring temperature on that space shuttle (STS-51-C) was 53°F at launch, the lowest temperature encountered by the shuttle up to that time.3 On recovery, the booster rockets showed both impingement and âblow-byâ erosions. The former is caused by hot gases impinging on an already sealed O-ring, while the latter occurs when it has not sealed, a much more serious event where hot gases âblow byâ the primary O-ring and impinge on the secondary one. This experience in 1985 was the first-time engineers observed evidence of the hot gasses penetrating the primary O-ring, with blackened grease found between the two O-rings.3 Even though they were very concerned about the 1985 event and concluded that âlow temperatures enhanced the probability of blow-by,â they went on to state that, for Challenger one year later, âthe condition is not desirable but is acceptableâ.3 This may be because they did not expect Florida weather to get that cold often and their belief in apparent redundancy due to the inclusion of the secondary O-ring. In spite of this âacceptableâ recommendation, erosion was a serious concern to Thiokol engineers especially because Challenger was to be launched at temperatures 30°F below that of STS-51-C. Having sat in cold weather for many hours, they calculated that Challengerâs O-rings would be at 29°F at 9 a.m., with gradual warming to 38°F by 2 p.m.1 Among the Morton Thiokol engineers who had very serious worries about the potential for a disastrous failure of the field joints were Allan McDonald, Roger Boisjoly, Robert Ebeling, and Arnold (Arnie) Thomson. They all played key roles in trying to persuade the decision-makers not to launch, but ultimately failed. At a teleconference held at 5:45 pm the day before the launch, Morton Thiokol engineers recommended delaying the launch until later in the day. But since not everyone was able to participate and conduct a thorough discussion, a second teleconference, between Morton Thiokol engineers, managers, as well as those representing the Kennedy Space Center in Florida and the Marshall Space Flight Center in Huntsville, Alabama, was scheduled for 8:15 p.m.2 Morton Thiokol engineers prepared 13 charts, mostly handwritten, to be presented at this teleconference. One addressed the temperature effect on sealing abilities and another focused on the effect of temperature on the O-ring material.4 A key chart that led to a misleading conclusion listed cases of erosion in both field and nozzle joints, the latter referring to the joint at the nozzle of the booster rocket.4 The intent of the Morton Thiokol engineers was to point out that the âworst case [O-ring] condition ever observedâ occurred in shuttle STS-51-C in 1985, which was launched under the coldest conditions ever. But this strategy seemingly backfired, because shuttle STS-61-A â the ninth and last successful flight of the space shuttle Challenger before the disaster, which had the only other serious blow-by erosion, was launched during 78°F weather, with a calculated O-ring temperature of 75°F. Figure 2 illustrates the O-ring distress data presented by Thiokol engineers in graphical form rather than in the list form used by Thiokol. O-ring thermal distress as a function of temperature.3 Note that each launch can have more than one incident because of multiple field joints. It is not surprising that Larry Mulloy, the manager of the Space Shuttle Solid Rocket Booster Program at Marshall Space Flight Center, retorted: âHow then can you conclude that temperature has anything to do with blowby?â4 After all, blow-by, the more serious type of erosion, occurred on both the hottest and the coldest rocket motors. Boisjoly tried to explain that the blow-by erosions seen in STS-51-C and STS-61-A were qualitatively different. The former had primary O-ring erosion and jet-black soot over the secondary O-ring, while in the latter there was no primary O-ring erosion and only light grey coating of the secondary O-ring.4 But the damage was already done. Morton Thiokol engineersâ recommendation (given at this second meeting) that no launch should be made below 53°F was met with derision. Mulloy exclaimed: âMy God, Thiokol, when do you want me to launch, next April?â2 Mulloy went on to say that all the data when considered together does not appear to show any correlation between temperature and erosion and, moreover, if the cold temperature âslowedâ the seating of the primary, the secondary O-ring âwould seal the jointâ.2 George Hardy added to this pressure to launch, saying he was âappalledâ by Morton Thiokolâs recommendation.3 In fairness to Hardy, he did state that no launch should be made without Thiokolâs concurrence.4 The data set from which Figure 2 is derived is not the only one that mislead Mulloy. Engineers also presented a chart that showed no leakage past O-ring seals at 30°F and at 78°F using data obtained from a bench test done on joints at less than full scale with boosters mounted horizontally. Sealing pressure in this test was applied using inert argon gas, which is very difficult to detect for leakage.4 Only two data points were presented, one at each temperature.2 A respectable statistician may be hesitant to make a conclusion based on just two data points. Moreover, the data came from a test that did not reflect real field conditions. But apparently the NASA hierarchy was influenced by such data. Seeing no agreement, Morton Thiokol manager Joe Kilminster called for a five-minute caucus which lasted for almost 30 minutes. Morton Thiokol managers, ignoring the concerns of their engineers, then voted to recommend the launch. Boisjoly and Thomson repeated their arguments to the managers to no avail. This decision, of course, resulted in a very sad ending. As reported in his New York Times obituary, engineer Robert Ebeling told his daughter, âThe Challenger is going to blow up. Everyone is going to dieâ, before leaving to watch the launch.5 He fully understood the precarious nature of the decision NASA had taken. What stopped Ebeling and others from effectively conveying this sense of danger to the decision-makers such as Mulloy and Hardy? As engineers intimately involved with the design and manufacture of the SRBs, Ebeling, Boisjoly, and others understood the inherent danger of launching in cold weather. Why, then, did they fail to impress upon their superiors what they knew in their bones to be true? Ben Powers at Marshall Space Center had this to say after the teleconference: âI donât believe they did a convincing job of presenting their dataâ.2 He is definitely correct. The report of the Presidential Commission (also known as the Rogers Commission) appointed to investigate the Challenger disaster contrasts Figure 2, which summarises the data shown by Morton Thiokol engineers, to Figure 3 which encompasses the history of all flights, not just the ones with erosion. It clearly shows that the vast majority of launches with O-ring temperatures above 65°F had no erosion and all launches above 75°F produced no erosion. In contrast, all launches below 65°F had one or more erosion incidences. You do not have to be a trained statistician to recognise this. Clearly correlation between temperature and O-ring erosion exists, as opposed to what Mulloy thought. Even NASA management, who were very keen to keep a steady schedule of launches, may have been swayed by this figure, at least to the extent of postponing the launch until the afternoon of 28 January, when the temperature was forecast to rise well above freezing, or wait for a much warmer day, which is not uncommon for Florida even in January. The history of all flights, not just those with erosion incidences, showing erosion in all launches below 65°F. At the end of their investigation the Rogers Commission stated several findings. A finding that reveals the inadequacy of NASAâs statistical analyses states: âA careful analysis of the flight history of O-ring performance would have revealed the correlation of O-ring damage and low temperature. Neither NASA nor Thiokol carried out such an analysis; consequently, they were unprepared to properly evaluate the risks of launching the 51-L mission in conditions more extreme than they had encountered beforeâ.3 Another finding, based on commissioner Richard Feynmanâs comments, was: âNASA and Thiokol accepted escalating risk apparently because they âgot away with it last time.â ⌠âa kind of Russian roulette.â [The Shuttle] flies [with O-ring erosion] and nothing happens. Then it is suggested, therefore, that the risk is no longer so high for the next flights. We can lower our standards a little bit because we got away with it last time. ⌠You got away with it but it shouldnât be done over and over again like that.â This is simple probabilistic reasoning: if the probability of success in a given launch is 0.95, then, assuming each launch to be an independent event, the probability you will succeed in all 24 consecutive launches is a very low 0.292! As was implied earlier in this article, lack of formal risk assessment seemed to be an obvious deficiency within NASA. The United States House of Representatives Committee on Science and Technologyâs report on the Challenger accident agrees, stating: âIn concurrence with the Rogers Commission, the Committee confirms that the safety, reliability, and quality assurance programs within NASA were grossly inadequate, but in addition recommends that NASA review its risk management activities to define a complete risk management program.â6 Apart from the criticisms levelled by the Rogers Commission and the House Committee, others have also taken issue with the data on which NASAâs decision to launch Challenger was made. Edward Tufte, in his book Visual Explanations, refers to âa scandalous discrepancy between the intellectual tasks at hand and the images created to serve those tasks. As analytical graphics, the displays failed to reveal a risk that was in fact present. As presentation graphics, the displays failed to persuade government officials that a cold-weather launch might be dangerous.â7 Fredrick Lighthall at the University of Chicago, who conducted an examination of the circumstances related to the Challenger disaster, makes some scathing criticisms.8 One such criticism is that not a single chart prepared by Morton Thiokol engineers for the teleconference presented O-ring anomaly counts and temperature in relation to one another. In his conclusion Lighthall states that the âdata presented in chart form during the teleconference were essentially irrelevant to the causal question that Thiokol engineers were attempting to answerâ. He goes on to say that that ânone of the participants had ever learned, or had long since forgotten, elementary ideas and methods of statistical analysis and inferenceâ. Furthermore, âthese failures of thought and perception were not from a lack of sophisticated expertise but from lack of simple, elementary understandings and methodsâ. It is unfortunate that the âlack of simple, elementary understandings and methodsâ produced a deadly outcome and tarnished the recognition that hard-working NASA and Morton Thiokol engineers should have received for making the space shuttle a reality. As Malcolm McConnell, the author of the book Challenger: A Major Malfunction,1 states: âthe rank-and-file people of NASA are among the hardest working, most productive, and most talented employees in the federal governmentâ. Yet their failure to use simple visualisation tools to present the full story about the link between temperature and O-ring erosion resulted in tarnished reputations, not to mention the tragic ending of seven lives. There is a hard-learned lesson in this. For sustained success, high-technological achievements should be coupled with data-driven risk assessment of these technologies, and then the resulting information must be presented clearly and unambiguously to facilitate sound decision-making. Without such clarity, faulty decisions are going to be made, and even the highest technical achievements are going to fall flat on their face.