One part of this process that we haven't talked much about is the reporting phase - taking the results of an analysis and presenting them to the target audience.
Discussion of the Challenger
disaster, Jan 28, 1986, on its 10th flight, with refs from Tufte's
The Challenger example gets talked about a lot in terms of
ethics and responsibility and there are various views on the topic.
One lengthy critique of Tufte's conclusions (which brings up several
very important points but also reads into Tufte's work an attitude
that I do not feel while reading it) is given here:
Since we haven't flown these for a while, just for reference, here is a photo of Atlantis on the launch pad:
The engineers at Morton Thiokol who designed the solid
rocket boosters for the shuttle opposed the launch of Challenger due
to the expected low temperatures that day, and faxed 13 diagrams to
NASA management to make their case. They failed in large part
because of what information they chose to present and the way they
presented that information, but also because of time and information
There is a nice overview here:
including the following:
‘We discussed what might happen below our 40 degree
qualification temperature and practically to a man we decided it
would be catastrophic,’ added Morton Thiokol's Bob Ebeling.
‘Thiokol recommended that we could not launch until the
weather warmed up in the afternoon,’ said NASA senior manager Jud
Lovingood. ‘Well I told them they couldn’t make that recommendation.
They had to give us a temperature that we could launch with.’
A formal presentation would have to be made, two hours after speaking with Lovingood and just 15 hours before launch, via a teleconference at which Thiokol would need to given their reasoning for a no launch decision – a power contractors held, but were scared to make given the effects on the Shuttle schedule.
Thiokol engineer Roger Boisjoly – one of two specialists (the other being Arnie Thompson) on the SRB joint seals – grabbed anything he could from his office to show how the temperature would lead to a failure of the SRB’s O-ring and the destruction of the Shuttle.
‘Unfortunately in our rush we didn’t have time for a dry run at what we’d present to NASA,’ noted Boisjoly. ‘I had no idea what my colleagues would present and I had no idea what I’d bring to the meeting.’
‘The entire Thiokol group recommended no launch,’
remembered Ebeling, as they recommended a minimum launch temperature
of 53F (11C). The expected rubber stamping of that recommendation
was expected from NASA on the other end of the teleconference.
However, they would be proven wrong.
There had been several earlier flights with O-ring
problems and the issues were being worked through to create a less
flawed design, so NASA knew there were continuing concerns, but the
amount of data available from the 24 previous shuttle flights was
limited. For example the Morton Thiokol engineers did not have
temperature data available for all of the previous shuttle flights
(air temperature, or the much more useful O-ring temperature.) Seven
earlier flights had O-ring issues, and those launches were at
temperatures (F) of 53, 57, 58, 63, 70, 70, 75, and only two of
those had serious 'blow by' issues, one at 53F and one at 75F. There
were problems when it was warm; there were problems when it was
cool. No shuttle had launched below 53 degrees F before.
Mr Mulloy, the Solid Rocket Booster Project Manager,
testified: It was Thiokol's calculation of what the lowest
temperature an O-ring had seen in previous flights, and the
engineering recommendation was that we should not move outside of
that experience base.
Another person present said "One of my colleagues that was
in the meeting summed it up best. This was a meeting where the
determination was to launch, and it was up to us to prove beyond a
shadow of a doubt that it was not safe to do so. This is in total
reverse to what the position usually is in a pre-flight conversation
or a flight readiness review. It is usually exactly opposite that."
The key table that the engineers produced was:
If instead of using chronological order the same data was
presented in ascending temperature order the pattern is a bit more
If instead we remove all the extraneous imagery and do a
simple plot of temperature vs damage (a weighted average of erosion,
heating, and blow-by) as shown below then the pattern becomes much
clearer, which is the point that Tufte stresses. This chart is almost
the same as the revised Rogers Commision chart in pdf form above.
To really do analysis you would still want to be able to get
access to the more detailed data - this just gives you a nice overview.
(note that 15 and 22 mentioned in the original memo are highlighted in this chart)
The Rogers Commission report on the
Challenger can be found at:
Discussion of Columbia disaster in
2003 on its 28th mission, with refs from Tufte's "Beautiful Evidence"
- how you organize, present text, and choose words can be just as
dangerous as how you present graphical information. Do all of the
necessary words even fit readably on a PowerPoint slide?
Unlike Challenger, this time the issue was less about what the engineers knew, and more about what they did not know and their inability to convince their managers to get them more information from the astronauts in space or department of defense imagery.
The results of an analysis needed
to be succinctly presented in a report or set of PowerPoint slides,
with the bulk of the analysis sitting in a very big report that may
not be read. Tufte spends a fair amount of time in this book talking
about the dangers inherent in a PowerPoint presentation.
first a bit of background from the Columbia report which we
will hit the highlights of ...
"Columbia was launched from Launch Complex 39-A on January 16,
2003, at 10:39 a.m. Eastern Standard Time (EST). At 81.7 seconds after
launch, when the Shuttle was at about 65,820 feet and traveling at Mach
2.46 (1,650 mph), a large piece of hand-crafted insulating foam came off
an area where the Orbiter attaches to the External Tank. At 81.9
seconds, it struck the leading edge of Columbiaʼs left wing. This event
was not detected by the crew on board or seen by ground support teams
until the next day, during detailed reviews of all launch camera
photography and videos. This foam strike had no apparent effect on the
daily conduct of the 16-day mission, which met all its objectives."
"Post-launch photographic analysis showed that one large piece
and at least two smaller pieces of insulating foam separated from the
External Tank left bipod (–Y) ramp area at 81.7 seconds after launch.
Later analysis showed that the larger piece struck Columbia on the
underside of the left wing, around Reinforced Carbon-Carbon (RCC) panels
5 through 9, at 81.9 seconds after launch (see Figure 2.3-2). Further
photographic analysis conducted the day after launch revealed that the
large foam piece was approximately 21 to 27 inches long and 12 to 18
inches wide, tumbling at a minimum of 18 times per second, and moving at
a relative velocity to the Shuttle Stack of 625 to 840 feet per second
(416 to 573 miles per hour) at the time of impact."
"The objectʼs large size and the apparent momentum transfer
concerned Intercenter Photo Working Group personnel, who were worried
that Columbia had sustained damage not detectable in the limited number
of views their tracking cameras captured. This concern led the
Intercenter Photo Working Group Chair to request, in anticipation of
analystsʼ needs, that a high-resolution image of the Orbiter on-orbit be
obtained by the Department of Defense. By the Boardʼs count, this would
be the first of three distinct requests to image Columbia on-orbit. The
exact chain of events and circumstances surrounding the movement of each
of these requests through Shuttle Program Management, as well as the
ultimate denial of these requests, is a topic of Chapter 6."
and here are a couple photos of that same area on Endeavour,
taken in the summer of 2015
"Without on-orbit pictures of Columbia, the Debris Assessment
Team was restricted to using a mathematical modeling tool called Crater
to assess damage, although it had not been designed with this type of
impact in mind. Team members concluded over the next six days that some
localized heating damage would most likely occur during re-entry, but
they could not definitively state that structural damage would result.
On January 24, the Debris Assessment Team made a presentation of these
results to the Mission Evaluation Room, whose manager gave a verbal
summary (with no data) of that presentation to the Mission Management
Team the same day. The Mission Management Team declared the debris
strike a “turnaround” issue and did not pursue a request for imagery."
"Boeing analysts conducted a preliminary damage assessment on
Saturday. Using video and photo images, they generated two estimates of
possible debris size – 20 inches by 20 inches by 2 inches, and 20 inches
by 16 inches by 6 inches – and determined that the debris was traveling
at a approximately 750 feet per second, or 511 miles per hour, when it
struck the Orbiter at an estimated impact angle of less than 20 degrees.
These estimates later proved remarkably accurate."
"To calculate the damage that might result from such a strike,
the analysts turned to a Boeing mathematical modeling tool called Crater
that uses a specially developed algorithm to predict the depth of a
Thermal Protection System tile to which debris will penetrate. This
algorithm, suitable for estimating small (on the order of three cubic
inches) debris impacts, had been calibrated by the results of foam, ice,
and metal debris impact testing. "
"Until STS-107, Crater was normally used only to predict whether small debris, usually ice on the External Tank, would pose a threat to the Orbiter during launch. Engineers used Crater during STS-107 to analyze a piece of debris that was at maximum 640 times larger in volume than the pieces of debris used to calibrate and validate the Crater model (the Boardʼs best estimate is that it actually was 400 times larger)."
"For the Thermal Protection System tile, Crater predicted
damage deeper than the actual tile thickness. This seemingly alarming
result suggested that the debris that struck Columbia would have exposed
the Orbiterʼs underlying aluminum airframe to extreme temperatures,
resulting in a possible burn-through during re-entry. Debris Assessment
Team engineers discounted the possibility of burn through for two
reasons. First, the results of calibration tests with small projectiles
showed that Crater predicted a deeper penetration than would actually
occur. Second, the Crater equation does not take into account the
increased density of a tileʼs lower “densified” layer, which is much
stronger than tileʼs fragile outer layer. Therefore, engineers judged
that the actual damage from the large piece of foam lost on STS-107
would not be as severe as Crater predicted, and assumed that the debris
did not penetrate the Orbiterʼs skin."
"Prior to STS-107, Crater analysis was the responsibility of a team at Boeingʼs Huntington Beach facility in California, but this responsibility had recently been transferred to Boeingʼs Houston office. Even though STS-107ʼs debris strike was 400 times larger than the objects Crater is designed to model, neither Johnson engineers nor Program managers appealed for assistance from the more experienced Huntington Beach engineers, who might have cautioned against using Crater so far outside its validated limits. Nor did safety personnel provide any additional oversight. NASA failed to connect the dots: the engineers who misinterpreted Crater – a tool already unsuited to the task at hand – were the very ones the Shuttle Program identified as engendering the most risk in their transition from Huntington Beach."
"Columbia re-entered Earthʼs atmosphere with a pre-existing breach in the leading edge of its left wing in the vicinity of Reinforced Carbon-Carbon (RCC) panel 8. This breach, caused by the foam strike on ascent, was of sufficient size to allow superheated air (probably exceeding 5,000 degrees Fahrenheit) to penetrate the cavity behind the RCC panel. The breach widened, destroying the insulation protecting the wingʼs leading edge support structure, and the superheated air eventually melted the thin aluminum wing spar. Once in the interior, the superheated air began to destroy the left wing."
"By the time Columbia passed over the coast of California in
the pre-dawn hours of February 1, at Entry Interface plus 555 seconds,
amateur videos show that pieces of the Orbiter were shedding. Analysis
indicates that the Orbiter continued to fly its pre-planned flight
profile, although, still unknown to anyone on the ground or aboard
Columbia, her control systems were working furiously to maintain that
flight profile. Finally, over Texas, just southwest of Dallas-Fort
Worth, the increasing aerodynamic forces the Orbiter experienced in the
denser levels of the atmosphere overcame the catastrophically damaged
left wing, causing the Orbiter to fall out of control at speeds in
excess of 10,000 mph."
and for a bit of perspective:
One debris strike in particular foreshadows the STS-107 event. When Atlantis was launched on STS-27R on December 2, 1988, the largest debris event up to that time significantly damaged the Orbiter. Post-launch analysis of tracking camera imagery by the Intercenter Photo Working Group identified a large piece of debris that struck the Thermal Protection System tile at approximately 85 seconds into the flight. On Flight Day Two, Mission Control asked the flight crew to inspect Atlantis with a camera mounted on the remote manipulator arm, a robotic device that was not installed on Columbia for STS-107. Mission Commander R.L. “Hoot” Gibson later stated that Atlantis “looked like it had been blasted by a shotgun.”18 Concerned that the Orbiterʼs Thermal Protection System had been breached, Gibson ordered that the video be transferred to Mission Control so that NASA engineers could evaluate the damage.
When Atlantis landed, engineers were surprised by the extent
of the damage. Post-mission inspections deemed it “the most severe of
any mission yet flown.” The Orbiter had 707 dings, 298 of which were
greater than an inch in one dimension.
One issue is whether anything could have been done. Given its orbit there is no way that Columbia could have docked with the (still incomplete) International Space Station, but Atlantis was on the second launch pad in Florida being readied for its mission, so its possible that a rescue could have been attempted. It is also possible that the astronauts could have done repair work to keep the shuttle together long enough for them to be able to bail out at a reasonable altitude.
Below is the Tufte analysis of the powerpoint slides used by
the Debris Assessment Team, which is also part of the official report:
again from the report:
"As information gets passed up an organization hierarchy, from people who do analysis to mid-level managers to high-level leadership, key explanations and supporting information is filtered out. In this context, it is easy to understand how a senior manager might read this PowerPoint slide and not realize that it addresses a life-threatening situation."
"At many points during its investigation, the Board was
surprised to receive similar presentation slides from NASA officials in
place of technical reports. The Board views the endemic use of
PowerPoint briefing slides instead of technical papers as an
illustration of the problematic methods of technical communication at
The complete Columbia report can be found at
Now we are going to go back to the utility dataset we took a look at in week 2 to introduce you to R, to see what more you can get out of it now that you are more familiar with data visualization in R.
Back in week2 we did this within R studio. This time we are
going to use a Jupyter Notebook, which can be better for documenting the
flow of an analysis process. Jupyter notebooks can be installed locally
but for this assignment we will use the web based version at https://mybinder.org/v2/gh/binder-examples/r/master?filepath=index.ipynb
once that page finishes loading you have an interactive environment showing data from the default mtcars dataset with several cells of documentation, code, and resulting graphs. You can highlight each cell and use the scissor icon up top to delete them all, then use the + icon to add in new cells.
You can add plain text into the cells so in the first cell type the names of your group members.
Add another cell below that one. Note that you can use the up
and down arrows to move a cell around.
Pasting 'read.table(file = "https://www.evl.uic.edu/aej/424/utilitydata2018.tsv", sep = "\t", header = TRUE)' without the 's into the first cell and then either hitting ctrl-enter or clicking on the Run icon or going to the Cell menu and selecting Run All will load in one of the datasets from week 2.
Replacing that cell with utility<-read.table(file = "https://www.evl.uic.edu/aej/424/utilitydata2018.tsv", sep = "\t", header = TRUE) will allow you to read in that data and store it.
Adding a new cell with + and typeinto it:
utility$newDate <- ymd(paste(utility$Year, utility$Month, "01", sep="-"))
will let you load in lubridate to help convert the time format.
Type the following into the third cell will allow you to load in
ggplot and make a plot:
ggplot(utility, aes(x=newDate, y=Temp_F)) + geom_point(color="blue") + geom_line()
Type the following into a 4th cell will let you plot another chart from the same data:
junes <- subset(utility, Month == 6)
ggplot(junes, aes(x=newDate, y=E_kWh_per_Day)) + geom_point(color="blue") + geom_line() + coord_cartesian(ylim = c(0,80)) + geom_line(aes(y=junes$Temp_F))
The notebook helps to see these different charts at the same
time. You should use plain text in the cells to document what you are
doing and what you have found, and why you think your explanation is
correct and alternative possible explanations are not.
You may occasionally want to save and checkpoint.
Use the notebook to document your findings about the dataset. As last time, some things you can find are:
When you are done you can use the File Menu to download As a
number of different formats - HTML is the most suitable for turning this
in, though the power of notebooks comes from being able to save off the
entire notebook, give it to someone else, and let them see your work and
be able to interactively engage with it.
Viewer's Choice Presentations