[cdn-nucl-l] Sorry, wrong number

["Adam McLean" <adam.mclean@utoronto.ca>]

Posted in the Lawrence Berkeley National Laboratory Science Beat
newsletter on December 17, 2002 and at:
http://enews.lbl.gov/Science-Articles/Archive/EETD-wrong-number.html
Good lessons for effective communication with media in the energy
sector...

Adam

---------------------

Sorry, wrong number: The use and misuse of numerical facts in analysis
and media reporting of energy issues
  
Contact: Allan Chen, a_chen@lbl.gov  
    
How much oil is recoverable from the Arctic National Wildlife Refuge?

Does one megawatt equal the electricity use of 1,000 U.S. homes? 

How much does unreliable power cost the U.S. economy? 

How much electricity is used by office equipment? 
  
None of these four energy-related questions have simple, unqualified
answers -- and in every case, energy analysts and the mass media have
unwittingly contributed to the spread of inaccurate information, says a
group of researchers from Lawrence Berkeley National Laboratory and the
Environmental Protection Agency.

A new study to be published in the Annual Review of Energy and the
Environment 2002 traces how these four questions have sometimes been
answered incorrectly or misleadingly by energy analysts, and how their
answers have spread to public discussions through mass-media reporting
of energy issues. 

The authors of the study, from Lawrence Berkeley National Laboratory's
Environmental Energy Technologies Division, the Environmental Protection
Agency, and the private sector, use their work to suggest that since
accurate energy statistics often become crucial to public debates about
energy policy, analysts need to take special precautions to reduce the
possibility of their abuse. 

"This work has its origins in the recent debate over how much
electricity is consumed by computers, office, and network equipment,"
says Jon Koomey, leader of Berkeley Lab's Energy End-Use Forecasting
Group. "One analyst's widely publicized estimate published in a story in
Forbes magazine was repeated over and over by other publications and
quickly became accepted by the media and the public, even though the
scientific community reviewed this analysis and found it to be
inaccurate." 

The original Forbes article claimed that in 1999, 13 percent of
electricity consumption in the U.S. was consumed by Internet-related
computer equipment. But independent estimates prepared by researchers at
Berkeley Lab and Arthur D. Little, Inc., found that computer and
office-related equipment uses roughly three percent of U.S. electricity
production. 

The researchers found six news stories, three magazine editorials, and
five reports from major investment banks that cited the erroneous Forbes
numbers without stating that there was any debate over their accuracy.
An additional 20 stories cited both sides of the debate, with some of
these giving equal weight to both sets of numbers. "By repeating the
claims of the Forbes article, often without citing a source, the trade
and popular press enshrined erroneous statistics as common knowledge,"
says Koomey.

"Although we and other researchers have published corrected numbers that
are generally agreed upon within the research community, the incorrect
numbers continue to be cited in popular articles and news accounts,"
Koomey continues. "This led us to wonder if the same has happened to
other statistical facts about energy, and whether they have had a
distorting effect on public discussions." 

The researchers studied each of the four questions posed at the start of
this article -- megawatts of electricity used by 1,000 homes; cost of
unreliable power; amount of electricity used by office equipment; oil
reserves -- investigated how energy analysts arrived at their estimates,
and tracked how those estimates were reported by the news media. 

Perhaps the most contentious of the four questions addresses oil
reserves in the Arctic National Wildlife Refuge (ANWR).

Oil reserves are a question of definition

The study applies a three-stage process to analyze how researchers and
the news media have answered the question of how much oil is recoverable
from ANWR. 

Much of the controversy concerns a coastal plain in the refuge called
area 1002, which is about the size of Delaware. The authors write that
"what is surprising is the extent to which the media have misunderstood
and poorly represented the underlying science." 

The U.S. Geological Survey has conducted much of the research on the
size of the area's economically recoverable reserves. Their work has
been a multistep process that began with estimating the amount of oil in
place, based on the area's geology, without reference to how much it
might cost to remove this oil. 

The second step was estimating the amount of technically recoverable
oil, which takes into account the industry's current state of technology
for extracting oil, without accounting for the potential cost to
accomplish this. (USGS published its most recent such analysis in 1998.)


Finally, USGS analysts overlaid the technically-recoverable estimate
with an economic analysis -- an estimate of economically recoverable
oil. This analysis takes into account the quality and market value of
oil, the costs of exploration and drilling, the financial costs of
extracting and transporting the oil, and the financial rate of return
expected at particular oil prices. 

The final step of this exhaustive analysis is a computer model of the
probability of cost-effective recovery. Thousands of runs of this model
provided the USGS analysts with a forecast of the mean (50 percent)
chance of finding a given amount of oil at a given price by comparing
the high (95 percent) and low (5 percent) chances. 

The higher the forecast's specified quantity of economically recoverable
oil from the reserve, the lower the chance of cost-effective recovery.
The 95 percent forecast predicts the presence of a small amount of oil,
but it is virtually certain to be found and recovered. The 5 percent
forecast points to an extremely large reserve, but it is unlikely that
crews will find so much oil. 

"Another issue," says Koomey, is that "economically recoverable resource
estimates can either be conditional or fully risked. Conditional
estimates are appropriate for thoroughly explored regions with
well-understood geology, like the onshore oil fields of the lower 48
states. They assume a 100 percent probability of finding economically
valuable quantities of oil, and simply assess how much of it is there.
Fully risked estimates are more appropriate to remote areas like the
Arctic Refuge, where much of the detail about underground structures is
still unknown." 

The most recent USGS evaluation yielded the probability curves shown in
the accompanying figure. 
 
Economically recoverable oil potential (fully risked) in the Arctic
National Wildlife Refuge's 1002 Area, as a function of oil price  
 

"The amount of economically recoverable oil resources from area 1002
depends strongly on the long-term market price of oil," write the
authors. "Considering the prevailing range of prices over the past 15
years (from $15 to $25 a barrel, in 1996 dollars) yields a range of
mean, fully risked, economically recoverable resources from 0 to 5.6
billion barrels."

How does the size of the reserve fare in public discussion? 

Almost all of the stakeholders in the oil drilling debate are using
different sets of number from the same source -- the 1998 U.S.
Geological Survey report -- to support their arguments. Proponents of
drilling can quote estimates ranging from 16 to 30 billion barrels of
oil, while opponents can use the same set of figures and argue there is
anywhere from zero to 3 billion barrels of oil. However, the research
team has shown that the news media, rather than going back to the
original source of the figures, has quoted many different sources from
among both proponents and opponents of drilling, resulting in the
dissemination of a wide range of estimates. 

Using online search tools, the researchers located 35 different news
stories printed or aired between December 2000 and September 2001
regarding the amount of oil likely to be found in the ANWR. All were
written by mainstream journalists; editorials and opinion pieces were
specifically excluded from consideration, as were articles appearing in
advocacy or trade association publications. The newspapers and magazines
include some of the largest, most widely read, and most influential
publications, web sites, and news shows in the United States -- the New
York Times, Washington Post, Los Angeles Times, Houston Chronicle, USA
Today, Scientific American, CNN.com, and CBS News among them. 

As shown in the accompanying chart, the estimates vary widely -- "They
are literally all over the map," say the authors. Five of the stories
included specific references to multiple types of studies, so those are
plotted separately, for a total of 40 specific sets of resource
estimates. 
  
 The amount of oil in the Arctic Refuge, as characterized in recent news
stories 
  
The authors write: "The most frequently cited estimate was 16 billion
barrels, which appeared in 24 of the stories. . . . The average high
estimate cited was 13 billion barrels and the average low estimate was
7.6 billion barrels, leaving readers to conclude that a number somewhere
in the middle -- about 10 billion barrels -- would be roughly right. "

However, comparing the "average" 10 billion barrel figure from media
reports to the mean curve in the chart of U.S. Geological Survey
estimates shown above indicates that the reports on average overstated
the economically recoverable reserves, at $20 per barrel, in the 1002
area by about a factor of three -- at that price, about 3 to 3.5 billion
barrels.

The USGS's mean curve indicates that the amount of recoverable oil in
the 1002 area is likely to range from zero to 5.6 billion barrels at oil
prices of $15 to $25 per barrel, in 1996 dollars. The authors conclude
"this range was only reflected by a handful of the news stories covering
the topic in the last year, and most reports unwittingly left the
impression that the amount of economically recoverable oil resources
fell substantially above this range." 

Reducing the misuse of figures

Only a handful of media stories distinguished their estimates of oil as
being technically recoverable or economically recoverable. Although 21
stories referred to the USGS as the source of the numbers, few of the
analyzed stories actually quoted anyone from the USGS; the numbers
generally came from proponents or opponents of drilling, not from
impartial sources.

The way people cite ranges can be misleading. "A range of estimates is
not always what it seems to be -- people summarize data in ranges, but
are careless about using consistent definitions. Or they cite
incomparable numbers, or extend the ends of the range to be
'conservative,' no matter what the original data say," says Koomey.
"Ranges, whether cited in the media or in analytical reports, should be
checked against the original source before they are cited by analysts or
the media." 

Going back to the original source is always important. Referring back to
the original USGS study and explaining the derivation of the reserve
estimates could have revealed information that was relevant to the
public debate, but few if any journalists took that step. This omission
was particularly glaring because there really is only one source for the
estimates of recoverable reserves, and the various participants in the
debate merely chose those numbers from the USGS studies that supported
their positions.

"Journalists often assume that all debates have two equal sides," Koomey
adds. "In some areas, particularly in scientific fields, there are right
and wrong answers, and by highlighting a few critics instead of
presenting the balance of scientific opinion, journalists can do the
public debate a disservice. News coverage that apportions differing
amounts of weight to the sides of a controversy -- noting for example,
whether a large majority of experts are on one side of the question or
another -- give the public a more accurate picture of where the experts
stand."

Finally, it is important to understand peer review, which is the process
by which scientists evaluate new research to judge whether it is based
on principles and judgments that represent the current scientific
consensus. Results must be peer-reviewed and reproducible by others
before being accepted as reliable by the scientific community. 

"Journalists should be skeptical of 'research' results that are not
peer-reviewed, such as those popularized in the Forbes article," says
Koomey. There are many advocates who announce their results to the media
but do not publish in peer-reviewed journals. Usually, such
announcements are funded by particular organizations and have little to
do with science.

Additional information

More about the overestimation of computer-related electricity use

More about the controversy over electricity used by computers and the
Internet

More about the oil and gas resource potential of Area 1002 in the Arctic
National Wildlife Refuge

More about how the price of oil can fluctuate dramatically 
Further information in print 

"Sorry, wrong number: The use and misuse of numerical facts in analysis
and media reporting of energy issues," by Jonathan Koomey, Chris
Calwell, Skip Laitner, Jane Thornton, Richard E. Brown, Joe Eto, Carrie
Webber, and Cathy Cullicott, will appear in Annual Review of Energy and
the Environment 2002 (forthcoming), edited by R. H. Socolow, D. Anderson
and J. Harte, Annual Reviews, Inc., Palo Alto, Calif. 

Turning Numbers into Knowledge: Mastering the Art of Problem Solving, by
Jonathan Koomey, published in 2000 by Analytics Press, Oakland, Calif.
For more information, visit http://www.numbersintoknowledge.com.