The wonderful “A billion here, a billion there, and pretty soon you’re talking real money” statement attributed to Senator Everett Dirksen may be apocryphal, but it remains a prescient warning to our nation’s leaders. At a time when Congress is throwing billions of dollars around like pocket change based on claims of scientists and engineers, a real quote of Dirksen may be equally important (Congressional Record: June 16, 1965, p. 13884):
One time in the House of Representatives [a colleague] told me a story about a proposition that a teacher put to a boy. He said, ‘Johnny, a cat fell in a well 100 feet deep. Suppose that cat climbed up 1 foot and then fell back 2 feet. How long would it take the cat to get out of the well?
Johnny worked assiduously with his slate and slate pencil for quite a while, and then when the teacher came down and said, ‘How are you getting along?’ Johnny said, ‘Teacher, if you give me another slate and a couple of slate pencils, I am pretty sure that in the next 30 minutes I can land that cat in hell.
The nation needs Johnny. In fact, it may be time we hired a team of people like Johnny for every large science-based policy proposal Congress contemplates funding.
Carbon Capture and Storage: A Known Boondoggle
Consider, for example, the $4.4 billion Congress is putting into carbon capture and sequestration (CCS) research, nearly half of that to come from the Kerry-Lieberman climate bill. As Robert Brycepoints out in the New York Times, “That’s a lot of money for a technology whose adoption faces three potentially insurmountable hurdles: it greatly reduces the output of power plants; pipeline capacity to move the newly captured carbon dioxide is woefully insufficient; and the volume of waste material is staggering.”
Those of us familiar with the coal-fired power plant industry have long recognized that CCS may be slightly more than a pipe-dream, but will never be affordable or practicable for the vast majority of coal-fired plants. Yet no one in the bureaucracy has had the courage to stand up and refute this politically correct but scientifically bankrupt concept.
Lessons from a Broken Hockey Stick
Nor is CCS the only example. Perhaps the highest visibility science that needed a “Johnny” is the now infamous global warming hockey stick. Andrew Montford’s “The Hockey Stick Illusion – Climategate and the Corruption of Science” deconstructs the woeful practices leading to this canard that propelled the entire world toward economic investments that are likely as not to be entirely wasted.
In his book, Montford identifies individuals within the paleoclimatology community who argued that they would not have used the invalid data upon which the hockey stick depended. But, they did not come forward, nor were they allowed to “peer review” the work before its publication. And why was that?
It took an act of Congress to find a “Johnny” to sort out the hockey stick and explain why no one came forward. The resultant Wegman report concluded, among many other important things:
The politicization of academic scholarly work leads to confusing public debates. Scholarly papers published in peer reviewed journals are considered the archival record of research. There is usually no requirement to archive supplemental material such as code and data. Consequently, the supplementary material for academic work is often poorly documented and archived and is not sufficiently robust to withstand intense public debate. In the present example there was too much reliance on peer review, which seemed not to be sufficiently independent.
And Wegman recommended:
Especially when massive amounts of public monies and human lives are at stake, academic work should have a more intense level of scrutiny and review. It is especially the case that authors of policy-related documents like the IPCC report, Climate Change 2001: The Scientific Basis, should not be the same people as those that constructed the academic papers.
The “Plan B” Idea
Having indepdent peer review is not a new idea. In 2005, Steven Hayward suggested “Perhaps the time has come to consider competition as the means of checking the Intergovernmental Panel on Climate Change’s monopoly and generating more reliable climate science.”
The Heartland Institute made such an attempt with its “Non-Governmental International Panel on Climate Change” report and the Center for the Study of Carbon Dioxide and Global Climate has mounted an effort to continue this work. But these efforts have failed to produce a meaningful impact on the political process, largely because the authors do not operate within the “peer” community and do not publish in the peer-reviewed literature.
There are better models, however, and these could be adapted to science-based policy analyses. For example, by federal law, an investment prospectus must include sufficient specification of risk assumptions as to permit independent analysis of the proposal. In like measure, financial institutions’ annual reports require independent audit analysis before publication.
The key is independent replication of the analysis and evaluation of important assumptions.
In the cold war days, the military used an approach called “Team B.” This team (alternatively called the Red team) developed approaches to counter the efforts of Team A (the Blue team). Still in use today, it has some strengths and some weaknesses. Without question, however, it always sharpened the quality of Team A’s work, finding errors in analysis, weaknesses of assumptions and entire areas of interest that opened up Team A’s thought processes and analytical options.
Team B operated within the professional “community,” making its contributions instantly credible and mandatory for review by Team A and the eventual policy makers.
The Marshall Institute had engendered discussion of this approach for today’s scientific policy issues, as have members of the American Physical Society. In particular, Dr. William Happer, Cyrus Fogg Brackett Professor of Physics at Princeton University, member of the National Academy of Sciences, and Fellow of the American Physical Society (APS), has proposed a Team B approach within the APS. Without question, the APS has enough independent and qualified members to have been able to audit the critical work underlying the IPCC reports and inform journal publisher’s of the important analytical elements to which regular peer-reviewers should give special attention.
A Proposal
I have little confidence in Happer’s approach, however. It stands too far back from the critical scientific process and will make contributions too late in the policy process. Instead, we can take a page from the book of the existing science policy bureaucracy – environmental assessments.
Under the National Environmental Policy Act, a project using significant federal funds cannot move forward until some form of environmental assessment has been completed and approved. These assessments have, of course, been misused to delay and prevent projects. However, at their heart, they have significantly improved proposals with the potential to adversely impact the environment. Correcting to eliminate the pathologies of the process, something similar could be used in other areas.
In the context of science-based policy today, Team B would be thene plus ultra peer-review group required for any scientific or engineering initiative (including grants and grant programs) that Congress or the Federal bureaucracy planned to fund above a trigger level, or which would have such an impact on the private sector. The trigger might be $500 million, for example.
Before Congress authorized expenditure of such funds, the bill sponsor would need to offer a Team B report that validated the basis otherwise offered to justify the expenditure. An National Science Foundation program manager wishing to fund a subject area that expands high-cost research would need a valid Team B analysis of the science to ensure the area of investigation is justified. The Department of Energy, the Interior Department and the Environmental Protection Agency would need a Team B report on their major research initiatives and regulatory proposals. Any major studies commissioned or conducted by the National Research Council (and its subsidiaries) would require a Team B review.
In every case, a Team B report would involve an independent replication of the analysis, including assumptions, selection and screening of data, logic of computer code, statistical analysis and presentation of data. The Team B report would have to demonstrate its independence, using the tools discussed in the Wegman report.
This independence is a key aspect of a Team B approach. It would not be helpful to simply pit antagonists against each other; and, of course, one cannot permit the incestuousness observed in the Mann-Jones et al network. Rather, one wants a “one-off” approach. In nearly every case, one would want a Team B to include a statistician familiar with the statistical methods used, but not familiar with the authors and perhaps only peripherally knowledgeable about the scientific subject. They can obtain the information about the subject from the subject matter specialists on the team.
The real challenge is selecting subject matter specialists. In cases where funding opportunities is controlled by cliques, there is even a greater problem. Because all Team B work should be done in the light of day, perhaps all one needs is an honest Team B. (OK, call me Diogenes.) An open process, one that access the entire planet’s available and experienced scientific community through the internet, may be the guarantor of a quality project.
The Upside of Reform
While Wegman did not believe peer review ought to be through blogs, without question, without McIntyre’s efforts, as recorded on his blog site, Mann’s corrupt hockey stick would never have been exposed. A competent Team B could harvest the intellectual contributions from an engaged web-community to good effect, especially with regard to highly technical issues. Further, the anonymity of the web would allow specialists to provide critical information without having to be worried about being outcast by the specialists’ cliques.
Surely a Team B approach would evolve and mature over time. What is certain, however, is that even the threat of a Team B review would occasion a return to trustworthy science and a fundamental return to the bedrock principles of scientific inquiry. As Wegman, Montford and McIntyre have made unutterably clear, if we need heightened oversight of financial institutions, we surely need such heightened oversight of the science upon which we found science-related policies having large economic consequences.