The ‘replication crisis’ has exposed serious quality flaws in a wide range of sciences. Environmental science used for public policy is rarely tested and may have significant flaws. Present approaches to science Quality Control risk failing policymakers and the environment. A new body is needed to undertake Quality Control of policy-science.
Fig. 1. A graphic illustrating 22 factors that can influence the decisions taken by UK government ministers (after Larcombe, 2007).
The need for a formalised system of Quality Control for environmental policy-science
Marine Pollution Bulletin, Volume 126, January 2018, Pages 449–461
Piers Larcombe and Peter Ridd
Abstract
Research science used to inform public policy decisions, herein defined as “Policy-Science”, is rarely subjected to rigorous checking, testing and replication. Studies of biomedical and other sciences indicate that a considerable fraction of published peer-reviewed scientific literature, perhaps half, has significant flaws. To demonstrate the potential failings of the present approaches to scientific Quality Control (QC), we describe examples of science associated with perceived threats to the Great Barrier Reef (GBR), Australia. There appears a serious risk of efforts to improve the health of the GBR being directed inefficiently and/or away from the more serious threats. We suggest the need for a new organisation to undertake quality reviews and audits of important scientific results that underpin government spending decisions on the environment. Logically, such a body could also examine policy science in other key areas where governments rely heavily upon scientific results, such as education, health and criminology. […]
How to achieve rigorous technical scrutiny for policy-science?
The above example illustrates how poor QC mechanisms for policy-science put at risk effective direction of resources regarding dealing with the GBR’s environmental problems, but it is logical that a similar problem may also exist for many other environmental issues. This is not a new observation. Commenting on general matters of science credibility, Duarte et al. (2015) called for a “systematic audit” of ocean calamities, and Browman (2016) suggested the need for organised skepticism. Given that governments often use the results of environmental science to make important decisions, it is for them to commit appropriate funds to the task.
Therefore, we propose that governments should establish a new independent organisation to undertake quality reviews and audits of important scientific results which underpin government spending decisions. Here we have named it an “Institute for Policy-Science Quality Control” (IPSQC), but the name is far less important than its intended role, and the way it is structured and funded. Although the focus in this paper is on the environmental sciences, there are similar problems with policy-science in other areas where governments rely upon scientific results, such as education, health, and criminology. The IPSQC would thus not necessarily be restricted to environmental policy-science.
Regarding the role of a new body, we suggest it would conduct a system of guaranteed and organised technical debate, with the aim to specifically and rigorously test for any significant deficiencies in the scientific work upon which the major public expenditure is based. It would appear inevitable for some early focus to be on existing policy-science associated with policy driving current public spending, and that over time the focus would shift more towards assessing the quality of policy-science relevant to the development of new policy. There also seems a clear potential formal role supporting the process of setting environmental regulations and in performing reviews as policy options are considered. Whilst some policy-science is used in these processes, it is not routinely rigorously checked, and funds are almost never set aside to replicate important work. Again, rather than acting to form policy, the intended role is to check the veracity of the science being used by policymakers.
Clearly, any such organisation performing such a role would need significant resources to fund external scientists or to employ its own. Viewing the Australian GBR example at least, if these roles are the implicit role of any existing organisation or organisations, the evidence regarding GBR policy-science indicates to us that it is not working. The precise mechanisms used by this new organisation could take a number of different forms. There are pros and cons to adversarial models (i.e. using a classical legal approach of prosecution and defence) and to ‘truth commission’ models (as used in post-apartheid South Africa). However, whatever the mechanism(s) used, there must be independence, openness and transparency in all aspects. As an example, in an adversarial model, the organisation might act like a defence attorney in a court trial, challenging the scientific evidence being used to support the government decision or intended decision. Depending upon the specific cases, this is likely to involve open questioning of scientists, commissioning attempts to replicate previous work, reanalysing data, checking experimental design, analytical methods and results, and ensuring that alternative interpretations are thoroughly considered and described.
A copy of the full paper is available from the lead author on request.