The Political Fragility of Metascience

Vannevar Bush being sworn in as head of the Research & Development Board (1947)

To the extent that the burgeoning field of metascience has a policy agenda, it is that science should be governed by rigorous science. We should test scientific practices using the scientific method.

In large part, this policy agenda has included encouraging the adoption of methods like the randomized control trial (RCT) as the gold standard for assessing a whole range of science policymaking issues, from addressing the reproducibility crisis in various fields to judging the effectiveness of various grantmaking structures.

So, to the extent that metascience has prescriptive ambitions beyond providing an ever-better description of scientific progress and how it comes about, it has fittingly landed on a fairly meta objective: more metascience! It advocates for the organs of government to invest more aggressively in ongoing metascientific inquiry in their respective domains.

To be clear, this core vision is a very attractive one. It calls for a generation of heroic bureaucrats – empowered by modern econometric tools – to engage in a dynamic campaign to reform and revitalize the productivity of the national scientific enterprise. I can buy into this vision, and I have little doubt that such a world would result in more effective science policy, and a healthier overall economy of science.

The problem is that the implicit theory of change for metascience seems at the present moment wildly fragile. There is an underlying assumption that the fruits of metascience will stand on their own. That is, the piling up of empirical results will produce bodies of evidence so strong that they motivate institutions to change how they direct and shape scientific inquiry.

But it is unclear if data itself can achieve such a result. Metascience can become effective only if the institutions of science can act decisively in reaction to its findings. 

This is particularly challenging since the most important RCTs are also the most dangerous, politically fraught ones. An experiment might reveal that an entire career’s worth of grantmaking had functionally no impact on the final development of a field. Another might show that the perverse incentives of a storied institution are to blame for stagnation in a field. Yet another might show that a much-touted researcher or research area was in fact a failure.

There are existential consequences to the outcomes of these analyses. Funding will be slashed, research programs will be ended. Will our institutions have sufficient bureaucratic courage to do so when they are so called upon? Getting the RCT done in the first place is one thing. Ensuring that its results have an actual influence on the direction of science policy is another. Shaping this direction requires more than a set of fastidious, high-power experiments. These are fuzzier questions of political power, institutional design, and bureaucratic risk-taking.

If that’s all true, then the core mission of metascience – more metascience – necessarily implies a broader political agenda. For metascience to have a truly systemic impact on the governance and management of science, it will need to simultaneously remake broader institutional cultures, structures, and incentives. Metascience advocates will be faced with a messier project: steering the norms of science institutions, implementing new organizational architectures in science governance and administration, and forging durable political coalitions that support these arrangements.

This is an exercise in canny, practical politics as much as it is a normative, philosophical project. What is the proper role of government in science? What are the public goods that decision makers should trade between in deciding what to advance in science? What are the respective responsibilities of government, business, and academia in advancing progress? These are not questions that are not resolvable through experiment, at least not on the timescales that matter.

Nevertheless, metascience needs a sharper point of view on these big questions since rallying public discourse and articulating a broader vision may ultimately be upstream of the positivist agenda of metascience. After all, RCTs do not make a better science, people do. Inspiring an experimental, risk-taking culture precedes the acceptance of a certain empirical approach, rather than the other way around. 

To survive and succeed, metascience will need to embrace skills quite foreign to its primary research work: the philosophical, the political, and even the aesthetic. Failing to do so will leave the destiny of metascience up to existing interests, many of whom will be funders, institutions, and political stakeholders hostile to the reforms needed to restore scientific dynamism.

---

Thanks to Matt Esche, who provided the conversation that inspired this post. Thanks also to Caleb Watney and Santi Ruiz, whose feedback greatly improved it.