Metascience in Dangerous Times

Ship in a Storm on a Rocky Coast, by Jan Porcellis (painted around 1618). Source: Wikimedia.

As Macroscience relaunches, a fundamental question has been on my mind. Namely, how must metascience adapt to understand the shape of science in the coming years?

Since we went on hiatus late last year, the Trump administration has fundamentally reconfigured the government’s involvement in American science and research. The norms that govern the degree to which political authorities intervene in the life of the academy have been reshaped. From cuts to research funding to the reworking of overhead rules, the underlying economics of the research university are being tinkered with in a way that they have not been since Vannevar Bush’s 1945 report The Endless Frontier.

While a future administration might reverse some of these changes, the last twelve months will have enduring effects on the national research ecosystem. Most obviously, financial pressure may lead research institutions to close or significantly restructure their operations.

There are permanent human capital implications as well. Funding uncertainty and visa terminations will force some researchers to exit the formal research ecosystem entirely. Those people will not return, and their fields will be indelibly changed as a result.

Technological shifts are further agitating this instability. While the role that artificial intelligence might ultimately play in accelerating scientific progress writ large remains unclear, now-commonplace technologies like ChatGPT and Claude are already shaping the building blocks of research. Literature review, citations, and grant preparation have changed, perhaps forever.

These changes will affect the metascience community and how it conducts research. A vulnerability of the metascience research typically conducted by university economists is that it assumes a certain fixity to the specific configuration of institutions, funding, and research that has dominated post-war science. This assumption leaves academic metascience ill-suited to this uniquely volatile moment. Practically speaking, institutional instability makes it more challenging to conduct the careful interventions and longitudinal trials necessary to come to sound empirical conclusions. Institutions may not be able to durably invest in metascientific experimentation alongside their other priorities.

Moreover, where research happens may shift. Rather than staying in the US, foreign talent may return home or travel elsewhere to conduct their research. Even domestic talent may abandon the machinery of academic research in favor of private industry or philanthropically funded focused research organizations (FROs), where they could pursue their work more nimbly. In all these cases, research may become more fragmented and privatized, limiting the visibility that metascience has into how science happens.

Thus far, metascience research has depended on the legibility of how we conducted science in the late 20th century. To pick a granular example, the value of citations — a controversial metric even at the best of times — assumes certain stable institutional priors: that research is published at all by participants in a field, that humans exercise judgment in choosing what to cite, that large bodies of research do not disappear or become unavailable. These assumptions are all being challenged in the present moment. The scattering of researchers into private entities may weaken the uniformity of publication norms. Researchers might use artificial intelligence to programmatically surface related research and curate citations. And government funding cuts may render longstanding archives of papers and research material unavailable, or at least unmaintained.

This goes beyond the cliche that we live in times of great change. The institutional homes of science are under dramatic pressures that introduce a wave of methodological challenges. These changes will strain metascience’s established toolkit.

It is comforting to think that nothing ever happens, and that perhaps after a few years of gyration things will by and large return to past norms. But I’m not so sure. We need to retool metascience so that it can deliver insights in a radically changed scientific context.

There is a lot to talk about here, but it strikes me that any new mode of metascience must take the following three elements into account.

First, the metascience community must build partnerships with the new power centers of scientific research. If the weakening of the traditional research university pushes the scientific endeavor into new auspices, metascience needs to follow it there. How might private companies be persuaded to support metascientific research at scale? How should we study bibliometrics in a world where private companies have weaker incentives to publish results?

Second, there are increased benefits to directly observing science. Rapid institutional and technological shifts mean that quantified, large-scale datasets may be the lagging (or even misleading) indicator of how scientific progress is happening. “Shoe-leather” metascientific observation will be more valuable; seeing what’s happening on the ground may be the only way to gain insight into research practices and incentives.

Third, we need experimental designs that provide useful empirics at higher speed. Research that yields meaningful explanatory power about how science works requires long-term institutional buy in. Institutional volatility makes that difficult. Research designs that take advantage of this volatility, or can “get in and get out” rapidly with useful results will become more valuable. This is as much a matter of the research tools being used as it is the efficiency of the research teams conducting the work.

To lay claim to being a “science of science” means that metascience itself must be nimble. We must study science as it is being conducted, not as we wish it was being conducted. This requires us to ruthlessly question our tools. Doing so will ensure that the field can continue contributing meaningful insights, even as science changes radically in the coming years.