In the past few weeks, metascience units have gained new prominence. First, the National Science Foundation (NSF) announced the Office of Metascience. This is, to our knowledge, the first metascience unit in the federal government. Then, the bipartisan House Build America Caucus agenda elevated metascience units as a top national innovation priority. And in Japan, Office of Science and Technology Policy (OSTP) Director Michael Kratsios announced a vision statement with 17 countries that included a call for “dedicated metascience capabilities in our governments,” which sure sounds like metascience units.
This was all welcome news to us, since we had published an article proposing metascience units at federal agencies in July. As we described them, metascience units are:
Nimble teams of government staff and academic experts housed within agency leadership offices. Using the tools of metascience, these units would run institutional experiments and analyze the treasure troves of internal agency data to provide evidence about how to most effectively advance breakthroughs.
The US government needs metascience units because (1) science agencies have limited information on how well science funding is being spent and (2) there are enormous opportunities to make science more efficient and increase the likelihood of breakthroughs.
However, metascience units are new in the US.1 If the US government is serious about creating these units across science agencies (as it appears to be), it’s important to get clear about what they should look like.
Since publishing our piece, we have received thoughtful comments from across the science policy ecosystem, including from people currently or formerly in government, in academia, and in think tanks.2
This feedback focused on eight important topics:
How different metascience units should be across different agencies.
Whether the units should fund external grants and contracts.
How independent the units should be from agency leadership.
How to make the units sustainable.
How the units should be staffed.
How the units should work with other agency analysis offices.
What sorts of evidence the units should focus on.
What they should produce, and how their findings should change what the agency does.
Thinking through our answers to these questions helped us sharpen and update our views. For example, a few comments convinced us that metascience units must have enough independence, likely enshrined in congressional authorization, to share research results even if those results are uncomfortable for agency leadership.
To keep the conversation going, we’re sharing our responses to each question here.
Question 1: Should metascience units look different in different agencies?
We positioned this question first not because it’s the most important question, but because it helps explain why we say “it depends” for subsequent questions.
Metascience units will likely differ quite a bit based on their agencies’ missions and structures.3 For example, the Pentagon is naturally going to be quite different from NSF (it’s massive, has multiple agencies under it, conducts classified research, etc.), and some agencies may have somewhat different hypotheses about what should be improved (e.g., the National Institutes of Health (NIH) will likely focus its metascience efforts on replications, at least initially).
In a small, relatively centralized agency such as NSF, a metascience unit working in the Director’s office can likely collaborate with directorates without substantial challenges. In an agency made up of highly independent offices such as NIH, the unit might need a different approach — such as coordinating with liaisons in specific institutes or centers that are working with the unit. In the Pentagon, to be effective the metascience unit might need to coordinate across DARPA and the military research labs or decide to limit its remit to a specific part of the massive defense R&D ecosystem. In short, diversity between institutions implies diversity in metascience unit approaches.
As we wrote in our initial piece, we also don’t think every research agency needs a metascience unit. It makes sense that the first two agencies to launch metascience units are NSF and NIH (the NIH unit is likely to launch as the Office of Research Economics, Planning, and Analysis), both of which exist to fund and conduct research.
Question 2: Should metascience units give out grants and contracts or just conduct research internally?
Science agencies should be able to run internal experiments to improve how they structure, fund, and conduct science. One way metascience units could do this would be to award grants, contracts, or other transaction agreements for third parties to conduct the experiments and evaluations, especially when the unit lacks the internal expertise to conduct specific types of analysis. This has the added benefit of signaling information to the research ecosystem and creating an expectation that research results will be published.
However, we think external grants for metascience research should not be a unit’s primary focus, for three reasons.
First, managing grants and contracts is labor-intensive, and the skills required to do so differ from the skills required to do metascience analysis. Under this model, it would be easy for a unit to lose its analytic capacity as it outsources its thinking to other groups. Second, grants and contracts can be expensive, they are often quite slow to set up and execute, and contractors and grantees can lack sufficient context to provide useful analysis. Finally, providing grantees and contractors access to agency administrative data can be difficult due to protections under the Privacy Act of 1974.
Separate from the work conducted by metascience units, agencies should also fund serious external metascience research that has no direct bearing on agency decisions. This kind of academic research should be funded by other departments or programs, as is done at NSF and NIH. At NSF, the Social, Behavioral, and Economic Sciences Directorate’s research on metascience would not be appropriate to bring into the NSF Office of Metascience (because it doesn’t focus on NSF’s performance). Similarly, at NIH, the Common Fund might fund external metascience research, but it is more appropriate that NIH’s proposed Office of Research Economics, Planning, and Analysis conduct the internal research on NIH programs. The structure of those two functions can look very different, but we need to fund both to grow our metascience understanding.
Question 3: How independent should metascience units be? How can they be incentivized to share bad news?
We heard some concern about the possibility that metascience units might keep uncomfortable or embarrassing evidence quiet. Why would an agency want to share information that makes it look bad?
There might be tension between independence and leadership relevance. If units lean too far into independence from agency political leadership and responsiveness to Congress, they might function largely as compliance offices — checking boxes on agency performance in response to Congressional requirements or inquiries without being useful to leadership decision-making. Conversely, if they’re less independent and function solely as tools for leadership, they may become politically captured and will have little incentive to share uncomfortable findings.
It’s challenging to define the appropriate level of independence for metascience units, and we don’t have a foolproof answer. There are, however, examples of offices where the evaluations are commissioned by leadership but released without leadership approval. Government statistical agencies release publications on pre-announced schedules, without prior review or approval by agency leadership. As an example of how this can look, the US Energy Information Administration makes requests from Congress and Cabinet secretaries public and then publishes the findings. Metascience units can also partially model the authorities of Offices of the Inspector General (OIGs), where the office has statutory independence, a right to access agency data, and freedom to publish findings.
These examples are not perfect analogues, however. A key difference between statistical offices and metascience units is that metascience units should provide policy advice to agency leadership. It’s often ambiguous what analytic evidence implies for decision-making and, since metascience units are explicitly designed to inform agency decisions, the units should both produce evidence and share perspectives on the implications of that evidence. There is also a critical distinction between any comparison of OIGs and metascience units: today most OIGs focus on ensuring compliance with rules via audits and investigations; metascience units must focus on using research to improve processes.
To move toward balancing responsiveness and independence, we think units could adopt (and Congress could authorize) the following transparency measures:
Publishing annual work plans, with major studies preregistered before they begin.
Releasing findings without agency leadership review.
Transmitting published analyses to congressional authorizing committees and sharing them with the public.
Building a strong norm that the agency head will ensure that the metascience unit can access vital information without difficulty or delay.
In addition to Congress authorizing institutional independence, the White House can also encourage publication of metascience unit findings. Indeed, in the Science: A New Golden Age report, OSTP suggested that metascience results should be public. It’s possible (fingers crossed) this kind of transparency could be a point of executive and legislative branch agreement.4
Question 4: How do you build a metascience unit that lasts, and does it need to?
Metascience units should last long enough to (1) produce a body of evidence about interventions that meaningfully change outcomes, (2) build evaluation and data infrastructure, and (3) diffuse a culture of experimentation within science funding programs. The minimum time to do this is probably around 10 years given how long it takes to connect science funding choices to outcomes. To reach those milestones, metascience units will have to overcome the four failure modes of similar projects that Jenn saw in the federal government: cycles of responsibility, change of administration, unsustainable funding models, and rejection by the agency.
Cycles of responsibility: When you zoom out over decades, where the government places the responsibility for experimentation and process reinvention follows a cycle. At the beginning of the cycle, innovation is everyone’s job. This continues until leadership comes to realize if it’s everyone’s job, it’s no one’s job. That realization activates the next phase of the cycle. Innovation capacity becomes centralized and made available as a service to other offices.5 This continues until leadership comes to see this capacity as unnecessary overhead, especially if innovation units don’t have strong champions within other offices. Then the cycle starts all over again. This may be inevitable in the very long term, but better institutional design can give the unit a longer lifespan and make it more effective during that time.
Change of administration: With every new administration comes new or changed priorities. If one initiative becomes associated too strongly with a prior administration, it will be more vulnerable in a new administration. This can be counterweighted by avoiding politically driven topics, authorizing and appropriating funds through Congress in a bipartisan fashion, and building buy-in from senior career staff at the outset, since they’re the ones who carry good work from one administration to the next.
For metascience units, political appointee involvement would give agency leadership more ownership but would leave the unit more exposed when administrations change and would make institutional independence (discussed in question 3) impossible. For these reasons, we think metascience units should not be helmed by political appointees but instead by a strong career director paired with metascience experts who set agendas on both near-term and decadal time horizons.
Unsustainable funding models: Reimbursable shared service models — where a unit provides a service within the agency or other agencies in exchange for financial reimbursement — can exacerbate the temptation to end centralized capabilities, especially when the units aren’t fully self-sustaining. For example, 18F (of the General Services Administration) and the Lab (at the Office of Personnel Management (OPM)) provided government-wide shared services, the former for building, buying, and sharing technology products, and the latter for providing federal workers with 21st-century skills in design-led innovation. While they both had numerous high-profile successes, when the General Services Administration and OPM cut non-essential functions in 2025, these units weren’t perceived to be central to their agencies’ mission and weren’t a profit center, so they were shut down.
Metascience units should not run on a reimbursable model but should have their own appropriation that funds the metascience expertise, control groups for evidence building, and the evaluation and data infrastructure. To support sustainability and independence, budget requests for metascience units should reach Congress intact as required under statute for OIGs (5 U.S.C. § 406(g)), to avoid a situation where political leadership at the agency or the Office of Management and Budget deprioritizes their functions.
Rejection by the agency: Functions that are perceived as compliance requirements or activities that aren’t central to an agency’s core purpose and that “tax” other organizations’ budgets invite pushback from the agency. Without internal champions outside of the unit, no one will be advocating for it whenever pushback arrives. So unit designers must be thoughtful about structuring units to cultivate internal champions. They can do so by explicitly serving three customers: Director/Administrator priorities, directorate/center priorities, and a bottom-up channel from program directors.6 By creating room in the portfolio (ideally the majority of it) for priorities that come from directorates/centers and program directors, units can achieve longer-term sustainability.
Question 5: Who should work in metascience units? What skill sets are needed?
Metascience units will benefit from a mix of permanent civil service staff and academic researchers who might serve in government via the Intergovernment Personnel Act Mobility Program. To meet research needs, agencies should recruit members of the six camps of metascience, depending on the agencies’ specific focus areas. We are partial to innovation economists, but these might not be the most needed profile everywhere. For example, if NIH is focused on replications, people from the Research Integrity camp might make more sense. As discussed in question 7, units shouldn’t just focus on quantitative research and so will need experts who are comfortable with different types of social science research.
Finally, metascience units also need people with expertise in agency operations. It can’t be all detailees and rotators from academia or nothing will get done. You want nimble bureaucratic ninjas.
Our suggestion for metascience unit leadership: have a metascience unit director who is a career staff member, with a deputy who is an external research leader. Having a career staff member leading the unit provides the continuity and institutional knowledge needed to ensure the unit can effectively work with agency leadership and across agency offices.
Question 6: How should metascience units engage with evaluation offices and other agency structures for collecting data? How can metascience units get other parts of their agency to participate in experiments and analysis?
Multiple friends mentioned that metascience units had commonalities with evaluation offices and asked how they were different and, more importantly, how they might work together. This question makes sense: while program evaluation was already integrated into many government programs, the Foundations for Evidence-Based Policymaking Act of 2018 (widely known as the Evidence Act) required agencies to develop evaluation plans and agency learning agendas. In response to this act, many agencies staffed evaluation offices that conducted analysis on program implementation, structured and disseminated agency learning agendas, and reported to Congress on their activities.
We agree that there are similarities between metascience units and evaluation offices, but there are important differences. For example, evaluation offices tend to spend most of their time evaluating whatever their agency is spending the most money on. This makes sense, but the consequence for a place like the Department of Health and Human Services is that most of the emphasis is on evaluating things like Medicare and Medicaid and not research funding. By comparison, metascience units should be relentlessly focused on analyzing the effectiveness of science funding approaches and identifying potential improvements.
There are many lessons from evaluation offices and Evidence Act activities of the early 2020s as well as lessons from evidence translation to policy that should be applied to making metascience units successful. Many of those have been incorporated into our recommendations.
Further, some agencies likely have evaluation office capacity that can help metascience units get started. Evaluation offices have different levels of capacity; some primarily serve as compliance offices while others have inculcated a culture of learning in their agencies. Those that have been effective in conducting internal analysis could provide helpful support to fledgling metascience units.
One thing we want to reiterate from our original metascience units article: where the unit sits within an agency is important. As we wrote in response to an OSTP Request for Information, metascience units should be placed in leadership offices “to ensure cross-agency visibility and protect against capture by any particular program.” That’s the plan at NSF. The issue with a unit placed several levels down is that it will pick up incentives and culture from wherever it sits. For example, one grouped with offices conducting compliance or performance-reporting work will tend to think in terms of compliance and performance reporting. Which leads us to the second part of the question.
Getting other parts of agencies to participate in metascience analysis is crucial and can be one of the weak knees of an evaluation effort. Agency leadership buy-in might be necessary but not sufficient. So how can the units get other parts of agencies to cooperate? As was noted in the Science: A New Golden Age report, metascience units can have authority to require other parts of the agency to collaborate on research and the funding to encourage them to expand projects that might cost extra money.
To be useful to the agency — and not be perceived as solely extracting information from agency staff — metascience units must be responsive to bottom-up questions and project ideas. We’re not recommending a figurative suggestion box that nobody uses. Rather, metascience unit staff should conduct outreach across their agency, provide readily accessible avenues for offices to request help, and ensure that they respond to all requests. For example, an agency office might request help in figuring out the right structure for “golden tickets” — which allow individual peer reviewers to advance a project for funding even when other reviewers disagree — in its external review processes and analyzing the effectiveness of the golden-ticket approach in selecting novel projects. In short, while metascience units analyze the effectiveness of science funding across an agency, they should be obviously useful to the offices that are funding the science.
Finally, having centralized access to data will be hugely important for making coordination with offices across their agencies effective (and less painful than it otherwise might be). Metascience units should be able to access grant and contract data without having to constantly harass different offices. Agencies therefore need centralized repositories of agency-wide grant and contract data.
Question 7: What sorts of evidence should metascience units focus on? Should metascience units conduct qualitative analysis?
Much of the impetus for launching metascience units is that they can conduct quantitative analysis of agency-level data that is useful for decision-making. That is often done via simulations based on real data, historical analysis of data, and randomized controlled trials (RCTs). RCTs are a gold-standard approach to demonstrating causation, but they are also difficult to pull off in government programs due to challenges with (or staff resistance to) randomization and getting sufficiently large sample sizes. These experiments should be designed primarily for questions that are “big if true.” They must have enough statistical power to identify meaningful results — that is, detect realistic changes that leadership will care about.7
Metascience units are not RCT units, though. Small studies can help in making the case for bigger studies, and simulations can often provide significant insights while being much easier than RCTs. Finally, we think there is an important role for case studies. Especially with questions of institutional design or reform, quantitative data will miss crucial details, and the sample size might be too small to reach statistical significance.
As an illustrative example, NSF recently announced the X-Labs program, which intends to fund a small number of large labs to conduct different types of frontier research. Due to limited sample sizes and differences between grantees, it would be unrealistic to study X-Labs via an RCT. But we do need high-quality information about what is working well about the project and what can be improved, both so that NSF can make changes and for future government programs that might be modeled on X-Labs. Studying programs like X-Labs well will require qualitative or mixed-methods approaches.
Question 8: What should metascience units produce, and how should their findings change what the agency does?
Metascience units are designed to conduct research that helps agency leaders make decisions about how they structure, fund, and conduct science. In pursuing their mission, metascience units should produce analytic products — research papers, policy briefs, etc. — that provide evidence about how effectively an agency is funding and conducting scientific research. But a metascience unit can still fail if the evidence it produces does not influence decision-making.
As a recent piece by Leya Mohsin says:
Rather than assuming the founding design is correct and static, new institutions have a rare opportunity to build in real feedback loops and the flexibility to course-correct, treating the initial design as a hypothesis rather than a final answer. In the R&D context, proposed metascience units are a promising way to tackle this — but if, and only if, mechanisms are constructed to ensure that the evidence they generate has a meaningful impact on decision making.
What does this require? While we should have some humility about how even the most theoretically sound designs will work when they make first contact with a federal agency, we have some ideas about how to increase the likelihood of evidence informing decision-making.
Two approaches that we have already described could help. First, units should conduct research that is useful to agency leadership and take on projects requested by agency offices. Second, units should make concrete policy recommendations whenever they produce research findings.8
In addition to those, units can build capacity for effective communication between the metascience unit and program designers. This doesn’t mean having people whose job title is “communications.” Rather, metascience units should recruit staff with strong science communication and policy entrepreneurship skills — who can understand research findings and seize opportunities. Cultivating a learning mindset and prioritizing evidence-based iteration can also help substantially; if results are not informing decisions to the extent desired, what can the unit change to increase the likelihood of utilization?
We would like to thank friends who provided thoughtful feedback on the initial Science Agencies Need Metascience Units piece, including Kelly Bidwell, Amira Boland, Albert Bravo-Biosca, Erica Goldman, Kei Koizumi, Lauren Supplee, Christina Yancey, and others.
They are not the first globally, though: the UK government launched the UK Metascience Unit in 2024.
In addition to receiving direct feedback, we were happy that Zach Pirtle, Mahmud Farooque, and Michael Bernstein wrote their perspectives on what metascience units should do. Though it preceded publication of our initial metascience units article, Loren DeJonge Schulman and Leya Mohsin’s “What the Metascience Community Should Learn From the Federal Evidence Movement Before Making Our Mistakes” influenced our thinking on how to stand up metascience units as well.
Leya Mohsin recently published a helpful blog post on designing for function in R&D organizations.
From the Science: A New Golden Age report: “Their findings should also be published externally, building the broader evidence base on what works in science funding and creating accountability to act on what is learned.” At the Science: A New Golden Age Summit on October 8, 2026, Assistant Director for NSF’s Technology, Innovation, and Partnerships Directorate Erwin Gianchandani also said that NSF Office of Metascience research results would be published.
Or a burden on other offices, depending on how much they are forced to accommodate the new innovation unit.
A fourth customer is Congress, but that is not strictly relevant to the risk of rejection by the agency.
While some studies are too small to be conclusive, it may be possible to have multi-unit experiments at agencies where similar granting structures are used across units.
One way to ensure that research results inform agency decisions is to require agency heads to respond to metascience unit reports, as is the case for reports from an OIG. Our concern here is that metascience units might develop an adversarial relationship with agency leadership, which would be counterproductive.