Imagine you want to encourage more or better innovation. You want to create a new cure or understand an important social phenomenon. How do you make that happen? Funders have numerous approaches that they can use.

In July, IFP (with Kyle’s help) released the Atlas of Innovation, a website that helps policymakers and philanthropists navigate decisions about how to fund innovation.1 This lecture provides an in-depth complement to the site: Kyle delves into the different economic structures that underpin grants, contracts, prizes, and other funding approaches.

Subscribe now

In the image below, from Azoulay & Li’s paper Scientific Grant Funding — which Kyle discusses — the authors provide intuition on when we should think about using different types of innovation funding mechanisms. It’s worth digging into this chart on grants, prizes, patents, and contracts.

Grants: As described in the Atlas of Innovation, grants support “researcher-led work within a defined scope.” They are funds generally provided to researchers with limited strings attached and an expectation that the research will produce public benefits. According to Azoulay and Li, grants are useful when the innovator shouldn’t or can’t capture the value created through their research and the output desired is open ended.

Prizes: A prize is “a competition that rewards teams for meeting a defined innovation target.” Funding could be provided for reaching a target like solving a mathematics problem or creating an important new technology. Like grants, they are most appropriate when the innovator shouldn’t or can’t capture the value created and where there is a public benefit.

Patents: Patents provide “firms a period of market exclusivity;” they incentivize innovation by protecting innovators when they produce profitable ideas. In this case the government is not providing innovators money, but rather is protecting them from competition. Patents are best when the government doesn’t have a desired end-point (e.g., they don’t specify priority technologies to be patented) and the innovator can and should capture the value created.

Contracts: “A contract is an agreement in which a funder contracts with a specific performer, often a company, to conduct R&D.” Contracts are like prizes in that there is a known end-point — in this case the thing the performer is being contracted to do — but the outputs of the research will be captured by the funder or innovator.

How science is funded is a critical area of study if you’re interested in making government-funded research more effective. Often, federal agency staff are unaware of the tools they have at their disposal, as well as when they might be more or less appropriate to use. This gives them — and you! — the information needed to effectively fund research and innovation.

Subscribe now

Here are the readings that go with this lecture:

• Carnehl, Christoph, Marco Ottaviani, and Justus Preusser. “Designing Scientific Grants.” Entrepreneurship and Innovation Policy and the Economy 4.1 (2025): 139-178.

• Hegde, Deepak, and Bhaven Sampat. “Can private money buy public science? Disease group lobbying and federal funding for biomedical research.” Management Science 61.10 (2015): 2281-2298.

• Myers, Kyle. “The elasticity of science.” American Economic Journal: Applied Economics 12.4 (2020): 103-134.

• Acemoglu, Daron. “Diversity and technological progress.” The Rate and Direction of Inventive Activity Revisited. University of Chicago Press, 2011. 319-356

• Carnehl, Christoph, and Johannes Schneider. “A quest for knowledge.” Econometrica 93.2 (2025): 623-659.

Subscribe now

Thanks to William Higbie and Harry Fletcher-Wood for their support in producing this video and the transcript.

Subscribe now

Lecture Transcript:

The Direction of Science

My name is Kyle. I am an associate professor at the Harvard Business School, and today we’re going to be talking about the direction of science. Some fun stuff. I like to show graphs with things not labeled and have people guess what’s going on.

We’re going to be talking about the direction of science, and I want to be clear about what direction means. You will already have heard from Ben and Chad Jones, the wonderful Joneses. Soon Matt Clancy will talk with you. A general message of a lot of that work is that on average, when we put money into this thing called R&D, or science, or innovation, a lot of great stuff happens. Today is going to be a lot about — conditional on putting a dollar into this magical growth engine, there are lots of specific things to invest in. Where that money goes is what I mean when I say “the direction of science.” We’re usually interested not in the level of investment across all areas but the ratio of investments in different areas.

Reading the direction of science through publication data

To give you a sense of what variation in the direction of science can look like through one lens, I’ve got a few graphs here of publication shares. Publication metrics over time are always hard to interpret. You should take any trend line you see with publication data with a large dose of medicine. But I’ve got a few here. This first one is five different categories inside some type of science. I’m wondering, does anyone know what these topics might be, looking at the trends?

This is energy science.

You could see a very interesting boom in research on natural gas in the tail end of the 2010s and around ye olde epidemic. There were some interesting technological developments with respect to fracking and what it could do in good ways and also possibly in bad ways. You can see, as it expanded, the different areas that it ate into.

Here’s another one that also has a bit of a surge right around 2020.

You’re probably thinking it’s one thing — this is computer science.

That surge is not COVID. That’s large language models. If you broadly define artificial intelligence, very broadly speaking these are five things going on over the past 50 years. You can see the large language models — there were people talking about them back in the ‘70s and ‘80s, but it wasn’t until very recently... You can see they’ve totally dominated, in particular symbolic AI. Symbolic AI is, as opposed to the stochastic approach of most large language models, symbolic AI was a — “We’re going to hardcode a very mechanical, deterministic system and see if we can use that approach to develop intelligence.” It hasn’t faded completely away, but you can see most people these days work on LLMs.

Now this is the one that you probably thought the other two were.

This is a look at disease science over that same period.

You can see there are two interesting things in this data. One is the HIV/AIDS epidemic occurring in the ‘80s and drawing a lot of resources, resulting in a large growth in the publication share. These are shares within these different categories — this isn’t shares across all of science. Then you can see, unsurprisingly, the research on coronaviruses and COVID-based diseases spiking with the pandemic, and still a lot of research going on in that space today. This data goes up till about 2023, where it’s nice and reliable.

From outputs to inputs

Those were outputs, looking at the direction of different kinds of outcomes: publications. All the money has been invested, the papers have been written. A lot of what I want to talk about today is the input side: where do we put our investments, and what do we mean when we talk about managing the direction of science?

One way to think of that is investment in a very particular thing. For example, the ITER fusion reactor in Europe is a joint venture that started in 2006. This is my favorite example of a large-scale targeted research investment that is not going quite according to plan. Twenty years into this project, we’re now 15 years behind schedule and about €15 billion over budget. This is an emblematic example of the difficulties of knowing how much things are going to cost in the world of frontier science and managing that.

Now I’m going to tell this in a way that gives you a much more optimistic lens on intervening in the direction of science, although in truth I make no causal claims with this graph. But it is a fun graph. In 1960, 1970, the National Cancer Institute (NCI) did exist but was very weakly funded, on the scale of maybe $100 million per year, and cancer mortality was on the rise. In 1971, Richard Nixon starts the war on cancer, and it leads to a large and sustained boost in funding for the National Cancer Institute. Was the war on cancer winning? Unclear. We kept spending a lot of money.

In 2008, David Cutler, whose approach I’m drawing very much from, wrote a very nice paper that says, Are We Finally Winning the War on Cancer? The beautiful graph he has, which I’ve replicated here, shows a dramatic decline in age-adjusted cancer mortality in the United States.

How much of that is purely due to lung cancer survival rates going up because people are not smoking as much, and how much do you want to attribute to the NCI? These are tough questions. Your eyeball wants to give it all to the NCI, but also — I don’t know, it started to pay off after 30 years. Maybe this was a large portion of it. We don’t know. A lot of the frontier questions people are asking right now are: what are the returns on investments in these targeted programs?

Spending at the NIH

We’ll spend a bit of today on the US National Institutes of Health (NIH), both because it is large dollar-wise and because it’s a place that has some empirically useful features for studying.

This is the share of the budget going to each of the major institutes and centers of the NIH as a fraction of the total NIH budget. You can see some interesting trends and changes there in terms of where our tax dollars are going.

There have been a lot of papers where people take that data and plot it against something like disease burden. That comes from a notion that, even if you’ve never taken an econ course, your brain sees that spending and says, “I hope we’re spending money where there’s a lot of need for scientific advances.” I’ve drawn a 45-degree line, which represents what it would look like if, for every 1% increase in burden that these diseases have on the US economy, we would spend 1% more dollars on that disease.

Now I’m going to show you the data.

The real elasticity is not one. It’s 0.41. It’s positive. We do spend more money on diseases with larger burden, per WHO disability-adjusted life years, but it’s not a 1% increase.

What should it be? These are interesting and difficult questions, and the kinds of things we’ll be talking about in our lecture today. We’ll come back to this graph at the very end, thinking about — if we took the World Health Organization estimates as the true measure of disease burden, what is the source of the residuals from this line? Why is musculoskeletal so far below the average trend? Why are genetic disorders, or aging and dementia, a little bit above? Is that due to distortions or frictions in the political budgeting process, or is it due to research on dementia being very productive right now? Another reason you will invest in one topic and not another is because every dollar you invest gets you more. So holding fixed demand, you always have to think about supply: the classic econ challenge of separating supply and demand curves.

Outline for today

We’re going to proceed in three parts:

1. A very high-level, institutional-level focus on how budgets are set, as far as we can understand them.

2. A more micro-level look, thinking about scientists’ response to decisions made by scientific institutions.

3. A discussion of some frontier theories about how this works. As evidenced by those empirical charts I showed you before, doing empirics on some of these questions is quite challenging. It would require a counterfactual not war on cancer. For those of you who are empiricists in the crowd, it’s hard to find the right control group for the war on cancer. This is a place where we do need to try to rely on theories when we can.

Who determines the direction of science?

A shout out to Bhaven Sampat, who introduced me to these two fellows, one of whom is the author of the first paper we’re going to discuss. On the left is Vannevar Bush, and on the right is Harley Kilgore.

Vannevar Bush was the director of the Office of Scientific Research and Development during World War II and wrote a very famous memo called Science: the Endless Frontier that founded the National Science Foundation and laid the groundwork for a lot of science policy in the United States. He was a scientist himself at MIT, and as most scientists will say, his answer was, when in doubt, just give the scientists the money and let them figure out what to do with it. That was his answer to the question, who should determine the direction of science?

Harley Kilgore was a senator from West Virginia, a very sharp man who saw what was happening after World War II as the NIH and the NSF started to scale up and thought, “Not so fast, Mr. Bush. Perhaps the public and our policy institutions should have a bit more sway over where this funding is going, because you scientists like to do things that you like to do, which may not be in line with the public’s best interest.” The Bush–Kilgore debates were rampant in the ‘40s and ‘50s, and in today’s political climate have come back into popularity.

Disease group lobbying and the NIH

The first paper that I want to spend a little bit of time discussing was this nice paper by Deepak Hegde and Bhaven Sampat, Can Private Money Buy Public Science? Disease Group Lobbying and Federal Funding for Biomedical Research, which is still one of the better lenses looking at how much of the scientific funding in the United States gets influenced by political forces — in this particular case, disease group lobbying.

Before this paper, and even still now, a question is: how much does lobbying matter in practice? If you are interested in these political economy questions, there’s a terrific paper by John de Figueiredo and co-authors that has a terrific title, Why Is There So Little Money in Politics? which gets at the question you can see in my numbers here: the government disperses a tremendous amount of money.

The NIH budget these days is approaching $50 billion. And yet — these numbers were based on me working with Claude for a few hours to try to figure out the most up-to-date numbers about the size of disease lobbying — we got to an estimate of about $30 million. That’s many orders of magnitude different. But an interesting question is always, perhaps the returns to those dollars are very steep at first and then become very flat, and so we still want to know: are those dollars influencing where our public investments are going?

Before we jump into it, it’s important to think about how those dollars could influence. We’re a lobbyist. We pay money and spend time with the senator and tell them that the disease we are interested in is very important and more resources should be allocated to it. How could the senator from West Virginia influence that?

One way is by setting the budgets of the institutes and centers, which is an explicit mechanism that Congress has control over. They say the NCI budget is a certain amount of money. Another is they could try to fund specific initiatives. In the case of the NIH, this is actually quite rare. Most of the congressional authority comes through the institute and center (IC) budgets, although there are instances of some congressionally mandated set-asides. I believe Type 1 diabetes is one of these, which for you PhD students out there — I tried to look at that program because it has some interesting budget shocks. I never got too far, but that might be something interesting to look into, because it is one of the few congressionally mandated set-asides.

Then there are also these things called soft earmarks. This is clipped out of the 2024 appropriation language for the National Institute of Neurological Disorders and Stroke.

At the top, you can see the line item for this institute, and that’s the big thumb on the scale that Congress has power over. The challenge is, at a scale of multiple billions of dollars, it’s hard to move that number around.

Underneath these things, if you look in the appropriation language, you’ll see many paragraphs that look like what you see here, which will start with some disorder that Congress claims to be interested in. It’ll say things like, “The committee is concerned that one out of every fifty individuals has a brain aneurysm.” This is the soft earmark. It’s called soft because there’s nothing about this where Congress is saying you have to spend x amount of dollars on something. But it does go into the legislative record, and the NIH for sure is aware of these things, and the staffers on the Hill make sure the NIH is aware of them.

They have a number of regressions that I’ve done my best to set up as a nice little directed acyclic graph that we can just walk through, to see what this paper is finding is happening in the data. This is a three-stage event.

First, for every disease, you look out in the world and there’s some burden that it has on the population, some scientific opportunity, which we’re going to proxy with the number of publications on that topic, and some base rate of funding already going into that disease. The first question is: how do those things affect how much lobbying occurs?

I’ve reported the coefficients here. These are all logs, so you can roughly interpret them as elasticities. They’re small but non-trivial. That 0.05 means that for a 10% increase in disease burden, there’s a 0.5% increase in lobbying spending. So there is more lobbying spending when these things go up, but there’s plenty of variation due to the residual there as well.

The next question is: how do all of our three states of the world and lobbying affect the rate at which we’re seeing these soft earmarks?

You can see there is some direct effect from disease opportunity and prior funding. Prior funding is effectively zero. There’s a clear effect from lobbying. I did my best to convert the estimate into an elasticity. It’s usually on the scale of 0.1 or 0.2, depending on the numbers you use from their data. That means for a 10% increase in lobbying, there’s a 2% increase in the count of these soft earmarks in the NIH.

They’re soft. They don’t legally require the NIH to do anything, but it doesn’t take a savvy political scientist to understand how this might influence the NIH. The third and most interesting stage in the paper is thinking about how soft earmarks then themselves affect grants.

A fun thing I’ve shown with the zero on the bottom is that once you condition on soft earmarks, lobbying has no effect on the amount of money going to a particular disease. I wouldn’t call that a state-of-the-art approach for a mediation analysis given modern techniques, but it’s a good first-blush look at saying, “If lobbying does anything, it certainly looks like it does it via soft earmarks.”

The first interesting number there is the 0.05 on the all-dollars, which says that the soft earmarks have some pretty small bite on all of the funding. But if you zoom in on these requests for applications and program announcements — which is when the NIH itself sets aside some money for research on a particular topic — you see a much larger effect of the soft earmarks on the money going through those stores.

The most interesting question here is how this affects science. Unfortunately, I have to leave that in my little dashed gray box, because we don’t really know. I’ve summarized the questions here.

One of my favorite things this table does is break down their analysis into a bit of a decomposition, so that in normal English we can understand just how much the political lobbying going on for disease advocacy is influencing the NIH budget.

They break out the chain of events: the average amount of lobbying, how many earmarks there are in total, how many we think are due to lobbying, the total amount of money going out the door, and how much of that we think is due to the earmarks. We come up with an important number to have in our heads, which is that about 8% of the NIH allocations are due to these earmarks, with about 40% of the earmarks appearing to be due to lobbying. This is a great number to know, because my prior before reading this is that you could talk yourself into a lot of different numbers. This is a really good first step in understanding what’s going on.

A global public good

Another related paper I really enjoyed by Margaret Kyle, David Ridley, and Su Zhang, Strategic Interaction Among Governments in the Provision of a Global Public Good. It’s related to the findings we just saw with the Hegde–Sampat paper. I’m just going to read you the abstract, because it’s also a wonderful exercise in a well-written abstract that tells you everything you need to know:

“How do governments respond to other governments when providing a global public good? Using data from 2007-2014 on medical research funding, we examine how governments and foundations in 41 countries outside the US respond to funding changes by the US government.”

The idea here is, what does Panama do when the US increases its spending on malaria-related research? Getting to the identification challenge, they have a pretty nice approach where they do something similar to the Hegde–Sampat: they say, because funding is positively correlated with things we can’t see, we’re going to use the representation of research-intensive universities on congressional committees as an instrument. The idea being, if a senator from Wisconsin gets onto the Health and Human Services committee, and Wisconsin has a lot of people that want to study malaria, and we get more money into malaria research in the US, it’s only because that senator rotated on, and not because malaria science became more powerful, or the demand for malaria science grew.

They have a wonderful elasticity in the abstract that’s good to know, which is that for every 10% the US spends on a disease, there’s a 2-3% reduction by the other governments. This is a classic substitution — you see versions of this in many different sectors outside of science, but it’s very important. Science is a global public good. This is a challenge of managing the direction of science: lots of other countries will benefit from their spillovers. “The US government is double-funding something” — that might be a pessimistic take. An optimistic take might be, “In the long run, we’re attracting talent to our country, so the returns to scale are actually increasing.” There are lots of interesting debates around that. But at least in terms of the short run, this is a very important paper.

More generally, because of the challenge of studying how the resources get determined, there are a lot of very interesting open theoretical and empirical questions about how these decisions are made at institutions like the NIH or the NSF, where I think there could be lots more work to be done.

Which scientists do we fund?

For the rest of this, we’re going to be working through the lens of, “Let’s all assume we’ve agreed we’ve got to spend $10 billion on cancer research. Which scientists do we fund? How do we give them the money?” All of those kinds of questions.

Before we get into grants specifically, another terrific review paper is by Pierre Azoulay and Danielle Li, Scientific Grant Funding. It starts with a step backwards in the chain of thought and says, “When and why might one want to use grants?” which is the bread and butter of what you see many public scientific institutions using, “As opposed to the three other mechanisms you might consider when trying to incentivize innovation.” They have this very nice little diagram that helps you think about when these different mechanisms are optimal in theory.

Grants are great when the question in question is very open-ended and hard to describe, in terms of the nature of idea search and in terms of appropriability. By appropriability, we mean, if the scientist is going to create $100 of value in the world with their discovery, how much of that value will they capture in some way? There are certain ideas whereby, just by the nature of that idea, the scientist will never be able to capture much of that $100 at all. That leads you to a world of grants, where you’re going to give the money up front and say, “Please do something nice for the world.” That please will have a lot of restrictions and rules and regulations around it, but that’s the world that gets you into grants.

We’ll come back to grants in a little bit. I thought it would be worthwhile working through the other mechanisms as well and giving some examples of what they look like, because in a full portfolio of managing the direction of innovation, these other things are very important. The wall of cites is me just trying to put some stuff there so you can see a few references to pursue if any of these mechanisms look interesting to you.

Prizes and contests

Prizes or contests are where we’re going to, ex-ante, specify a reward that we’ll give to somebody if they achieve a certain outcome. This obviously requires the output to be verifiable. The challenge here is going to be classic multitasking trade-offs to be balanced, which is, you better be sure you are incentivizing task A to get people to do A, because if you say we’re going to pay for A, but everybody can only actually do B, or you actually want B, you’re going to get A and not B. It turns out, when you think about frontier science and new ideas, the whole idea is you don’t know the solution to the question, so it’s very hard to ex-ante write down a description of something that can be verified ex post.

Where does this work? You see this in things like the Netflix Prize. This is a verifiable thing, where Netflix said, “We want to improve the mean squared error,” or whatever fit metric they had, “of our algorithm. Here’s all of our data. Let’s see if someone can come up with a better algorithm.” That’s verifiable, because you just submit your algorithm, run it on a benchmark set of data, and there’s no debate as to who wins the prize.

One interesting thing, which I’m always a bit cautious to talk too much about as an economist and not a political scientist, is that these ex-post rewards open the room for broadly construed politics, or risks of things that are a little bit harder to handle. In the Netflix case, when they tried to run it again, they wanted to publicly release the data, but it turns out there was some concern that the data they had released would actually allow the public to de-anonymize and identify individuals.

Another great example people often use is the famous longitude rewards, as an example of a successful prize in science. They were both successful and a good example of what can go wrong with prizes. John Harrison was a watchmaker in England, and he was the wacko who thought he could measure longitude just with clocks. The board of the longitude prize was mostly comprised of astronomers, who all assumed that, “Whoever wins this is going to come up with some new fancy way of using the stars to figure out longitude,” which would be a variant of how sailors had been navigating up until that time. So John starts sending in his chronometers that are crushing the competition when it comes to measuring longitude, and the board of astronomers kept moving the goalpost, asking more of him, and delaying his payments. It’s something like 80 years after his first chronometer was submitted that he actually got his final payment. That difficulty in committing to the payment is another political concern of using things like prizes.

Research contracts

Research contracts are kind of like a prize, but it’s a prize on the input side, which is saying, “If you put a certain amount of inputs in a certain direction, regardless of what happens, we will pay you for it.”

You have the same multitasking trade-offs; they’re just realized at the input stage. The challenge here is that you have to be very clear about the inputs. We have to have verifiable inputs, and the person using this mechanism has to know exactly what needs to be done.

A good example of this is ARPANET, the origins of the internet. After ARPA had the idea of how they were going to try to build the first internet, they needed to make these things. They’re called IMPs, but it’s basically a router — the machine that would stand in between two computers to send the signal back and forth. They had enough sense of how it should be done, but not quite the resources. So they developed a research contract with the company Bolt, Beranek and Newman, who’s now part of Raytheon and is a legendary company within the history of the internet. Fun fact I learned developing the slides: the first message they sent was “LO,” because they were trying to send “login,” but three letters crashed the system, so they had to do a few hours of recoding to get it to work again.

An example of leaning into research contracts when a little too early was in the 2000s. The Department of Defense, specifically within the Army, had a very big push which they called Future Combat Systems. As this quote from a RAND report afterwards indicates, they might have embraced this mechanism before they had a clear understanding of what was technologically possible. Writing contracts over things you don’t understand, unsurprisingly, can have some downsides.

Intellectual property

Last up before we get to grants is intellectual property rights writ large.

You’re probably thinking of patents as the default thing to think about here, but when we’re talking about IP in general, it really is just: someone has ownership over the idea and can do with it whatever they want, including selling rights to use that idea. These are really interesting things, in the sense that it’s going to direct people’s output — they’re going to shift their effort towards things that are well defined and have legally defensible boundaries, whether it’s via trade secrets, contractual licensing, or patent licensing.

The big challenge here is balancing static and dynamic trade-offs. The static positive incentive is clear. If I know that when I invent the next magical drug, anyone who wants to use it has to pay me, that’s a very large incentive for me to go after high-market things. The downside is that if I invent something that is very important for a lot of other people to invent other things, then there might be a dynamic friction whereby I would extract so much rents from other people who want to use my new invention for their new inventions that it would actually be suboptimal to give me full property rights over that idea. Balancing the current incentives for innovation versus the dynamic incentives for cumulative innovations is at the core of what a lot of these theories and empirics are getting at.

Grants

Now let’s start to get more into grants, which are the bread and butter for a lot of the frontier basic science where appropriability is very hard and the definition of ideas is very nebulous. The best we can do is going to involve a lot of frontloading of resources before anything verifiable comes out.

In terms of the trade-offs here, there’s good evidence from a lot of different papers that these things do work in practice. One of my favorite more dramatic examples is by my colleague Ina Ganguli. She has a wonderful paper, Saving Soviet Science: The Impact of Grants When Government R&D Funding Disappears, that showed how, during the fall of the Soviet Union, there was a push from George Soros to give a lot of money to Soviet scientists to try to help them continue doing work in science. She has access to the very simple algorithm that defined who got the money. There’s a very nice regression discontinuity for those of you who know what that means — a really good treatment-control setup. She shows that these grants, which were about a year’s worth of salary, were very influential in keeping scientists in the scientific profession and publishing during this time.

The downside here — another political-risk undercurrent — is that when you’re giving people money to do things when you don’t really know what’s going to happen, it’s easy to grab the title of these efforts and make it sound like wasteful spending. I have three examples here of things that, in different avenues, certain people would probably think is a poor use of our taxpayer dollars. All three of them are super important ex-post.

• Horseshoe crab blood is now the key way we test anything we inject in humans for safety before it goes out the door. During the pandemic horseshoe crab blood was one of the things that got short very fast as we were trying to ramp up production of vaccines.

• The GLP-1s that are taking the world by storm: one of the origins was researchers studying Gila monsters. It turns out Gila monsters don’t eat for a very long time, and the question was just, “How do they not eat for so long?” Ex-post you can see, “That’s going to prove useful for managing humans and their diet.” But in the moment, it’s the kind of thing that my four-year-old would just be like, “Let’s talk about Gila monsters and why they don’t eat so much.” It sounds a bit silly.

• Another one is, “We’re going to use a very expensive scanning electron microscope on coral reef just to look at it.” It turns out we learn how to make better bone grafts. These sorts of stories are all over the place.

That’s not to say there’s not wasteful spending, which is something we’ll come back to at the end. If you’re interested in this sort of thing, the Golden Goose Award is a wonderful website where they give out fun awards to scientific projects that have this flavor of seemingly silly-sounding but ultimately very influential research.

The granting enterprise

Let’s get into the design of the granting enterprise. I can’t recommend enough this paper, Designing Scientific Grants, by Christoph Carnehl, Marco Ottaviani, and Justus Preusser. It’s a terrifically written review that gives you a really good overview of the state of the predominantly theoretical art when it comes to how economists are thinking about designing scientific grants. I’m going to move through this and talk about a few things they cover.

One thing they do a good job of emphasizing is that this is a multi-step process, and decisions made further down that process will ripple back up through.

Seemingly trivial-sounding things, like “How long do we let people use the money in their grant?” can cascade backwards and ultimately influence the types of scientists who are applying for different types of grants. Keeping those dynamics in mind is very important.

They also have — partly because Marco is a co-author of both this terrific paper as well as the review — a really nice compressed-lesson version of this paper by Jérôme Adda and Marco Ottaviani, Grantmaking, Grading on a Curve, and the Paradox of Relative Evaluation in Nonmarkets. I’m going to walk through that now, because I think it’s really cool intuition. This and a couple of the other theoretical models I’m going to talk about are going to be slightly oversimplified in ways that are not the most technically correct version that’s in the paper, where you take everything very seriously. But it’s a good starting point, and not everybody needs to know the very difficult high-end details of how this theory works. The intuition delivers a lot of what’s important and can motivate a lot of interesting questions.

The first interesting thing is a fun result where perfect information actually leads a market to unravel. A classic result in economics is the paper by George Akerlof on the market for used cars. The idea being, in something that looks like a used-car market, if everybody is uncertain about the quality of vehicles, fewer people want to engage in that market, which means the only people that want to sell their used cars might have worse used cars, which means the people who might buy think everything is worse, which means everybody who sells only sells worse. That’s the kind of unraveling thought exercise that can happen in worlds where there’s imperfect information. This is going to push you to think about a world where unraveling can happen with perfect information.

The setup is very simple.

Let’s assume that every scientist has the same cost of applying and the same benefit if they get the grant. The quality that the funder is going to see is some noisy signal of the true quality of that application. Let’s say the funder is maybe a bit naive and is just going to say, “We’re going to fund the 20%, or 50%, or 10% best applications.” That’s the x̂. For ease of math, let’s say there are 100 scientists. Let’s say there’s no noise and that you see exactly the truth.

All 100 scientists see the announcement from the funder that only x̂ percent will win. Only x̂% of the 100 think they will win, so only 100 × x̂ will apply. But if 100 × x̂ apply, and the funder is naive and just sticking to their x̂, in this perfect-information world, only 100 × x̂ × x̂ will apply at this round of our iteration. You can see how, if you keep walking through this logic of how people’s expectations would update in this game, after that mathematical expression number of rounds, no one wants to apply.

Imagine that x̂ is 10%. With no noise 100 of us look out, and only the people who know they are the top 10 will apply, because they’re only going to fund the top 10%. Then they think, wait a minute, if only us top 10%, they’re only going to give an award to the top 10% of us, which would be the first person. Then the first person says, “I am but one person, so I guess maybe I will apply, depending on the rules. But if they’re only giving awards to the top 10%, then maybe I will not even apply, because I can’t submit a tenth of a percent of my application.” That’s the very stylized — not realistic at all — intuition of how, in this world of relative evaluation with perfect information, the market can actually unravel.

It turns out that a bit of information asymmetry keeps this kind of market from unraveling, because now people don’t know exactly their chances of winning, and that leads them to apply.

That sounds like, “Noise is in our favor.” It turns out noise is not exactly in our favor. I’m going to walk through this graphically using the really nice graph inside the review piece.

First, some fundamentals. The grant size, again for simplicity, let’s just call it 1. If you win a grant, you get a payoff of 1. Every person, in order to apply, has to pay some common cost, c. It’s a pain in the butt to write the application, and we all face the equal pain in the butt of writing it. But some of us have better ideas than others. Let’s consider a baseline setup where the funder is announcing this payline. Let’s say we can describe this system such that there’s this function W, which describes — for a given quality of an idea and the announced payline — the probability of winning. Then there’s some threshold θ^, which says, given the cost of applying, anybody with a θ above or equal to θ^ is going to send in their application.

That shaded blue area there is just the integral over W from θ^ upwards, and that tells you the number of applications you’re going to get. You can see it’s going upward, because people with higher-quality ideas are more likely to get funded. It gets concave eventually, because there’s some noise in the system, so even the person with the best idea has a chance of getting a bad draw. That’s why it doesn’t go all the way to 1.

Let’s do an intermediate thought exercise that’s not realistic but helps with intuition. Let’s add noise to the review process, but we’ll change the payline such that the same scientist, in terms of the quality of their idea, is on the margin.

That is to say, we’re going to add noise to our review process, but we’re going to change the payline so that θ^ stays the same. That threshold that determines who’s going to apply is still the same in this noisier world.

What’s going to happen here? First, what does noise do to the payoff function? It makes it flatter. You can imagine a world where I have a terrific idea and Zachary has a mediocre idea, but the noise is infinite, such that when you see the signal of our ideas, Zachary could get a gigantic awesome draw. Even if I get a pretty nice awesome draw too, the amount of noise in the system is still such that everybody’s like, “Oh my gosh, Zachary’s idea is great.” In the world where noise was enormous, we wouldn’t be able to tell; everybody would have the same win probability. That’s why that W̃ is getting a little bit flatter. But we’ve artificially kept the θ^ threshold the same, so the same person’s indifferent. Mechanically, what happens is we get fewer applications. That blue area is smaller, and that red area is all the applications we’ve lost. In this probabilistic world, what we’re losing is every person with these high thetas is just less likely to apply.

Then you say, “If fewer people are applying, you have more money. You could expand the budget.” That’s what we’re going to do next. We’ve added uncertainty, so we’re giving out fewer awards, which means we have budget slack. The funder naturally says, “Why don’t we increase the payline? We can fund more stuff.” This is where things get interesting. What happens when you bump that payline out? You also bump out the threshold that determines which scientists are applying. You’ve got to push X back into the low-quality distribution to get more people to apply.

What the funder is trying to do is say, “I always want to fund this many people. This is my budget-clearing thing. I’ve got to get my money out the door to scientists.” If we have noise and don’t change anything, we’re going to fund fewer people. We need to attract people to fill in the lost applications from this noise, and the only place to fill them in from is the inframarginal people, going backwards into lower-quality ideas.

If you increase the noise in a field, that will require lowering the payline to attract more people to apply, and the people you’re going to attract will by definition be marginally worse ideas, because they weren’t applying until you lowered the payline. That’s setting aside a lot of very interesting behavioral biases and all sorts of things that could be going on in the wild world of science. But you can see how, in this nice self-contained simple world, more noise is going to hurt the average quality of the people competing for that money.

The last step of this paper takes it into how this can raise some really interesting dynamic problems. To recap where we were: the proposition this paper makes is that if there’s more noise, it’s going to eat more of the budget of your institution. The logic goes as follows. On our prior slide, we showed how, when there’s more noise in a certain review period, you’re going to get more applications, because you’re going to have to lower the payline, so more people apply, and they’re lower-quality people as well.

This paper emphasizes the important point that a lot of the ways institutions operate in practice is that they tie budgets to applications. That is to say, your budget in period t+1 is some increasing function of how many applications you got in the period before. Now you can start to see where this is going to generate problems. If some area starts to get noisier, it’s going to get more applications, and it’s going to get more money. That’s going to be more money going to lower-quality applications.

Their empirical setup is looking at the European Research Council (ERC), because it’s a place where they have a decent way of measuring noise in the review process across different fields. This is me showing one of their figures, where on the x-axis is a measure of the inverse of the variance of the review scores. This is the inter-rater agreement.

A large number means that when a bunch of people see the same proposal, they tend to agree on the quality of that idea, and low numbers mean they tend to disagree. Already, in line with intuition, what you might call the hard sciences — biology, physics kinds of things — have more inter-rater agreement than the social sciences, where we’re dealing with the messy world of humans and we don’t have molecules that we can be very clear about all the time.

The hypothesis of this paper is: if the fields that have high inter-rater agreement start to have to compete for budgets with these low inter-rater agreement fields, those low inter-rater agreement fields are going to start to eat the budgets of the high inter-rater agreement fields. That’s exactly what they find in practice.

There was this big reform at the ERC where, instead of budgets being defined on very narrow levels, they started to expand it. If you had a noisy field through the lens of their model, that field would start to have more applications and would start to get more budget from other fields that had less noise. Going back and forth, you can see all of our noisier fields of the social sciences in green down here. The theory of this paper would suggest that if they started to have to compete for budgets with these up here, they would start to eat those other fields’ budgets, and that’s exactly what you see in practice.

There are lots of interesting things going on here, and taking noise seriously, and what it does to grant systems, is very important. It also raises all sorts of questions about what’s going on inside of peer review. If you’re interested in this, Kevin Boudreau has a very nice review paper that just came out a few weeks ago, focused on experiments done in this space. I call it the Wild West, because there’s lots of really interesting experimental work. Something that’s very unclear at this point is what the long-term effects of different peer-review arrangements are, because, as we talked about at the beginning with the war-on-cancer example, a lot of the real effects of science take so long to manifest that it’s hard to set up good experiments on what’s the right way to peer review, or what makes a good peer reviewer. These are lots of interesting open questions.

Designing grants

Back to our grant pipeline. I want to talk now about the design of the grants themselves: how much money, the structure of the grants, what can you do with the money, when should you be audited, all sorts of things. There’s lots of very interesting theory here. The one I want to focus on is a notion raised by Gustavo Manso’s paper, Motivating Innovation, which builds on a lot of earlier work on principal-agent models. If you want to get people to do high-risk things, you need to give them incentives that tolerate early failure and reward long-run success.

One of my favorite tests of this theory is a fun lab experiment that Florian Ederer and Gustavo Manso did, Is Pay for Performance Detrimental to Innovation? where they ran an adorable little lab experiment.

Imagine you have to play a 20-period video game and you have to sell lemonade. This is like some little game you would play on your phone, where you have to choose where you set up the lemonade stand, what price you’re going to charge, how much sugar you’re going to put in, etc. The analog is, you’re a scientist and you have to choose where you’re going to do your research, who you’re going to hire — lots of choices, and you don’t know what the payoffs are from all these different combinations of choices.

In the experiment, they have three arms.

1. Every period you will get a flat payment regardless of how much money your lemonade stand gets.

2. The pay-for-performance arm, which is, you’re going to get a fixed share of the profit your lemonade stand has.

3. The exploration contract, which has some analogues to the kinds of things you might think would be optimal for a scientific environment: you’ll get no payment for the first 10 periods, but then you get a higher profit share in the last 10 periods. Implicitly, you can see how the goal there is, you’re not getting paid at the beginning anyway, so try some stuff, see what happens, so that when your payments start to come in, you’ve found something that is high-reward.

They’re going to measure what this does in terms of how it affects people’s strategy, with the idea that — this is another use of variance as a measure — if this really does incentivize exploration, you should see people trying a lot of different things. So they’re going to look at the standard deviation of these different strategic choices. Then they’ll look at the actual amount of money these people have made, and they’ll also do some nice moderation, trying to see how much this might depend on these subjects’ risk aversion.

Here’s the graph plotting the standard deviation of the strategy choices across the three arms in the two chunks of periods, the first 10 and the last 10.

Exactly as the theory would suggest, those in the exploration contract tried a lot of different things compared to the others in the first 10, and then settled down. So there’s a first stage of sorts: they did different things. Was it actually helpful in the context of this game? The answer is yes.

On the left, it turns out the school was the best place to put the lemonade stand, so they’re showing the share of the subjects that found the school, and you can see the exploration contract is where basically everybody found it: 80% of the people found it.

The interesting thing here is the actual profits to the subject in the different arms. Whether you measure this by their maximum profit per period, or the profit in the last period, or their average over all of the periods, they did the best. They explored, they found the right combination of things, and once they found it, they stayed on. One other interesting thing is that, comparing the pay-for-performance contract, people who were risk-tolerant did pretty well in that contract, but people who were risk-averse fared quite poorly in the pay-for-performance contract, where you were getting a share of profits every period. Whereas in the exploration contract, your risk aversion didn’t matter, because you’re shielded from the risk entirely in those first 10 periods when you try different things. This was a proof of concept that this might be an interesting thing to really push on.

Then there’s a nice paper by Pierre Azoulay, Josh Graff Zivin, and Gustavo again, Incentives and Creativity: Evidence from the Academic Life Sciences, that does the best of trying to say, “What’s a real-world analog of this kind of arrangement with scientists?” They focus on the Howard Hughes Medical Institute (HHMI) investigator program, which does a lot of things that, compared to the traditional NIH grant that a lot of these people would be eligible for, look like the exploration contract: it’s longer funding, there are fewer reviews, there are fewer strings tied to what you can do with the money.

They wanted to say, “If we do our best to find two scientists, one who got an HHMI grant and one who looked as similar as possible on observables but didn’t get an HHMI and instead got an NIH grant, how would they fare?”

This is one of the results you can see in the paper.

If you focus on the weighted controls, you can see that our HHMI and NIH scientists are trending together towards this common period where they’re both going to get resources, but one gets HHMI and one gets the NIH, and afterwards they depart. This is by no means an RCT, because we don’t have the kinds of data on the HHMI shortlist to know who was the marginal HHMI that just didn’t get it and who was the marginal NIH person that just didn’t get it. The authors did their very best to find a good treatment and control group, but at the end of the day there’s still a question of how much of that gap is due to the HHMI structures per se versus HHMI being very good at spotting scientists who are talented.

Which is why my colleague Wei Yang Tham and I were then like, how much of this is due to the grant per se? That was our question in Money, Time, and Grant Design. How much is this really because people have more money or more time in their grants? We don’t have $1 billion that we can randomize to run that experiment, so we did what we thought was the next best: we ran a survey with actual scientists across all fields of science, not just the life sciences. We asked them what they would do if they got grants of different amounts and with different durations over which that money was available to them. The implicit argument being, if these contractual arrangements around the grant design really matter in practice, then when we give people fake long grants, they should say they’re going to be more likely to take risks in their work, as the theory would suggest, if the grant design itself was the important constraint on their risk-taking.

This was the setup of the question we asked.

Let’s say you got a grant, and we randomized the amount and the duration of these grants. Then we pretty simply asked them, “What they would do?” and we structured the possible answers to be things that aligned with the different theories about how these different incentives could influence scientists’ strategic choices.

In a certain way, this paper is a bit of a null-effects paper.

We’ve got our good old-fashioned statistical significance stars there. We see the obvious thing, which is that if we give people longer grants or more money, they’re very unlikely to say they’re going to move fast, which makes a lot of sense and is a bit of a sniff test that they’re taking these things seriously. If we give people more money, they say they’re slightly more likely to say they’ll take more risks. But if we give people longer grants with the same amount of money — that version of the exploration contract — they don’t say that that’s something important in terms of their risk-taking.

We do some work in the paper and we find that actually, if you look at well-resourced post-tenure researchers, when you give them more time, they are willing to take more risks. Those are the kind of people that look a bit more like people in the Pierre, Josh, and Gustavo HHMI paper, where the scientists are awesome scientists already. Whereas we have a very large population. When you zoom in on people, you can probably find some scientists for whom the grant design is a constraint, but on average, one way of thinking about this is that they’re not a large constraint.

We spent some work in the paper on actually estimating people’s preferences, getting back to the idea that if people have different preferences over the grant designs, then maybe grant design is less about changing what a given scientist does and might be more about changing which scientists show up to your grant door in the first place.

We’re able, with our data, to come up with good old-fashioned indifference curves, where you can say how willing scientists are to trade off dollars versus time.

One of my personal favorite things we do in the paper — you have to buy a lot of assumptions, but you can get an estimate of how willing funders are to make this same trade-off. It looks like funders are much more patient.

That is to say, if you look at the observed grant design, the trade-off that rationalizes what funders put out in the world makes it look like they’re very willing to trade off time and dollars, whereas scientists basically want the money now. So the constraint on their work is less time and more money.

All the earlier work there was very much focused on dollars and years of grants, and less focused on what happens if we put a funding opportunity on a particular topic. Can we get people to focus on that? That is what I focused on in this paper, The Elasticity of Science. I’m going to jump right to the conclusion, which is this graph.

What I’m able to do is say, if I want to move a scientist from point A to point B, and I want to make them make that move only by giving them grant funding, how much money do I have to give them to make that move? That is the elasticity of science I’m focused on in this paper.

This is a graph that says, for every percent change you want to get somebody to make in the direction that they’re pointed with their research, the costs are going to go up pretty quickly. If you look in the data, each researcher, every period, naturally is working on a new topic, and you can think of them as taking a step in a new direction. The standard deviation of steps you see naturally is about 25% of the mean. On average, everybody takes about a 25% difference to their next project. The question is, if I didn’t want to let endogenous natural forces lead people into certain directions, and I just wanted to take money and push them in a particular direction, that would take something like $3-4 million to get them to take a step of the size they naturally take anyway. I would get them to take it in the direction I want, but I would have to pay a pretty hefty price for them to do that.

For reference, the grants at the NIH are on this scale of $3-4 million, and that’s for doing the work of the project that they’ve agreed to already take the step towards. So one very loose way of thinking about this is that if you wanted somebody to do that same amount of work in a different direction, you would have to double the size of the grant. That’s a very casual way of interpreting this, but I think it conveys the magnitudes that I’m finding.

I think this paper was important because a very understandable default assumption for a lot of economic modeling is that adjustment costs are trivial and people can move around, which for certain types of work, especially in most of the 20th century, wasn’t the craziest assumption. Part of what I was trying to convey in this paper is that that is not a good assumption to make in science. It’s very difficult. What drives these costs? It could be that I just don’t want to do it. I’m a scientist and I’m willing to give up a lot of resources to not do that thing you want me to do. Part of it could be that I have a lot of expensive equipment that can only do certain things. There are lots of different reasons there that I haven’t totally unpacked, but the net magnitude is something to take seriously.

Now I want to spend the last bit of time walking through this wonderful paper, A Quest for Knowledge, by Christoph Carnehl and Johannes Schneider, because it’s one of the most interesting theoretical papers in the economics of science that takes direction very seriously as of late. I’m going to try to walk you through these slides in a way that can give you a start of some intuition as to what’s a different way of thinking about what the direction of science even means. Part of the beauty of this paper is it gives you a framework for how to think about what it means, in mathematical terms, to think about the direction of science.

What they do is, first, say there are truths about the world, and those truths are answers to questions. Let’s assume that we can line up every question on some arbitrary scale, and the answers to all of those questions are some number, and the path of that number is this jaggedy — the Brownian motion, that sounds very intimidating, but it’s this messy gray line that is the truth of the universe.

Now, what is knowledge? Knowledge is a point on that line where we know the answer to that question. That’s my red dot here. We know that the answer to Question 0 is 42. It’s nice to put mathematics on this; it sounds silly saying it, but this could be, “How hot do you have to get a PCR machine to adequately produce the right number of copies of the genome?” It’s 42 degrees. That’s the kind of thing we’re talking about here.

Then they bring in the very realistic thing — knowing this point exactly gives us more information than just that point. Since we know there’s some structure to the knowledge of the universe, we can make a conjecture about the answers to the questions around the question that we know the answer to.

You could think of these as probabilistic guesses. We don’t know exactly what the answer to Question -1 is, but we are pretty confident that if the answer to Question 0 is 42, the answer to Question -1 is somewhere between 46 and 38. In this case we’re wrong, as you can see, because we can see the truth, but that is our best guess given what we know. What happens if we learn the answer to Question -1.5?

That buys us our nice red dot up there, and it really shrinks the uncertainty around the truth in between the two questions we know the answer to.

Thinking of the world this way opens up a nice way of thinking about what the optimal direction of science should look like. It formulates it as a very specific thing in terms of where on this question number line we should try to answer the next question.

The way that Christoph is doing this graphically is: let’s go back and say, our x-axis is still our question line, and we know the answer to 0. Now I’ve scaled this by Q. Q is the thing that determines the planner’s uncertainty tolerance. If Q is very low, that means that as the planner — when we’re making decisions about things based on conjectures — we really want to know that what we’re going to do in the world, based on our guess of the answer to the question, we are going to be sure. If we really need to be sure, then we’re really going to be talking about projects taking very small steps in question space. Since we’re very risk-averse, we want to only think about taking very small steps, because the value of information is very high. If we’re very risk-tolerant, then we can jump around on our question number line a lot.

The y-axis — just think of that as the value of knowing the answer to that question. That’s what that triangle is. Don’t worry about the fancy math. That’s the value to society from knowing the answer to Question 0. The question is, where do we go next? Let’s go to -2Q.

What does -2Q buy us? It buys us that green space, the marginal value of answering that question. Why does it look like this? This little curvature is based on that curvature you’re seeing in between these two known questions.

We get some value because we now have more certainty; our conjectures are better about questions that are in between the two questions we know the answer to. This peak is because right in the middle is where we have the least amount of certainty. Our conjectures are the worst, which is why we cut out that value the most in the middle. The only thing we have full value for is these two things we know the actual answers to. That’s why it goes all the way down to zero there.

We could go to -2Q. That sounds like a good idea. That gets us some value. We could go to -3Q.

What’s the value of going to -3Q compared to going just to -2Q? Calculus takes you all the way. This is a really nice graphical way of showing that the marginal value of jumping all the way out to -3 is worth more than jumping out to -2. You just take the area of these two things and see how much value is created. It’s a bigger jump. You can see we have more uncertainty; we’ve carved out more of our value by taking that bigger jump, but we’ve bought ourselves more knowledge closer towards that new question.

Let’s keep going.

Well, no, because eventually you’re going to take a leap that’s so big that the uncertainty resolved in between your two answers does not actually buy you much. You can see in this area, going to -4 actually is not as helpful, because this marginal value of just going to -3 is a bigger chunk of value than going all the way out. The idea is, if we know the answer to this question and we shoot for answering a question that’s super far out, that’ll help us super far out there, but that does nothing to help us learn about what’s going on in between those two points.

Hopefully this is starting to give you intuition that the value of picking new directions is what it buys you in between things we already know, which also means that pushing the frontier can mean pursuing questions in between questions we already know.

In this case, let’s say we actually know 0 and we also know -6. Where do we go from here? You can see value can be created by shooting for Question -3 in the middle, because that’s resolving a lot of uncertainty about what’s going on in between zero and -6. In this setup, it’s optimal to shoot for the dead middle spot, because that’s where the most value is created. But this is where math is tricky and complicated: it’s not always the case.

If we know 0 and we know -8, it turns out that going to -4 is actually not optimal, and stretching a bit out to like -4.5 is actually optimal. This is where math is helpful but also shows you that intuitions can go awry sometimes.

Moonshots

One thing this paper walks through is this idea of moonshots, which is a super interesting concept that’s really hard to think about formally, but this model gives you a nice way of thinking about what moonshots do.

First, on the left, we’re going to walk through what happens in a world where we don’t do a moonshot. In the first period, t equals 1, let’s say we take a medium shot out to Question 3. In the second period, we let the market proceed in equilibrium. Where do people go? They say, okay, I can create the most value by going out to 5.1 in this case. The third period, where do we go? I push the frontier again and go out to 7.2.

What does a moonshot do?

A moonshot says, in Period 1, I don’t want to just settle for 3. I want to go all the way out to 6. That, in and of itself, in that first period, is worse — less value is created. But, in a dynamic sense, under certain conditions, it can actually create more value. If you let the market move forward in equilibrium from there, it says, “We know 6 now, where do we go optimally?” It turns out optimally is to fill in and deepen back at 3, because we can create a lot of value in that middle space that just opened up. Then, in the third period, where do we go from here? If we know 3 and 6, it turns out we can get further than we would have gotten without the moonshot. We can get all the way to 8.1. These magnitudes are irrelevant and very hard to think about in practice. The logic of what a moonshot does is, it makes the value of these connecting ideas, like the three over here, valuable.

To summarize it, I’ll take a phrase of Pierre Azoulay’s: this paper very much is about trying to get you to respect the geometry of knowledge, insofar as the optimal science policy is not a time-invariant thing.

It depends on where all our current knowledge points are. Does that mean we should take some deepening steps to fill in gaps, or does that mean we should take some expansion steps? It introduces this idea of the optimal flow of science looking like a cycle of deepening and expansion, where we throw out a fishing rod very far, we catch a fish, and then now we know where the fish are in between us and that fish, and we fish in between for a while, but then eventually we fill all that out. Then we throw out further again and expand, and then we fill in. That cycle nature is a very interesting concept to engage with.

Conclusions

Moving forward, and where I think things are going in these modern times. First, I think all of these things are still very relevant.

One of them that hopefully has shown through a bit in today’s discussion is that we’re getting a better sense of what matters. We can reject a lot of null hypotheses, but there’s a lot of work to be done in terms of how much a lot of these things matter. To the point earlier about framing research questions and noise, what are the important sources of noise? I don’t think we have a very good sense of those kinds of questions. Two big questions that I’m interested in right now.

“Allocative efficiency”

One is thinking more about allocative efficiency, which gets back to our graph from before.

A fun thing you can do with this graph is say, “Why are there residuals?” Let’s assume even for a moment that the WHO knows how to measure these things and this is a correct measure of demand. In the theoretical ideal, the only reason there would be residuals conditional on demand would be that productivity is different across these places. We don’t invest as much in musculoskeletal because it’s a really hard thing to study. Even though the demand is the same, the supply curve is just flatter, so we’re going to invest less, and that would be economically rational from a very classic utilitarian objective.

Just to give you a sense of how big these gaps are as something to investigate: I’m plotting here those residuals in percentage and dollar amounts, so you can see the scale of these residuals.

The yellow lines are just based on regressing the NIH budget on the US burden. The blue lines add in the global burden, which is important. You can see, for example, in the infection immunity one at the top, just looking at the US it looks like we invest a lot relative to demand, which could mean that infectious disease research is very productive. Or it could mean that we put a weight on infectious diseases that affect not the United States. In fact, that’s exactly what you see: once you include the global burden for infectious diseases, that residual cuts in half. So a lot of what might look like investment not due to demand in the US is actually investment because the US, by revealed preference, cares about the rest of the world. To the right extent is a very interesting question, but you can see that happening in that portion of the data.

In practice, we have no idea. There is so little research that uses the tools of what you would call modern industrial organization and productivity analysis of standard firm-based markets. We don’t have a very good sense of what this allocative efficiency looks like in science. I have a paper with my good friends Fabio and Wei Yang, Productivity Beliefs and Efficiency in Science, where we’ve done our best to take a stab in that direction, but I think that’s a very interesting place to pursue. It’s very hard, because a lot of the things you need to do these kinds of studies are not traditionally recorded, but I think it’s very important.

AI and scientific infrastructure

Last, I can’t not talk about AI, even though I don’t want to talk about AI. Something interesting with respect to the direction of science and AI is not who’s going to come up with the tool that makes science go super fast because it can discover drugs — that is a very important question. I, as an economist interested in social welfare, am interested in the question of how AI will change scientific infrastructure, in terms of how we allocate credit, resources, and the structure of science.

There are fun history examples of even seemingly small things.

AI is clearly a big technology, and if you look back at the world, it took one famous statistician saying, “I think 0.05 is a decent rule of thumb,” to create a tremendous amount of distortion in the publication market. There are lots of very interesting papers going out. P-hacking all traces back to Fisher and a sort of off-handed comment. Another one is citations. Citations initially were just meant to be bibliographic management and acknowledgements to people reading articles. Now they are at the core of many promotion decisions and all sorts of resource-allocation decisions and have real impact. This Hager, Schwarz, and Waldinger paper is a really good paper showing how that played out in practice in the early days of impact factors and citations. The reference to Goodhart’s law is: “Once a metric becomes gameable, it ceases to be a metric.” Once an input becomes an output of measure, you’ve lost the game, because people will just go after that thing and not the ultimate goal that you actually had in mind.

Within the context of AI, the way I’m currently thinking about it, in terms of where we’re allocating resources in the direction of science, is that a lot of how this used to play out worked via a currency called words. The cancer researcher would use big fancy words that convinced other researchers that they deserved money to pursue cancer research and not another infectious disease project. In a world where currency holds value, through the lens of traditional price theory, that can be an effective way of allocating resources, because the cancer researcher had a better idea, so it was easier for them to use words to get resources for their idea. Now, if Zachary truly has a better idea than me, but I can put my crappy idea into Claude and polish it up so that it sounds equally good, that starts to present some problems for how we allocate resources. This is not unique to science per se. People are already looking at this in labor markets — in recruiting, this is a very common problem, anywhere that words were a currency. I think clearly words were a currency in science, and so we’re going to have to think about how things are going to change in this new era, which is very exciting.

I think it’s a tremendous opportunity — if all new ideas are consumed via models. One challenge with all of the research on the direction of science historically has been that we couldn’t observe an idea, or define it, or see how my idea made its way to William’s work. But in a world where my idea is actually digitizable and goes through a model where we can measure its Shapley value, and William has a willingness to pay for OpenAI, who know the Shapley value, who can pay me, and I have data — I think really interesting technological advances might facilitate a more efficient market. That gives me hope amidst the wave of AI slop that appears to be oncoming.

I know Chad makes a very convincing argument that in the long run population is what matters, and when you’re thinking about growth rates. Ideas getting harder to find is a tremendously important idea.

But as a pragmatist who cares about growth levels and has a non-zero discount rate, I think there’s a lot of work to be done to increase productivity, even if it’s just a level effect. I think a lot of value can be created by taking grant design more seriously and taking AI more seriously as a part of the resource allocation.

• Metascience team lead Jenn Gustetic (formerly of NASA) and Senior Metascience Fellow Dan Turner-Evans (most recently of the US Senate) sharing pieces on the impact of AI on the talent pipeline (possibly bad) and on scientific research (probably good),

• Metascience Fellow Matt Esche, who has the longest tenure of anyone on the metascience team at just over three years, discussing the potential for new cancer vaccines, and

• Brand new Associate Metascience Fellow Hunter Wieman (coming from graduate school at Princeton University) reviewing an article on the challenges of conducting randomized controlled trials on AI.

Subscribe now

1. Social Science at the NSF (Matt Clancy, Abundance and Growth Blog).

In early 2026, when the National Science Foundation (NSF) published a budget request to Congress that would eliminate funding for the Social, Behavioral, and Economic Sciences (SBE) Directorate, I was surprised. In addition to funding important social sciences research, SBE manages important national surveys such as the General Social Survey (GSS).

While the motivation for eliminating SBE isn’t totally clear (the budget request said that NSF would continue grants that align with Administration priorities — perhaps the implication is that other grants do not), Clancy’s piece is helpful in showing that this isn’t the first time a president has targeted SBE. The Reagan Administration cut social science funding substantially, and Congress approved a rescission package of a third of NSF’s social science budget.1 However, skepticism about whether NSF should fund the social sciences goes back to its beginnings. As Clancy notes, some drafts of the founding documents for NSF included a ban on funding social sciences, though that language did not ultimately end up in the legislation.

Something I wish Clancy’s piece had delved further into is: why did the Reagan Administration and Congress cut social science funding in the 1980s? Based on what I’ve been able to find, even at the time it wasn’t totally clear.

A Science piece from April 1981 sounds like it could have been written in 2026:

Rumors have been rife, and many people believe that the Office of Management and Budget went over the entire federal budget with magnifying glasses looking for social research to stomp on. The proposed cuts for social and behavioral sciences at the National Science Foundation have been detailed in unprecedented fashion, and have done more than anything else to arouse paranoia among social researchers.

OMB’s 1981 explanation for proposed cuts was that “support for the behavioral social and economic sciences is being reduced significantly because much of the support of these sciences is considered of relatively lesser importance to the economy than support of the natural science.” As with now, it is perhaps a mix of a perceived liberal slant in social science and a belief that, if cuts to science are needed, social science is less important to the economy.

What is the status of SBE now? Contra reporting released in April 2026, SBE has not been shut down preemptively, and its grantmaking picked up in June. However, as of August 16, SBE has only spent about half of what it would have in a typical year.2

Subscribe now

2. The Mystery of Short Indian Children (Wilson King, In Development)

Here’s a mystery: why, despite rapid economic growth in India, have children remained short? This has not been the case in many other countries as they’ve developed economically (see the chart on economic growth and height in Africa). This is important because height, while not itself essential, correlates with health generally as well as cognitive development. As Wilson’s article states, the low height of Indians in India is not genetic; if their families move to wealthy countries, they will grow substantially taller.

Per Wilson, there are multiple potential reasons for this:

First, in India, there is a preference for sons, which might mean that daughters are malnourished. Second, there might be a preference for first children over subsequent children. Finally, India has lagged behind other countries in sanitation and hygiene despite its growth. That could cause children to remain sick, stunting their growth.

It’s likely a combination of these, and — while tackling something like a son preference is particularly difficult — improving sanitation is within the government’s power. Knowing this allows India to make policy decisions and financial investments that could improve the lives of millions of people and, through making them more productive, further enhance economic growth.

Why am I talking about height disparities in a science policy blog? It illustrates how social science, including all the papers discussed in Wilson’s article and linked above, can answer societally important questions.

I recently attended a workshop on the future of NSF’s SBE program, hosted by Arizona State University’s Consortium for Science, Policy & Outcomes. One of the discussions focused on whether there should be a dedicated unit at NSF that funds social science, or whether it should be spread out across the agency.3

I’m skeptical of breaking up a social science unit and using it to improve multidisciplinary projects because the primary benefit of social science research is not through interdisciplinary work with the physical and natural sciences. It’s through answering society’s most pressing questions. This piece on children’s height is a great example of this. Many of the questions that currently vex Americans can be answered, or at least better understood, through social science research:

• What is the impact of AI on the labor market?

• How did the COVID-19 pandemic affect children’s development?

• Why are people getting married at lower rates, and why are they having fewer children?

• Why do young people seem to be struggling with mental health issues? Is it all the screens?

• What are the impacts of immigration on economic growth and labor markets?

Not to mention questions about how to make science more efficient and increase the likelihood of breakthroughs (important metascience papers like this and this came out of SBE funding). You’re not going to answer these questions by sprinkling social scientists around geology and quantum computing programs.

Now whether social science actually does tackle problems important to most Americans is worth considering. Parts of the social sciences have gotten bogged down in poststructuralist approaches rather than focusing on empirical, socially important questions. But SBE doesn’t tend to fund those things anyway.4

Let’s keep funding social science at NSF.

Subscribe now

3. The Tragedy of the Cognitive Commons: How AI Could Disrupt the Regeneration of Professional Expertise (Nolan Lovett, preprint for Human Resource Development Review)

From Jenn Gustetic: AI is eroding the need for humans to conduct entry-level research tasks, and this paper suggests that might soon cause problems. I found the paper interesting because of its ties to the R&D management and implementation camp of metascience. It includes testable hypotheses about how AI is reshaping professional expertise. This may be particularly relevant to the R&D enterprise because declining STEM PhD admissions could compound the effects of AI.

Lovett proposes that by increasingly using AI for tasks that early-career employees used to do, our human capital across professions is experiencing a “tragedy of the cognitive commons.” While organizations make near term efficiency gains replacing entry-level research tasks with AI, the expertise depletes over time for the profession overall. The paper suggests two mechanisms that could weaken the future pipeline of employees with domain expertise (who can verify and refine AI outputs). The first mechanism is position elimination: fewer entry level hires. The second is reduced internalization of domain knowledge, through less cognitive struggle by entry-level employees in creating research products.

The paper posits that human capital systems need both “internalized mastery” and “distributed mastery” in the era of AI. Internalized mastery is the deep domain knowledge that is developed through sustained cognitive struggle and increasingly complex professional experience; it provides the foundation to do substantive validation. Distributed mastery is the ability to “orchestrate” AI systems to produce high quality outputs within a specific domain. Without internalized mastery you only get surface level validation, which decreases the quality of outputs over time.

Training and credentialing should create opportunities for both “tracks” within professions. If the stock of internalized mastery falls, substantive validation degrades, and validation error rates rise, then the quality of those professions decline. The paper proposes some policy ideas but suggests empirically testing the approaches. For example, governments might provide training subsidies in AI-exposed occupations, tax credits for firms that maintain developmental pipelines, or competitive grants to professional associations to experiment with governance approaches. We could also consider expanding federal training grants to the research community.

Perhaps these could be good metascience experiments.

Subscribe now

4. AI for science needs reasoning, not just data (Eric Schmidt and Suhas Mahesh, MIT Technology Review) and Conjecture Machines: AI agents and the new validation bottleneck in science (Don Wallace, Conor Griffin, Sean O’Neill, Thang Luong, Owen Larter; Google DeepMind website)

From Dan Turner-Evans: AI agents are becoming increasingly good at doing science, and current and former Googlers are taking notice. Eric Schmidt, the former CEO of Google, and Suhas Mahesh at Schmidt Sciences argue that agents, rather than custom models like AlphaFold, will ultimately be the form of AI that leads to the greatest acceleration of science. While most existing scientific datasets have already been used to train new AI models, agents can be continuously applied to any field of science. They can automatically log every condition and decision — thereby increasing the likelihood of reproducing any finding while also increasing institutional memory — as well as rapidly testing out new ideas in simulation or against any paper ever published.

Meanwhile, the policy team at Google DeepMind explains why agents have gotten so good at doing science, how they are being incorporated into the research process, and the policy implications of these changes. They argue that better models, stronger scaffolding around these models, and improved customizability have made agents immediately useful for scientists for mundane tasks, coding, and idea generation and validation. But they caution that (1) graduate training needs to be preserved in an age of AI, (2) not everyone has access to agents, (3) not all data is ready for agents, (4) AI-generated ideas must be validated, and (5) peer review is under increasing strain.

AI agents are already proving to be incredible tools for math and computer science, but I am not yet convinced that they can drive breakthroughs in lab-based disciplines that are limited by data and experiments rather than logic. However, I do find the idea of agents helping with reproducibility compelling, and I know many scientists across disciplines who are experiencing noticeable productivity gains from AI. Still, the DeepMind team is right that more work is needed to validate their outputs. I’m still trying to figure out what graduate training should look like in this new era, but I plan to share more thoughts on this soon (and welcome your ideas!).

Subscribe now

5. How to build a cancer vaccine, and whether they will work this time (Abishaike Mahajan, Owl Posting)

From Matt Esche: It’s been an outstanding summer for breakthrough cancer treatments. Pancreatic cancer, with its RAS mutation long thought to be ‘undruggable,’ saw the new drug daraxonrasib approved by the FDA. And with the positive clinical results from Merck and Moderna’s cancer vaccine for melanoma this August, abundant personalized vaccines for cancer treatments look a bit closer on the horizon. These results make me excited to think of all the therapies being developed now for previously untreatable diseases.

Mahajan’s blog post, which was written before the Merck/Moderna vaccine trial results, tempers my expectations a bit. Walking through decades of cancer vaccine approaches, the post shows how long it’s taken scientists to tease out the right approach, with a surplus of failures along the way. Even when an approach does work, the same design doesn’t simply translate to the next cancer, and we still don’t know why certain approaches in cancer immunology have failed.

To get there, we’ll need much more evidence and clinical results. There will be many more Phase 1 trials before we see widely-available personalized cancer vaccines. That’s why I’m glad to see movement on improving clinical trial policy, such as new pilots to improve the speed and quality of Phase 1 trials. Here’s to continued biomedical breakthroughs alongside clinical trial abundance bringing new therapies to market soon.

Subscribe now

6. AI RCTs are booming - to be useful they must evolve (Tim Ohlenburg, Oliver Hanney, Sharif Kazemi, Joseph Levine, and Shahrukh Wani; VoxDev)

From Hunter Wieman: As AI is poised to transform the economy, economists and funders are rushing to study its effects. Last month, Anthropic’s Economic Futures Research Fund announced a $200 million commitment to “support ambitious external research on interventions to prepare society for the economic impacts of AI.” Much of this will go to randomized controlled trials (RCTs) that study the effects of AI initiatives and AI adoption. The authors of this piece warn that the default outcome could be a glut of economic studies that fail to provide meaningful insight.

Designing economic RCTs that generalize is already difficult, and the authors believe that AI will exacerbate this issue. Two mechanisms stood out to me in particular:

• Testing Obsolete Models: Already, many studies of AI’s societal impacts use models that are practically obsolete by the time the study is completed; this could increase as AI progress accelerates.

• Contaminated Control Groups: Due to rapid AI adoption, control group participants may start using AI tools as the study progresses.5 And even if the control group avoids using AI at all, the post-RCT environment will be one where a large fraction of the population is adopting AI.

As a result, researchers conducting RCTs on the economic impacts of AI should ensure they design and test programs whose effects are not specific to a particular AI model, and whose results will remain relevant as AI improves and diffuses through the economy.

This is very hard, and I don’t have any silver bullets. But if I had to design an RCT on the social or economic effects of AI, I would focus on two questions:

Can this RCT help us identify mechanisms that are likely to persist as AI progresses? In a rapidly changing world, theory-agnostic estimates of treatment effects become less durable. Researchers should instead design RCTs around interrogating theories about the mechanisms linking interventions to outcomes (e.g., what broad characteristics of AI models allow them to augment human workers instead of replacing them). Understanding how and why something works can generalize to new environments in ways that isolated estimates of treatment effects do not. With AI, this understanding might help us anticipate which newer models will produce similar outcomes to current ones and which will have qualitatively different effects.

For each plausible trajectory of AI progress and diffusion, what questions do we want answers to? A study need not apply to every possible future to be a good bet. In a world with more uncertainty around the future, we should be focusing on high value bets that give important information in some specific futures instead of ones that will robustly give some information in all futures. For example, we could design some studies under the assumption that AI will replace significant swaths of human labor and others under the assumption that it will instead augment and empower workers. The ambition should be less to future-proof every RCT than to build portfolios of studies that collectively ask high leverage questions across different paths of AI capabilities and adoption.

Economists interested in metascience (those in the “innovation economics” camp) often talk about applying experimentation and econometric methods to science. This piece underscores just how difficult and important it is to design programs and experiments well so that these methods actually answer important questions.

Merton identified five norms that he argues govern scientific communities…

Communalism means scientific discoveries belong to everyone. No one owns E = mc² the way you own a patent. Scientists give up property rights in exchange for recognition and esteem.

Universalism: claims are evaluated on evidence, not the scientist’s identity. Talented people from anywhere can contribute — that’s the theory, at least. Maybe you’re smiling when I say that. But in theory, it doesn’t matter who speaks; it only matters what they have to say, and the substance of it.

Disinterestedness: that doesn’t mean scientists aren’t self-interested, but rather that institutional arrangements reward behavior that appears selfless….

Originality is completely central. The currency of science is credit for being first.

Skepticism enables error correction and filters unreliable claims through organized community scrutiny.

This concept was both new and immediately intuitive. This was my experience in graduate school, though no one explained it. The ethical code of science is as strict as it is unspoken.

Subscribe now

But as Pierre noted, this ethical code is imperfect, and there are challenges to some of these values. The Matthew Effect, in which famous scientists are rewarded for being successful, challenges the universalism norm. The presentation features this quote, which I enjoy.

“If I have not seen as far as others, it is because giants were standing on my shoulders.”

- Computer scientist Hal Abselson.1

This turns around the famous Isaac Newton quote about standing on the shoulders of giants, and gets at the phenomenon of the ideas of obscure or early career scientists being attributed to better known scientists (e.g. their advisors).

Learning about science as an economic institution is valuable because that’s one of the parts of economics that you can interact with regularly and one which you can (maybe, on the margins, especially if you work in policy) influence.

Subscribe now

Here are the readings that go with this lecture (readings with asterisks were required for the class):

• Dasgupta, Partha, and Paul David. 1994. “Toward a New Economics of Science.” Research Policy 23(5): 487-521.

• Merton, Robert K. 1957. “Priorities in Scientific Discovery: A Chapter in the Sociology of Science.” American Sociological Review 22(6): 635-659.

• (*)Azoulay, Pierre, Toby Stuart, and Yanbo Wang. 2014. “Matthew: Effect or Fable?” Management Science 60(1): 92-109.

• Murray, Fiona, Philippe Aghion, Mathias Dewatripont, Julian Kolev, and Scott Stern. 2016. “Of Mice and Academics: Examining the Effect of Openness on Innovation.” American Economic Journal: Economic Policy 8(1): 212-252.

• (*)Stern, Scott. 2004. “Do Scientists Pay to Be Scientists?” Management Science 50(6): 835-853.

• (*)Aghion, Philippe, Mathias Dewatripont, and Jeremy C. Stein. 2008. “Academic Freedom, Private Sector Focus, and the Process of Innovation.” RAND Journal of Economics 39(3): 617-635.

• (*)Furman, Jeffrey, and Scott Stern. 2011. “Climbing Atop the Shoulders of Giants: The Impact of Institutions on Cumulative Research.” American Economic Review 101(5): 1933-1963.

• (*)Hill, Ryan, and Carolyn Stein. 2025. “Race to the Bottom: Competition and Quality in Science.” Quarterly Journal of Economics 140(2): 1111-1185.

Thanks to William Higbie and Harry Fletcher-Wood for their support in producing this video and the transcript.

Subscribe now

Lecture Transcript:

Science as an Economic Institution

We’re going to talk about science as an economic institution. What is science, and how does it actually work? So far, we’ve treated scientific knowledge as an input to innovation — something that exists, gets produced, and contributes to technological progress. But we’ve mostly treated it as a black box. Today we’re opening the black box. We’re going to examine science from two distinct but complementary perspectives.

First, we’ll think about science as a type of knowledge. What makes scientific knowledge different from other forms of knowledge? Second, we’ll examine science as a social institution with its own incentive structures: what motivates scientists, and how does the reward system shape what research actually gets done?

These aren’t just academic questions. Understanding how science works has direct implications for science policy, for how we:

• Organize research universities,

• Structure funding agencies; and,

• Think about the relationship between privately funded applied R&D and basic research.

The policies we adopt — how much we spend on science, which fields to prioritize, whether to allow academic patenting, how to design peer review — all rest on assumptions about what science is and how it operates. It lays the foundation for all those other debates. That’s what we’re doing today.

We’ll start by asking what science is: what makes science different from another type of knowledge, like craft knowledge or engineering knowledge? Once we understand what science is, we’ll examine the relationship between science and technology. Does basic research flow naturally into applied research and then into commercial products? That’s the famous linear model that has historically justified massive public investments in basic science. We’ll look at evidence for this model, but also complicate the picture by examining institutional frictions that impede knowledge flow.

We’ll dig into how specific institutional features — access to frontier knowledge, funding, IP rules, access to materials — actually shape scientific productivity. The key insight, which probably won’t be that surprising, is that institutional details matter enormously for how much science gets produced and how effectively it contributes to innovation. Time permitting, we’ll talk about scientific competition as well.

What is science?

“What is science?” seems like a super simple question, but it’s actually hard to answer precisely. If you ask a philosopher of science, you’ll get debates about falsifiability, paradigms, research programs, and epistemic communities. Ask a scientist and they’re going to say something like, “I know it when I see it.” But they’ll struggle to articulate exactly what distinguishes their work from engineering, from craft, from trial-and-error problem solving.

But Harvey Brooks provides a useful framework for understanding the relationship between science and technology.

Science is about understanding why: testing hypotheses, refining theories, building systematic knowledge. Technology is about capturing and orchestrating regularities in nature to do something we believe might be useful. Brooks argues that science contributes to technology in six ways:

• New knowledge provides ideas for technological possibilities.

• Tools and techniques enable efficient engineering design.

• Research instrumentation migrates into industrial practice.

• Research develops human skills that transfer to applied work.

• Scientific knowledge enables assessment of social and environmental impacts.

• Knowledge bases make applied work more efficient.

But the relationship is actually bidirectional: technology contributes enormously to science, by raising novel questions and practical problems that reveal gaps in theory, and by providing instrumentation that enables more difficult research. You can just ponder the effect of Claude Code on academic research, happening right now, as an exemplar of how technology can impact the conduct of science. But the electron microscope, synchrotron radiation, superconducting magnets — all emerge from applied work and revolutionize basic science.

But Dasgupta and David ask a deeper question.

This is a classic article — if you’re into this topic you should really read it. It’s a manifesto for the so-called new economics of science. They say: given that science and technology are interconnected, why are they organized so differently? Why have universities doing open science alongside firms doing proprietary R&D? Their answer is that science and technology serve different goals, requiring different institutional structures.

Science adds to the stock of public knowledge — discoveries anyone can build on. Technology generates rents from private knowledge — innovations that provide competitive advantage to some firm. So this requires a different incentive system. Science operates under priority-based rewards and openness norms. Scientists compete to be first to publish, earning credit and reputation through disclosure. We’re going to talk about the Mertonian norms that support this system.

Technology operates under a completely different set of rules. Rewards come from secrecy and exclusivity: patents, trade secrets, proprietary know-how. You profit from being first to market and maintaining advantages — not from publishing.

Dasgupta and David argue that this duality is actually functional: both systems have internal logic suited to their purposes. But mixing these incentive systems can create tensions — when you ask scientists to both publish and patent, when universities must commercialize while maintaining academic freedom, that’s when there start to be real strains between those institutional boundaries. We want to understand this distinction to grasp how science contributes to growth, and how policy can enhance or impede that contribution.

Science as a social institution: the Mertonian norms

I want to shift from science as a type of knowledge to science as a social institution with distinctive norms and incentive structures. Robert Merton Sr., the sociologist of science — the father of the Nobel Prize–winning economist and my colleague at MIT, Robert Merton Jr. — identified a set of norms he argued should govern scientific communities.

Merton wasn’t claiming scientists actually follow these norms perfectly. He was describing an institutional ideal against which we can measure reality. The productive research agenda, for both sociology and economics of science, has been to examine who violates these norms, how, and why — and what consequences those violations have for scientific progress.

The violations aren’t just interesting as failures; they’re informative. They reveal the actual incentives that scientists face, the institutional pressures that shape their behavior, and the ways in which the system succeeds or fails at producing reliable knowledge. You can think of those Mertonian norms as a baseline model. Just as we use perfect competition as a benchmark to understand market failures, we can use Merton’s idealized norms to understand where the scientific system deviates from its stated principles, and why those deviations matter. Sometimes the violations are completely dysfunctional — they impede knowledge production. But sometimes they also reveal adaptations to real constraints. Either way, studying that gap between norm and practice tells us how science actually works, and how we might design better scientific institutions.

Merton identified five norms that he argues govern scientific communities, and there’s a cool acronym: CUDOS.

• Communalism means scientific discoveries belong to everyone. No one owns E = mc² the way you own a patent. Scientists give up property rights in exchange for recognition and esteem.

• Universalism: claims are evaluated on evidence, not the scientist’s identity. Talented people from anywhere can contribute — that’s the theory, at least. Maybe you’re smiling when I say that. But in theory, it doesn’t matter who speaks; it only matters what they have to say, and the substance of it.

• Disinterestedness: that doesn’t mean scientists aren’t self-interested, but rather that institutional arrangements reward behavior that appears selfless. We can talk about that one too.

• Originality is completely central. The currency of science is credit for being first.

• Skepticism enables error correction and filters unreliable claims through organized community scrutiny.

You may have ideas about how short we fall of actually honoring them. What Dasgupta and David argue is that these norms aren’t arbitrary — they’re functional adaptations that solve coordination problems in knowledge production.

• The priority system means that disclosure of results is actually a self-interested act.

• Universalism expands the pool of contributors.

• Skepticism filters unreliable claims.

Together, these norms create a reward structure that efficiently increases the stock of reliable public knowledge — even though they also create some problematic incentives.

The Matthew effect: a violation of universalism

I wanted to give you a sense of what it looks like to study a violation of those norms. The one I’m going to pick is the Matthew effect, which is a violation of universalism. Merton coined the Matthew effect; it’s named for a biblical verse — “unto everyone that hath shall be given” — that describes cumulative advantage in science. Scientists with early recognition get more resources, better collaborators, greater visibility, which leads to more recognition, creating a self-reinforcing cycle. The rich get richer.

The Matthew effect is a direct violation of the norm of universalism. According to that norm, claims should be evaluated based on impersonal criteria, not the author’s identity or reputation. But the anecdote you see on the slide shows this norm can be widely violated.

The same paper, the same ideas, different evaluation based solely on authorship.

One version of this is that it’s bias. Another version is that it’s partly rational: eminent scientists’ work gets more attention because they have proven track records, making the allocation of attention more fluid. But it also means that work by unknown scientists gets systematically undervalued. The Hal Abelson quote — “if I have not seen as far as others, it is because giants were standing on my shoulders” — tries to capture how your ideas sometimes get attributed to more famous people.

It’s an empirical fact that scientific output and rewards are extremely skewed. A small fraction of scientists produce most of the highly cited papers, get most of the funding, and train most successful students. But what explains this skew? There are two different interpretations. One is that science is fundamentally meritocratic: the most talented people just produce more and better work, and the reward system accurately reflects real differences in ability. The other is that the Matthew effect operates: early luck and accumulated advantages amplify small initial differences, so the skew reflects status effects rather than pure merit.

Those aren’t mutually exclusive, but their relative importance matters quite a bit for policy. Should we concentrate resources on proven stars, or does doing so amplify the Matthew effect and end up crowding out talent?

I’m going to tell you briefly about a paper of mine. The title is ‘Matthew: Effect or Fable?’ We were trying to make progress, understanding that it’s methodologically challenging to separate the two explanations I just outlined. Both predict the same outcome: skewed distributions of scientific success. So how do you distinguish between “good scientists get cited more because they’re good” versus “famous scientists get cited more because they’re famous”?

The setting here is biomedical scientists, some of whom receive a status shock — and the status shock is receiving a very prestigious appointment, the Howard Hughes Medical Institute investigator appointment. The way we’re going to do this is to look at the pre-appointment output of scientists that are at risk of receiving the status shock. The paper gets published; some of those scientists get appointed, so they get a discrete positive shock to their status. Then we can think about how the rate of citations to those articles treated by the appointment contrasts with what happens for articles associated with scientists that are very similar but somehow didn’t benefit from the status shock.

So it’s a classic difference-in-differences. This is, by the way, a research design we’re going to see multiple times today: we’re comparing citations to articles before and after some shock, for both a treatment group and a control group.

To give you a sense of what we’re doing: here are two papers.

They appear in the same journal in the same month. The last-author scientist, the principal investigator, is the big honcho in the lab. The person on the right gets appointed HHMI in 1990. Those two papers in their first year have roughly equivalent numbers of citations. Then we’re looking at how many more or fewer citations the paper on the right gets relative to the paper on the left. The whole point is to be pretty transparent about the research design.

This is what we find.

The bottom line is that there is such a thing as an HHMI appointment effect, which we interpret as evidence for the Matthew effect, but it’s not super big. It’s a few additional citations a year.

There are lots of claims about the importance of status in science, and scientists coasting on their prior reputation, that seem to imply those effects are absolutely massive. But whenever we look at them carefully — for example, in the paper by Carolyn Stein and Ryan Hill, ‘Scooped,’ they also look at the Matthew effect and find evidence for it — it’s much less quantitatively important than we might have thought. Maybe we should be slightly less cynical, in other words.

Why participate in open science?

There’s a fundamental economic question lurking beneath all this: why would anyone voluntarily participate in this system? The Mertonian framework and the Dasgupta–David model implicitly assume researchers want to be part of open science, but they don’t really explain why.

One way to think about this is from a firm’s perspective. I’m running a biotech firm. Why would I let my scientists publish? Publication spills knowledge to competitors. It takes time away from commercial work. Yet many firms, especially the most innovative ones, do exactly this. Similarly, from a scientist’s perspective, why accept a job that pays less if it means you get to publish?

This is where Stern (2004) becomes crucial. It’s a bridge between understanding science as an institution and understanding science as an economic phenomenon. That paper takes seriously the idea that there are inherent tensions between what researchers want — freedom to choose projects, the right to publish, the ability to collaborate openly — and what firms might prefer, which is to direct research toward commercial goals and keep findings proprietary.

The key insight is that these conflicts create a compensating differential in the labor market. If scientists truly value participation in open science, firms that allow it should be able to pay lower wages. Scientists are essentially paying to be scientists. But this raises a fascinating set of questions:

• How large is this effect?

• Does it apply to all scientists?

• What does it tell us about the returns to organizing research in different ways?

My colleague Scott identifies two different economic mechanisms that could explain why firms allow scientists to publish, and they have opposite implications for wages. First, there’s a preference effect. Scientists may have what economists call a “taste for science”: they derive direct utility from participating in open science. This could be intrinsic — for example, Richard Feynman loved physics for its own sake — or it could be about career concerns: if I work for Google and publish the transformer paper, even if I leave Google tomorrow, I’ve built enormous career capital. The labor-market implication is clear: if scientists value publishing, firms that allow it can pay lower wages. This is the compensating differential.

Second, there could be a productivity effect: firms themselves might benefit from allowing publication. The classic story here is absorptive capacity, a very important concept from Cohen and Levinthal in the late 1980s. To learn from external science, you need to participate in it. If you want to be part of the scientific conversation — to know what’s happening at the frontier and attract the best collaborators — you need your own scientists publishing. This is what people sometimes call science as a ticket of admission to the broader research community. But notice that if publication makes my scientists more productive, I might actually pay them more to compensate them for the extra value they’re creating. Any amount of rent-sharing then predicts that letting my scientists do science will be associated with higher wages.

So the question is, which effect dominates? That’s an empirical question, and a tricky one, because in the cross-section you’d expect firms that allow publication to employ better scientists. That creates a thorny identification problem: are science-oriented firms paying more because of the productivity effect, or because they’re hiring better people? Scott’s methodological innovation is designed to solve that problem.

The fundamental problem is unobserved heterogeneity. Suppose I observe that biotechnology firms allowing publication pay higher average salaries than firms that don’t. I can’t conclude much, because firms that allow publication probably hire better scientists — the ability bias could be enormous. The traditional approach in labor would be to look at job switchers, people who move from restricted to open-science environments. But as studies like Gibbons and Katz have shown, switchers are selected: people switch jobs for reasons correlated with their productivity, so you still have a selection problem.

Scott’s insight is really simple but really productive. Prior to accepting a job, scientists receive multiple job offers simultaneously. These offers vary in their scientific orientation — some allow publication, some don’t — and in compensation. But crucially, they’re all offers to the exact same person. So the market has already assessed this person’s ability, and that assessment is baked into all the offers. This lets you estimate a classic hedonic wage regression at the offer level with individual fixed effects. You’re asking: holding constant everything the labor market knows about your ability, do offers that allow publication pay more or less? It isn’t just a technical fix. It’s reconceptualizing how to think about compensating differentials when ability is unobserved. The key is exploiting variation in job characteristics for a given individual, which purges all the ability confounds.

This is the key regression table.

It’s a small sample — this was done at a postdoc conference. I was a grad student when that paper was being written, but it hasn’t aged. Here we’re regressing wages on whether I permit my scientists to publish. The key thing is that the sign of the coefficient flips if we control for individual fixed effects. What might have looked like a productivity effect in the cross-section turns out to be a compensating differential. Roughly, firms that allow publication pay their scientists 25% less.

Where should research happen?

That empirical result has fueled efforts to formalize Dasgupta and David and science as an economic institution. That’s the paper by Aghion, Dewatripont, and Stein (ADS). If scientists pay for freedom, who’s going to give them freedom? Because this subsidy from scientists to society is what makes open science economically viable. But that only works if scientists are actually getting what they’re paying for.

That leads to very standard questions: given that scientists value freedom, when should we organize science in academia versus the private sector? That’s what Aghion, Dewatripont, and Stein look at. They take the Stern compensating differential seriously and ask what it implies for when research should happen in academia versus the private sector. The traditional answer — if you read Arrow (1962) — to the question, “Why does academia exist?” relies on appropriability: if you can’t patent basic research, firms won’t do it, so we need universities.

That explanation is a bit problematic given how much intellectual property protection has expanded, especially under things like Bayh–Dole. So ADS 2008 offer a completely different rationale, based on control rights. Think about what fundamentally distinguishes academic from private-sector research. It’s not about funding sources or subject matter. It’s about who decides what to work on and how to work on it. In academia, scientists choose their own projects. That’s not just a nice feature — it’s definitional. That’s what tenure protects. In firms, managers can and do tell scientists what to work on. That’s also definitional. That’s what ownership means.

Now we can connect this with Stern. We know that scientists value creative control and will accept lower wages to get it. That creates a fundamental trade-off. Academia’s advantage is that you can hire scientists cheaply, because you’re giving them what they want: freedom. The disadvantage is that they might work on things that they’re intellectually interested in but that are economically useless. Private-sector firms can direct scientists toward high-payoff projects, which is valuable, but they have to pay scientists more to compensate for taking away their autonomy.

The question becomes: when does the benefit of cheap labor outweigh the cost of misdirected effort? ADS formalize this trade-off in a very elegant model that generates sharp predictions about the optimal organization of research.

Imagine you’re trying to develop a new drug or technology. That requires going through a number of stages — let’s say k stages: maybe basic science, then proof of concept, then clinical trials, then manufacturing. You only get paid when you complete all the stages. Each stage is risky, even if you’re working on the right thing.

Here’s the key choice. You can organize research in two ways. In the academic mode, you cede control to scientists. They choose what to work on and how. The advantage is you pay academic wages — let’s call that Wᴀ. The disadvantage is that scientists sometimes choose directions that serve their own interests rather than commercial goals. With probability α, they happen to choose what you’d want anyway — their interests align with the commercial value. But with probability 1 − α, they go off and do their own thing. They still get utility Z from pursuing their own research agenda.

In private-sector mode, you retain control. You tell scientists exactly what to work on. They always work in a commercially valuable direction, but you have to pay higher wages — let’s call this Wᴘ — to compensate them for giving up autonomy. The cool thing about this model is that it captures the Stern compensating differential, but embeds it in a sequential, cumulative innovation process.

The question becomes: at what stage of the research process should you switch from academic organization to private-sector organization? Should early-stage research happen in the university and late-stage in firms, or vice versa? The model gives us a clean answer, and I think it has a really cool intuition.

You start with the very last stage. You’re one step away from commercial payoff V. Should you organize this as academic research or private-sector research? If you cede control to scientists, with probability α they do what you want and you get pV; but with probability 1 − α they pursue their own interest and you get nothing. On the other hand, you save money on wages. If you retain control, you always get V, but you pay more. The firm retains control when expected benefits exceed the wage savings, which is basically Z, the utility scientists get from autonomy.

Now step back one stage. You’re two stages away from commercialization. Should you retain control or cede it? Here’s what’s changed. The benefits from control are now further away and more uncertain. You’re not getting V next period; you’re getting the option value of maybe reaching the next stage, which then gives you another option value. These option values are smaller than direct payoffs because they’re risky and discounted. But the wage savings are immediate and certain — you’re paying academic wages right now.

So at earlier stages, the wage advantage of academic organization becomes relatively more important, and the focus advantage of private-sector organization becomes relatively less important. The result is an endogenous cutoff point. Early stages should happen in academia — that’s where wage savings dominate and where you want exploration anyway. Later stages should happen in firms — that’s where focus becomes valuable enough to justify the wage premium. The model tells you exactly where this transition should occur, based on the parameters P, V, Z, α and the wage differential. There are really cool extensions as well — for example, when a research line can branch into multiple projects.

Does science matter for technology?

Understanding how science works as a social institution is important, but there’s an elephant in the room we haven’t addressed: does this matter at all for technological progress? Could we just fund applied R&D directly and skip the whole scientific enterprise?

Vannevar Bush, writing his famous 1945 report ‘Science, the Endless Frontier,’ articulated what we now call the linear model: the idea that basic science flows naturally into applied science, which generates technology, which produces economic growth. It’s an elegant story, and it provided the intellectual foundation for massive public investments in basic research. In the US, the NSF, the NIH, the growth of research universities — all of this was justified by appeal to the linear model.

You might ask: is Bush right? That’s what we’re going to examine in this section. We’ll look at evidence for the linear model, but we’ll also complicate the picture considerably by examining how science and technology interact. Spoiler alert: it’s messier than what Bush thought, but it’s also more interesting. The key insight we’re going to develop is that science doesn’t just generate specific findings that get applied; it provides something more fundamental — a map of the technological landscape that helps innovators search more effectively.

Let me tell you my favorite example of the linear model in action: Thomas Brock and the discovery of Thermus aquaticus.

It’s 1967, and Brock was literally hunting around Yellowstone National Park studying extremophiles — organisms that survive in extreme environments. In the hot springs of Yellowstone, where water temperature fluctuates widely, he discovered the bacterium Thermus aquaticus, which could survive extreme temperature variation.

This was pure curiosity-driven research. Nobody was thinking about practical applications. Brock published his findings, characterized the organism, and — crucially — deposited it in the American Type Culture Collection (ATCC), a research-materials repository. He got tenure. Great. That’s how science works.

Now fast-forward 16 years. Kary Mullis, working for Cetus Corporation, was trying to develop a technique for amplifying DNA. The problem is that you need a DNA polymerase enzyme that can withstand repeated temperature cycles. This was hard — most enzymes degrade at high temperature. After considerable searching, someone (accounts differ on whether it was a graduate student or someone at a seminar) pointed Mullis to Thermus aquaticus, sitting in the ATCC repository. Its DNA polymerase, Taq, was exactly what was needed.

This led to the Polymerase Chain Reaction, which became absolutely foundational for all modern molecular biology and biotechnology. When you see the show CSI, every time someone gets genetic sequencing, it all depends on PCR. Mullis won the Nobel Prize, Thermus aquaticus was named molecule of the year, and the patent on PCR was worth about $500 million.

That’s the linear model in its classical form: curiosity-driven basic research generates knowledge that, 16 years later, becomes the foundation for a transformative technology. The usefulness of extremophiles was completely impossible to anticipate ex ante.

Science as a map

Here’s a puzzle. More science leads to more technological progress. Yet only a minority of patents directly cite scientific papers. So how is science helping?

An important framing point for the papers I’m about to talk about: they all examine science as a type of knowledge, as an input into technological production. They’re not about science as an incentive system — the priority system, the Matthew effect, compensating differentials. None of that is front and center here. Instead, these papers ask: what does scientific knowledge do for inventors? What is its functional role in technological search?

We’re going to begin with Fleming and Sorenson. Their answer is: science provides a map. Scientific knowledge tells us how the world works — underlying principles and mechanisms. Technology captures and orchestrates these regularities to do something useful. But new technology steps into the unknown, either using novel processes or combining existing ones in novel ways. Science can help inventors navigate the uncertain, unknown terrain. It provides an imperfect map of the space. It tells you where to look, what combinations might work, which are likely to fail. Science reduces search costs by providing cognitive guidance.

Here’s the key insight: not all technologies benefit equally. Some technologies are forgiving — you can combine components many different ways and things still work. Think modular systems: for these, trial and error is fine; you don’t need deep scientific understanding. But other technologies are what you might call fussy: components must combine precisely, or everything falls apart. Small changes can cascade into big problems. For these tightly coupled systems, trial and error is prohibitively expensive. So having a scientific map becomes valuable, because it lets you avoid the vast space of failed combinations and directs you toward the narrow space where success is at least possible.

That’s what Fleming and Sorenson test: does science matter more for fussy technologies? That’s the key result. It’s an interaction effect in a cross-sectional regression.

Patents that cite science always tend to receive more citations than patents that don’t — that’s the baseline science premium we’ve known about for a long time. But what they show is that the gap between patents that refer to science and patents that do not is actually much wider for those so-called “fussy” technologies.

Let’s try to unpack what that means. If you’re inventing something where components are loosely coupled — modular, flexible, forgiving — you can afford to do a lot of trial and error. Having scientific understanding helps a bit, but the gains are modest; you’ll probably figure it out eventually through experimentation. But if you’re inventing something where components are tightly coupled, where small changes in one component cascade through the system and break everything, trial and error is very expensive. The search space is huge and most of it leads to failure. So having a scientific map, which tells you which regions of that search space are worth exploring, becomes really valuable.

There are follow-on papers that do this much better. One is the paper by Ahmadpoor and Jones called ‘The Dual Frontier.’ But Fleming and Sorenson was the first paper of that type, and it’s been very influential.

Pasteur’s Quadrant: basic and applied at once

We’ve seen evidence that science matters for technology through multiple mechanisms. That raises a natural follow-up question: how should we organize the boundary between the activities of science and the activities of technology? What kinds of institutional arrangements best support research that connects scientific understanding with technological application?

The linear model gave us a clean answer: keep them separate. Universities do basic research; firms do applied research and development; knowledge flows from universities to firms through publications. Simple. But is that how the world works?

To tackle this question, we need to rethink some categories. The linear model treats basic and applied as sequential stages: first you do curiosity-driven basic research, then someone else does goal-oriented applied research. But as it turns out, that’s too simple, because research can live in both worlds simultaneously. It can be both basic and applied. So you get an inherent tension between the scientific norms and the commercial norms. Let’s explore that.

What you’re getting in this lecture is a bit of a greatest-hits of examples. We keep coming back to those examples because they can really help tell the story. Thermus aquaticus was one. Another is the OncoMouse.

In 1984, Philip Leder and Timothy Stewart at Harvard Medical School developed the first mouse with genes inserted that predisposed it to cancer. They used an oncogene — a gene that, when activated, promotes tumor development. This is simultaneously a scientific breakthrough and a very practical tool.

On the scientific side, it demonstrates that you can insert specific genes and produce predictable effects on cancer susceptibility. They’re testing a fundamental hypothesis about the genetic basis of cancer. It’s advancing our understanding of why: why do organisms develop cancer? What’s the role of specific genes?

At the same time, it’s a how. How do you test potential cancer therapies? If you want to develop new drugs, having mice that reliably develop particular forms of cancer is enormously valuable. You can test whether your drug prevents tumors, shrinks existing tumors, extends survival. So this OncoMouse is simultaneously answering a scientific question and providing a very practical research tool.

Leder did what good scientists do: he published in Cell, one of the top journals. The scientific priority is established. But Harvard also did what universities increasingly do post Bayh–Dole: they filed for a patent, which was granted in 1988, and licensed the patent exclusively to the firm DuPont.

This is where the friction starts. DuPont didn’t just want to sell mice. They wanted reach-through rights: the idea is that if you use the OncoMouse to develop a therapy, DuPont gets a piece of the action. They wanted the right to review publications using their mice. This created enormous tensions with the research community. Scientists felt these restrictions violated the norms of open science. But from DuPont’s perspective, they’d made a major investment and wanted to earn a return on it.

This framework, developed by Donald Stokes, reconceptualizes the basic/applied distinction in a way that’s much more useful than the linear model.

Instead of treating basic and applied as sequential stages, Stokes says they’re independent dimensions. On one axis you have quest for fundamental understanding: are you trying to advance scientific knowledge about how the world works? On the other axis, considerations of use: are you motivated by a practical problem? Are you trying to develop something useful?

This gives you four quadrants:

• Pure basic research — high on fundamental understanding, low on considerations of use. We could call it Bohr’s quadrant. Niels Bohr wanted to understand the structure of the atom; he didn’t care about applications. Thousands of papers, zero patents.

• Pure applied research — low on fundamental understanding, high on use. We might call that the Edison quadrant. Thomas Edison ran thousands of experiments at Menlo Park trying to make better light bulbs, better phonographs, better everything. He explicitly didn’t care about publishing scientific results; he refused to let anyone publish about the Edison effect, even though it was named after him. Thousands of patents, zero papers.

• Use-inspired basic research — high on both dimensions. This is the Pasteur quadrant, named after Louis Pasteur. Pasteur was working as a consultant for the French wine and dairy industries — very applied problems: why does wine ferment? How do we preserve milk? But his investigation led to the germ theory of disease, one of the fundamental scientific breakthroughs of the 19th century. Pasteurization is simultaneously an industrial process and a profound scientific insight.

The most important research often lives in Pasteur’s quadrant. It’s simultaneously advancing fundamental understanding and solving a practical problem. The OncoMouse and most modern biotech live in Pasteur’s quadrant. This creates inherent tensions, because you’re trying to satisfy both scientific norms and commercial incentives at the same time.

Openness as a natural experiment: Of Mice and Academics

The next study is Murray et al., poetically named ‘Of Mice and Academics.’ These authors — you can notice the author overlap with the ADS 2008 paper — examine a natural experiment in openness. They’re testing the ADS framework empirically using the OncoMouse story.

The theoretical framework from Aghion, Dewatripont, and Stein suggests two distinct effects of openness. Vertical exploitation means more follow-on research: if access costs are lower, more people can build directly on existing work. But there’s also horizontal exploration: openness might enable entirely new researchers to enter the field and pursue novel research directions that wouldn’t have been explored under restricted access.

Here’s the natural experiment. By the mid-1990s, DuPont held patents on two crucial mouse-engineering technologies, Cre-lox and OncoMouse. These patents covered hundreds of different engineered mice, and access was highly restricted — complex licensing agreements with reach-through rights on anything you discovered using the mice, which created significant delays and costs. The research community lobbied for years for more open access.

In 1998 and 1999, the NIH director Harold Varmus — who had won the Nobel Prize and understood the scientific stakes — negotiated memoranda of understanding between the NIH, DuPont, and the Jackson Laboratory, the mouse repository. These agreements dramatically reduced access costs for academic researchers: simple one-page material transfer agreements replaced complex licensing, and the Jackson Lab was committed to distributing mice openly.

The point is that this memorandum was largely unanticipated in its timing and its scope. There had been lobbying for decades, so researchers couldn’t time their work around it. The agreements covered the entire population of DuPont-controlled mice, so there’s no selection bias. Crucially, there’s a natural control group: knockout mice and spontaneous-mutation mice were created using different methods and weren’t subject to DuPont’s restrictions, so their access costs didn’t change. This lets Murray and co-authors implement a very clean difference-in-differences design.

This is the key result for vertical exploitation.

Here again, I’m not doing full justice to the paper — but it’s a great paper, you should absolutely read it. The figure shows annual citations to mouse articles over time. What’s a mouse article? Each time you create an engineered mouse like this, there’s a paper associated with the mouse. You can then think about the extent to which follow-on researchers cite this mouse in their own research. You can do it for treated mouse papers and for control mouse papers — hence the diff-in-diff.

The figure shows the annual citations to mouse articles over time, comparing treatment — OncoMouse and Cre-lox — to control: knockout and spontaneous. The vertical line marks the NIH agreements in ’98-’99. Before the agreements, the two groups track each other relatively closely. Citations to both treatment and control mice are declining over time, which is natural: all papers get cited less as the field moves on. But the rates of decline are parallel. This validates the parallel-trends assumption.

After the agreements, you see a clear divergence. Citations to treatment mice increase substantially relative to the control group. The boost is 20-30%, depending on the specification. So that’s vertical exploitation: more follow-on research using these now-open research tools. This makes intuitive sense. Before the agreements, many researchers who might have wanted to use OncoMouse or Cre-lox mice were deterred by access costs and licensing restrictions. They either used different tools, chose different research questions, or left the field entirely. After the agreements removed the barriers, more researchers could access the tools they needed, and follow-on research accelerated.

Notice one thing that didn’t happen: citations to the control group don’t decrease. This isn’t researchers substituting from knockout mice to OncoMouse. If that were happening, you’d see the control group decline when the treatment group rises. Instead, both increase, with treatment increasing more. The openness shock expanded the total amount of research; it didn’t just redistribute research across tools. This is important, because it suggests the pre-agreement restrictions were genuinely blocking research that had positive social value, not just creating a queue that would have happened anyway. There are additional findings that are very interesting in this paper. There’s also the horizontal exploration effect: they provide evidence of much more diverse experimentation.

Funding design and tolerance for failure

We focused on one particular institutional friction with this paper: what happens when scientific and commercial incentives collide at the frontier, especially in fields like biotech that live in Pasteur’s quadrant. But there are many other ways that institutional design affects productivity within science, independent of commercialization. How do scientists access frontier knowledge produced elsewhere? How do funding agencies decide which research to support? How do scientists access physical research materials they need to build on each other’s work?

This is where I’ve done most of my research. I’ll tell you about just one paper. It’s about science funding, tolerance for failure, and scientific exploration. It has been and will continue to be a very fertile area of research. It’s called ‘Incentives and Creativity: Evidence from the Academic Life Sciences.’ We want to think about whether what explains scientific productivity at the individual level is not just the amount of funding people receive, but how they’re funded — the incentives inherent to the funding contract.

What we want are scientists that are at risk of receiving grants with different incentive structures. We want to see if particular types of grants are associated with greater scientific creativity. To do that, we study the Howard Hughes Medical Institute (HHMI) investigator program. I talked about it earlier with the Matthew effect study. HHMI is the most important private funder of academic biomedical research in the US, and probably the world.

Roughly every three years they select 50 youngish biologists from elite institutions. It’s very prestigious — that’s why we studied it as a status shock. If you get appointed to HHMI, it becomes part of your institutional identity: you might be MIT and HHMI, or Sloan Kettering Institute and HHMI. But it’s also a major source of funding for those scientists.

What does HHMI tell people? They say: “Your job is to change your field.” Here is a pot of money; in five years we’re going to ask whether you’ve used that pot to change the nature of the questions being asked in your field. It’s not just a prize. A prize would be: “You won, we give you money, and then our job is done.” This has real bite, meaning an HHMI appointment is something you are at risk of losing.

What’s important and interesting is that I said it’s five years — that’s not true. It’s really ten years, because at the end of the first five years as an HHMI, what HHMI wants to know is: have you used the freedom you were granted? Have you at least tried? The axe is really going to fall at the end of the second five years. So ten years in, they start asking, “You’ve had ten years to try to change your field, have you?” At that point lots of people are being discontinued — kicked off the program. But that’s a much longer time horizon relative to the NIH R01 grants, which are the standard government grants in biomedical research in the US, where it’s 3-5 years. It’s really not forgiving of failure — people are going to scrutinize all the stuff you’ve published in the last 3-5 years.

Importantly, the NIH funds you for a particular project, whereas HHMI funds the person. That means if, in the middle of your 5-10 years, you decide that the idea you had isn’t going to work, you can just use the funding to do something else instead — which is not possible at the NIH.

What we did in this project is — we’d have loved it if HHMI were ready to randomize after a first cut of selection into their program. They’re not willing to do that. They also don’t really have a regression-discontinuity design — a hard cutoff. What we ended up doing is selection on observables. This is the picture.

It’s a kind of synthetic-control study, if you want to place it in the methodological landscape. We have lots of different outcomes, but relative to the controls, the HHMIs don’t produce more papers — but they produce more important papers. They also fail more often, meaning they produce more papers that are complete citation duds. They swing more for the fences than the controls. The controls revert to their mean, whereas the treated scientists really expand and enlarge an impactful research agenda following selection into the program.

I’m papering over lots of details, and you can read it. This paper was published in 2011. There are tons of things that are wrong with it, and we try to level with the reader on that. It’s totally possible to do much better than what we’ve done. But today it’s my second most highly cited paper, and that’s because it was the first one that asked that type of question.

Synthesis

Let me try to synthesize what we’ve learned.

We asked whether science is more than just a type of knowledge, and the answer is yes. Science is a distinctive economic institution with specific incentive structures that solve market failures in knowledge production. The priority-based reward system, where scientists compete to be first to publish, creates incentive compatibility: it aligns private incentives with the social benefit of rapid knowledge disclosure.

This solves what we might call the paradox of directly paying for ideas. How do you reward discovery when knowledge is non-rival and hard to exclude? The priority system does this by turning knowledge production into a winner-take-all race. Actually, it turns out it’s not winner-take-all, but maybe winner-take-most, where the reward is reputation rather than control rights over the knowledge itself.

The question you want to ask is: is science really central to endogenous growth and technical progress? We’ve seen some evidence today that disruptions to open science reduce both scientific productivity and downstream technological innovation. Institutional design matters just as much as funding levels. The details — how we structure access to knowledge and materials, how we design funding mechanisms, how we balance openness against IP — those are first-order policy instruments. Strategic behavior can undermine open-science objectives, and the institutions that worked well in 1950 may not be optimal for 21st-century science. The big question is: how should we evolve them? That’s one of the frontier questions for both policy and research.

He (Waldinger) estimates the effect of shocks to human and physical capital during World War II. He can look at random destruction in physical capital — university buildings in Germany — and at departments that lost star scientists. He uses the dismissal of Jewish scientists in Nazi Germany between 1933 and 1944 as the shock to human capital. He attributes about a 33% decline in publications, at the department level [to dismissals]. He contrasts that with the destruction of universities during the Allied bombing campaign...He’s able to use this as an exogenous shock — some departments were affected by these bombings and some weren’t…What’s striking about it is that the decline in publications is much larger relative to the decline due to the physical capital. He has this quote: “The dismissal of scientists in Nazi Germany contributed about nine times more to the decline of German science than physical destruction during World War II.”

On one level this might seem obvious — buildings don’t write academic papers; scientists do. But we might expect that access to working labs would be necessary for scientists to produce research (especially before the advent of personal computers and the internet). And it is important, but not nearly as crucial as having talented colleagues.

Good science (depends on) the friends we make along the way.

Subscribe now

Here are the readings that go with this lecture:

• Agarwal, R. and P. Gaule. Invisible geniuses: Could the knowledge frontier advance faster? American Economic Review: Insights, 2(4) (2020): 409-24.

• Bell, Alexander M., Raj Chetty, Xavier Jaravel, Neviana Petkova, and John Van Reenen. Who Becomes an Inventor in America? The Importance of Exposure to Innovation. Quarterly Journal of Economics, 134(2) (2019): 647-713.

• Ganguli, I., P. Gaulé, and D.V. Čugalj. Chasing the academic dream: Biased beliefs and scientific labor markets. Journal of Economic Behavior & Organization, 202 (2022): 17-33.

• Waldinger, Fabian. Bombs, Brains, and Science: The Role of Human and Physical Capital for the Production of Scientific Knowledge. The Review of Economics and Statistics, vol. 98, no. 5 (2016): 811-831.
  

  Subscribe now

Thanks to William Higbie and Harry Fletcher-Wood for their support in producing this video and the transcript.

Subscribe now

Lecture Transcript:

The Supply of Innovators

My name is Ina Ganguli. I’m a professor in the economics department and the management department at the University of Massachusetts Amherst. I was trained as a labor economist, then I got into the economics of science and innovation — my research is at the intersection of those two fields.

I’m going to talk about what I call “the supply of innovators” and knowledge production, which is at that intersection. I’ll tie it back to things that Ben and Chad talked about — going from the macro to looking very micro at the decisions that people make. Then I’ll take you back to the early days of what labor economists who worked on these topics were thinking about, going through some models and conceptual approaches. I broadly call it “earnings” and “entry into science.” We’ll go through several models and some of the empirical evidence. I’ll talk about:

• Cobweb models,

• Compensating differentials. One of the papers that I’ll cover looks at how much scientists pay to do science. There’s a well-known Scott Stern paper.

• The Roy model, which is a way to think about selection: who self-selects into the innovative sector.

• Then, getting into the broader labor economics literature, what have we started to look at in this area of the economics of innovation: biased beliefs, exposure to science, diversity and the economics of talent.

John van Reenen will talk about this too, but there’s this literature on the lost Einsteins that’s gotten to be quite popular recently — looking at what happens to talent that isn’t nurtured.

We’re going from the macro: thinking about endogenous growth models and how knowledge is produced. Those of us who work in this area on labor and innovation think very micro: what are the choices that individual people make?

Here we think of a simple Romer-type model.

LA is the number of people searching for new ideas. If we’re going to have ideas produced, you’re going to have people who are searching for them. Things are changing now — we can think about AI replacing those people. But back to the basic model: we have LA, and if you increase LA — the number of people — then you will get more ideas produced. We’re going to examine LA. Who’s in it? The number of people.

If we think about how to measure that — who is it that’s searching for new ideas? — there are different ways you can think about it.

• People who work a lot with patent data will look at inventors: people who have invented an idea that’s been patented, conditional on patenting.

• A lot of my research — the research of those of us who work on scientists — will use publication data and then back out scientists from that.

This is data from different sources — an estimated number of researchers across countries.

You can see that in most countries it’s increasing. I’m not going to have time to show a lot of data, but I’ve done some work recently with Megan MacGarvie where we’ve looked at international students studying in the US and trends from different countries. There’s work showing that maybe these researchers haven’t been that productive: the number of researchers is going up, but whether they’re actually producing as many new ideas is something that people have been looking at. Right now in the US — but I think in other countries too — this topic is very policy relevant. There’s a lot of focus on what’s happening to the scientific workforce.

There was a recent Science article talking about different aspects of the Trump administration’s agenda and that it’s likely to reshape the scientific workforce. Why is that? What kind of policies are we talking about?

Immigration: restricting immigration from certain countries, or making it more costly to get an H-1B visa — which is what you get if you’re coming as a high-skilled worker to work in a company. Now that’s expanding to universities. Michael Clemens will talk much more about that.

There’s also government scientists — there have been a lot of reductions in the government scientific workforce. There’s funding that has been reduced, and so fewer students can be hired to be graduate students. There’s a lot going on right now. I will say, as someone who works on these topics, in the last year I’ve had a lot of journalists come and ask me: “What’s the evidence? Will there be a brain drain? What will the impacts be on US science?” Those of you who are interested in this topic, it’s an important time to be working on it, and I hope some of you will think about getting into these issues.

If we’re thinking about the different inputs into the knowledge production function, there’s human capital, but also physical capital — the lab equipment. We can think about computers, AI potentially: how do we think about that — is it human or not? Here I want to think about the people. How important are the people?

I really like this paper by Fabian Waldinger. He’s done a lot of great work on World War II and science, and in particular the emigration of Jewish scientists from Germany. This paper focuses on this question of human versus physical capital. It’s a motivating piece of evidence for me when I do my work on the decisions that scientists make.

He estimates the effect of shocks to human and physical capital during World War II. He can look at random destruction in physical capital — university buildings in Germany — and at departments that lost star scientists. He uses the dismissal of Jewish scientists in Nazi Germany between 1933 and 1944 as the shock to human capital. He attributes about a 33% decline in publications, at the department level [to this]. He contrasts that with the destruction of universities during the Allied bombing campaign of World War II. He’s able to use this as an exogenous shock — some departments were affected by these bombings and some weren’t. That’s the physical capital shock. What’s striking about it is that the decline in publications is much larger relative to the decline due to the physical capital. He has this quote: “The dismissal of scientists in Nazi Germany contributed about nine times more to the decline of German science than physical destruction during World War II.”

I’ve done a little work recently with Ukrainian researchers, looking at the impacts of the full-scale invasion of Russia in Ukraine. They’re really worried about brain drain — scientists leaving — and just measuring that. There’s this idea that yes, there’s a lot of destruction and horrible bombings, but some of the scientists say that if we can build it back, it actually can be built back better — better labs and equipment. You can build back labs pretty quickly if you have the resources, but it’s really hard to bring back the people who have left. That’s one of the takeaways from Fabian’s work.

This is a little bit small here, but this is one of the tables.

You can see that he’s got the data on universities. There’s this variation in the intensity of how many Jewish scientists left during this time. Destruction is the bombing. He has these dismissal shocks and bombing shocks, and he looks at them by field, and that’s where he’s able to back out his estimates.

One thing then to think about, when I was talking about how to measure who is an innovator — who belongs in LA — is what it means to be an innovation worker versus a production worker. That’s really the question for today: what are the factors that shape who is part of the supply of innovators?

Everyone listening either has a PhD or is doing a PhD. As you know, it’s a long process to actually get to the frontier. That’s a specific feature of this group of workers: they need to bring themselves to the frontier before they can be productive. This idea of the gestation period — it’s long, and it’s getting longer. Ben talked about his “burden of knowledge” work [in Lecture 1] — it takes a long time to get the knowledge and resources to be able to be at the frontier.

A few other features: job prospects at the time of graduation are difficult to predict in advance, because when you enter the labor market, that’s many years into the future. If we think about models that economists use where people compare expected salaries in different jobs when choosing what to study, it’s actually hard at the time you make your decision to predict that. I’ll talk about this idea of biased beliefs, and the fact that people often don’t have reliable information about job outcomes of recent grads.

Pierre will probably talk more about this, but if you’ve read much Merton, about norms in science and what motivates scientists — it may be that scientists love the subject. As economists, how do we think about those motives for doing science that are not extrinsic? Merton talks about ribbon, gold, and puzzle as things that motivate scientists. There are things that, when you’re studying this group, may be different than other models that economists have.

This is a great segue into who selects into the idea sector. The idea sector is very heterogeneous: what people are doing is going to differ. But here we’re going to think about the fact that you need to have some skill, to specialize, and to reach the frontier. Some of the questions are:

• Do institutions provide the right incentives for the right people to work on innovations?

• What kind of policies and shocks stimulate entry into STEM careers?

• Who is in the pipeline that produces innovation?

• Are there barriers limiting diversity?

I took this from Pierre — this was his image of someone who I think we all recognize.

If we think about Einstein, we all know he worked in the patent office early on. What if he didn’t choose to go into academia? Today, if you look at some undergrads with quantitative skills, maybe they’re going into private equity or hedge funds. What about that? As a society, what would we want? I’m guessing most of us would probably agree we want Einstein to go into academia. But what if he didn’t choose it, or what if he actually wanted to just earn a lot of money? These get into interesting questions.

There are so many headlines, but in this time right now we have a lot of scientists — innovators — who have now left the government. What is going to happen to that pipeline in the US is a topic that’s really important.

With AI, there’s also the question about the computer science labor market and the role of AI.

In the last year or two, things have changed a lot, and so there’s a lot to think about in this topic.

The role of China: I won’t get to talk about international mobility today, but this is a big topic — where is the talent going to go? The US used to be more or less the top destination for international students and scientists, but that’s changing as where the science is being done is changing.

This is a very interesting graph that I’m hoping someone will work on to try to understand the causal factors.

This is the share of students who plan to major in computer science by gender, from 1970 to 2015. What people always talk about is that there’s pretty much parity around the ‘80s — this is from a survey of freshmen. What you can see is that it then spreads out, and even today there’s been a bit of an uptick, but for a while there were very few women compared to men.

I’ve heard some hypotheses that it was because parents bought their sons personal computers, so when they got to college in the ‘80s they were ready to go anyway. It’s interesting, all the different factors that can play a role in who ends up being in that pipeline.

Cobweb models

I want to get into the first model: cobweb models. This is Richard Freeman, who was my adviser.

He’s a labor economist, but one of the topics that he worked on in the ‘70s is looking at the US engineering labor market. This is the post-war period. He was looking at some data — Richard is really great about understanding the data and looking at the big picture trends. Sometimes we’re so interested in the causal questions that we look very narrowly. He had been looking at freshmen in engineering majors and noticed this interesting trend of the share who were enrolling versus the share graduating. What he noticed with this pattern was an interesting feature, for which he developed the cobweb model. It’s a very simple way of thinking about the labor market, but there are some aspects that can be useful when we think about what’s going on, even today.

What he recognized is that salaries four years earlier determine entry decisions. People in this engineering labor market would look at what the salaries of engineers are and then decide, “Engineers earn a lot of money. I’m going to be an engineering major.” But by the time they graduate, the labor market conditions have changed. This is very relevant now. I don’t know how many of you have talked to people in computer science departments, but even here at UMass there have been some big changes in enrollment of undergrads in computer science, because the labor market is not very good if you’re just graduating with a computer science degree.

This was from The Economist not too long ago, where they have job postings — job openings for software developers — and you can see there’s been a decline very recently.

At some point a few years ago it was, “Computer science majors are earning a lot, this is a hot field,” and then things changed.

That’s the basic idea with the cobweb model: that labor markets don’t adjust quickly to shifts in supply and demand. There are two assumptions that are needed:

1. Time is needed to produce skilled workers. You need some time to be an engineer or be someone who’s working on AI — not working with it, but developing the AI models.

2. People decide to become skilled workers by looking at conditions in the labor market at the time that they enter school.

What happens is that a “cobweb” pattern forms around the equilibrium, which arises when people are misinformed. It’s really this idea of misinformation, which is a thread that runs through most of the concepts we’ll talk about today. If we look at most models, we assume that people have information about the labor market — so what happens when you don’t have that?

You might have seen this in an undergraduate labor course.

The basic idea is that you start at some equilibrium employment and wage. Then you have some demand shock. Let’s say for AI specialists there was increased demand. That’s going to shift your demand curve out. In the short run you’re going to have an increase in the wage. With the demand shock, you go up to W1. But in the short run there’s a limited supply of, in this case, engineers — they aren’t produced immediately — so you have this inelastic short-run supply curve. Students are going to see that you have this higher wage, so you’ll have more students enter. But when they enter the market they’re only going to earn this lower W2 wage. This continues for a while, and you get this cobweb pattern.

The big picture is that in the long run, that initial demand shock would lead to a higher equilibrium wage and employment, but it’s a series of booms and busts that get you eventually to that point. As students thinking about what you’re going to major in — there’s a whole literature on biased beliefs; that students often don’t even know what salaries are in different fields — but even if they did, that’s current. How can you predict what it’s going to be in the future? This is an older model, a pretty simple way of thinking about the labor market, but there are some useful features for those of you who are thinking about these topics.

Compensating differentials

I’m going to talk very briefly about compensating differentials, because Pierre will talk about this paper more. In the labor economics literature there’s been a lot of interest in this — there have been new papers in the last few years trying to understand what workers value in a job. You get your wage, but you may actually be willing to take a lower wage if there are some characteristics of the job that you value.

Going back to Adam Smith, it’s the advantages and disadvantages of the job that have to be equated across jobs. The original way of thinking about it is that if you have a firm with unpleasant working conditions — say very manual work, or very hot on the job — then those jobs should pay workers more. Jobs that offer more pleasant conditions could offer workers less, because workers are willing to give up some of their pay for the pleasantness.

There’s a whole literature on how you can estimate that price for these non-wage characteristics. It’s challenging, because if we just go to a data set — say one that shows what workers earn and the features of their job — you would actually tend to find that the more pleasant jobs are paid more, because there’s a whole bunch of things, like education, that are correlated with the kinds of jobs that people are getting.

There have been experimental approaches to try to measure this. I went from Richard to Scott Stern, who in 2004 wrote this paper, “Do scientists pay to do science?” It was an exciting paper because it was one of the first that came up with this quasi-experimental approach to measure this trade-off. He’s measuring this wage-versus-science trade-off. This goes back to what we’re talking about with Merton: that there are things that scientists value. Going back to the discussion earlier about who innovators are — in this case we’re thinking about scientists, who might have very particular preferences for doing science, or for having autonomy in what they’re going to research.

Stern’s method was a survey of postdocs in biology — within one field — and he was able to ask them about their different job offers, the wages, and the amenities. He had this data set that had a very small sample size — 66 individuals and 164 offers. Today you probably couldn’t get away with such a small sample, but at the time it was really exciting, because he could look at one individual, look at the different offers that they had, and estimate what’s called willingness to pay.

Once you put in individual fixed effects — you account for these unobserved individual differences — he estimates a willingness to pay of 20-25% lower wage to be able to do their own research. Why is this important? One of the questions that people are often interested in is who wants to go into industry versus academia? He was able to show that people really value being able to do research — if the firm permits publication, they’re willing to take a lower salary. It gets into this idea of what motivates scientists.

This was one of the regression tables.

He has self-reported salary. There have been recent studies — not with scientists, but in the labor literature — where they were able to randomize job offers and salaries. Obviously the world is different in terms of what would be evidenced in this space, but this was a very important paper in terms of the approach.

Earnings & Entrance into Science: the Roy model

I don’t know how many of you are familiar with the Roy model. If you’ve taken a labor class, then you will have been exposed. Thinking back to industry versus academia, or finance versus biology, the question we’re thinking about is how workers sort themselves across different sectors. You could also think about the idea sector versus management, or whatever you’re going to choose to work in.

It’s always interesting to go back to the paper. It was all words, no equations. This is the first page of Roy.

The original Roy was actually about workers choosing between hunting or fishing. You might have seen the Roy model used a lot in the immigration literature, thinking about who chooses to immigrate to different places, or not. Borjas 1986 is the reference that people often use. He essentially took Roy and wrote down the equations, so that you could have the mathematical setup to look at selection.

Here’s a very simple version.

The idea is that you’re going to have workers who are making decisions about which sector to work in. It’s a pretty simplistic way of thinking about it, and obviously you can get to more complex models, but essentially workers are comparing earnings in each sector. Their objective is to maximize earnings. There’s positive selection, where the very skilled — the people in the upper part in the distribution of skills on the x-axis — if they’re going to choose a given sector, then they’re going to be positively selected. If it tends to be people from the lower part of the skill distribution, then that’s called negative selection. One of the Roy model’s predictions is whether there’s going to be positive or negative selection, based on the distribution of wages in a country — how compressed they are.

For our world, when we think about innovators, the other question becomes whether there’s correlation between the skills that individuals have in the two sectors that people are thinking about. You want to think about an individual that we’re observing choosing between the idea sector and finance. The question is: are the skills needed in finance and the skills needed in the idea sector correlated?

I wanted to point you to this paper by Shu. We were talking about different sources of data. Patents are bread and butter, but I urge you to think about sources of data that you might have access to that people haven’t used before. Shu got great data on different cohorts of MIT students, with all kinds of things like the courses they took, their GPA, even in high school, and was able to link that to patents as well. One of the papers was looking at this question of the Roy model and the decision about working in finance versus in STEM. This issue of whether the best and the brightest are actually going on to finance, and whether that’s what we want as a society, is an interesting question.

What she finds — using data on what students are doing in college and in high school — is that finance doesn’t attract the best and the brightest STEM students, but it does attract those with more finance-relevant skills. She essentially shows that finance and science and engineering demand different skills, and still finds that the best STEM students have a preference for going into science and engineering.

She finds what is called negative correlation of skills: the skills required are very different in students going into finance versus science. Those going into STEM have better academic records in high school and focus more on developing academic skills during college. This is where it’s interesting if you know the literature by David Deming on the importance of social skills — people right now are also talking about how, with AI, that people who have the social skills might be the ones who do better in the labor market. She found evidence that people entering finance had more leadership experiences in high school and focused more on developing social skills during college. The idea is people selecting into each sector have different skills.

She also has some work looking at the financial crisis in 2008 and shows that that’s a shock to the returns to doing finance.

After the crisis, those who stayed in finance actually have better academic qualifications. She can look at who’s the marginal student who’s going to stay in finance versus not.

That’s just to give you a flavor of the Roy model. There have been some recent papers that also use the Roy setup. You can think about a setting where you have a change in the returns to a degree, or in the salaries, and you can look at changes in who selects. She has this admission index, which is her measure of skill — you need to observe something about the skills or the ability of the individuals, to understand who are the more qualified, choosing, let’s say, the idea sector versus not.

Earnings & Entrance into Science: Biased Beliefs

Here we’re going into the role of information. This has also gotten to be a very popular topic in general labor economics. For our world of the economics of science and innovation, there is a focus on PhD students, because in some fields, to be an innovator, you need to get a PhD, and it’s a long process. There is a lot of focus on this group: people who choose to do a PhD and then go on to do a postdoc.

Several years ago there started to be this discussion that the job market for PhD students was really bad. There were a lot of discussions around whether we have enough jobs, and whether students know. One thing we know in economics is that it’s pretty transparent — you can go and see from different departments what the placements are of PhD students. In the sciences, most people do a postdoc, and so it’s actually hard for students to see what the careers are like of people in their field.

PhD and postdoc training — if we go especially to the STEM fields — is designed to prepare individuals for academic careers. It used to be more so. Now we’ve seen that there’s much more awareness, by faculty PIs, that students are going into industry or entrepreneurship. But in STEM fields, about 10% of PhD graduates go on to get a faculty position. In 2016, there were 2,700 new chemistry PhDs, but only 152 openings in research universities.

You probably all know the PhD Comics, about how ambition changes as you go along the PhD.

At the beginning, you think, “I’m going to win the Nobel Prize,” and then at some point you start to get more information.

How do you get this information? Some of it is through your peers, but should there be more structured effort to have data for PhDs? I’m going to tell you about a study that I did with some co-authors where we tried to look at this. There was this idea that students don’t know what the job prospects are when they’re finishing. The labor literature had grown a lot on this topic — that students are not fully informed when making educational choices. Experimentally, there were several influential studies that showed that if you give accurate information, students update their beliefs and their decisions. There was a paper by Rob Jensen giving students information on the returns. There was work by Wiswall and Zafar showing that NYU students, if you give them information on salaries in different careers, change their majors. It’s been pretty well shown, across a range of settings, that students and parents don’t have accurate information.

I wanted to talk to you about this paper because one of the things in this field of the economics of science and innovation is that there are still not a lot of papers where they’re doing experiments. I know that there are some efforts with J-PAL and the Innovation Growth Lab to try to encourage and provide resources to do experiments. If you’re thinking about doing experiments, that’s exciting. I’m happy to talk more; I’ve been involved in a few of them.

This one is still challenging to do, but it’s an easier way to do an experiment, which is embedding an experiment within a survey. We wanted to see whether our PhD students are informed about the labor market, and whether providing information can change their beliefs. We did a several-year project where we did a survey of about 1,300 PhD students in chemistry in 2017. If I had more time, I could tell you about just getting the email addresses of these PhD students. We did 54 US chemistry departments. There’s no list out there of who all the PhD students are in a field, so we had to scrape those from websites.

Then we did a randomized information intervention. The focus was providing a subset of students — since it was an experiment — actual historical academic placement records by program. We showed them a website that had, by department, what share of graduates got a tenure-track faculty job at a research university. As you can imagine, it was quite low: in most places it was 5-10%. Then, in another treatment, we also linked to scientist profiles on the American Chemical Society web page.

We did a follow-up survey one year later, and we tried to look at placement data four years later. The whole thing took about five or six years to get the paper finished. In our field there’s so much of a focus on causal estimates. This is a paper, and I have another one like this, where there often ends up being a lot of interest in the descriptive results too — just showing what the beliefs are.

There’s some related work by Henry Sauermann and Michael Roach, where they also did a big survey of many PhD students and tracked them over time in different fields. They didn’t do an experiment, but they also have some of the descriptive pieces that are very interesting. The takeaway, if you just look at the beliefs, is that people are — we called it “at times excessively optimistic.” They were probably very good. But in general, people did not seem to be informed about their actual chances. There’s some interesting variation: international students, and students earlier in the program were especially overly optimistic.

This was one of the descriptive pieces — this is the share of PhDs from their own program getting US tenure-track positions.

The mean was 24.5%, the median was 20%, but about 5% was the actual number. Most of the sample was reporting above that.

We did this information intervention. One of the issues with this project was that the results were a bit puzzling: we found that they did update their beliefs about their own chances of becoming faculty, but not about the market overall. It was a puzzle of why they would update. We also didn’t find any gender differences, which people are always interested in. Then we waited and collected — back in the days before what everyone’s using now, getting LinkedIn data — we looked up each person and tried to see whether they actually did a postdoc after graduation. We found no change across the treatment and control in doing a postdoc. Once people do the PhD, they’re down that path. Maybe providing this information earlier — but the issue is that so few people of the general undergraduate population would want to go on to do a PhD, so you’d have to think about what the right sample would be. That could be an interesting thing to do.

Here’s some of the data on initial and final beliefs.

You can see that there’s a bit of a downward adjustment, and in particular the people who were the most misinformed — who had the highest initial beliefs about their own chances.

The takeaways are that there might be other reasons that people do a postdoc, thinking about international students, the fact that a postdoc can still be valuable if you’re going to choose an industry career, family reasons, and maybe their preferences are quite stable, because you’re down this path.

Earnings & Entrance into Science: Exposure to Science

This is a very recent topic I’ve gotten into, which is this idea of exposure to science. Going back to who chooses to go into this innovative workforce — you could think about entrepreneurs, or invention. What I’ve been working on is this idea of who decides to go on to do a PhD. We know that one of the biggest predictors of who goes on to do a PhD is if your parents have one. What is the reason behind it? Is it because you have more information, a role model?

What we wanted to get into is this topic of being exposed to what science is. There are a lot of stories. You probably know Jennifer Doudna, who won the Nobel Prize for her work with CRISPR, and was also involved in the last patent interference, the dispute between Doudna and Zhang on who gets the patent rights to CRISPR. If you look into her background, she talks about how she went to a smaller college, and she said, “My biochemistry professor in college gave me a chance to work in her lab over the summer, which is critical, where I really figured out, ‘Wow! I love lab work, this is really great, this is exactly what I want to be doing.’”

What if Jennifer Doudna had not gone and worked in this lab? Would she have gone into science, and would we have had CRISPR? That’s a counterfactual we don’t know, but in this paper we’re trying to get close to that.

There was a big scandal on econ Twitter when there was a discussion about the $15-an-hour minimum wage, and Jennifer Doleac said, “I’m required to pay undergrad RAs $15 an hour. I will definitely hire fewer of them. The silver lining: less paperwork for me.”

People were shocked that she would say this: “You’re not going to pay your undergrads $15 an hour.” It was interesting because these are economists. If you think about a price shock here — that now you have to pay your undergrads this much — if you have fixed budgets, you could imagine that this could be an issue, that people wouldn’t hire their RAs. She had this other tweet: “I provided research opportunities at my lab for undergrads as a way to expose them to the research process, to facilitate the pipeline into academia. There are other, more efficient, cheaper ways I could get the work done.”

That’s what we were looking at in this project: how much do those early lab experiences matter, and can something like the minimum wage also impact it? There are other policies that can play a role in this idea of who’s in the pipeline. Today it’s interesting because we think about NIH funding being cut, NSF funding — PIs are working with smaller budgets, and who they’re going to be able to train can then impact who’s in that pipeline going forward.

We found that writing a labor economics paper, and then a paper that’s interesting for the economics of science and innovation community — we could have the same results, but how we frame it mattered for who was interested. The labor economists are interested in this question, but there’s a whole minimum wage literature — trying to navigate that has been an interesting process.

We are using variation in minimum wage increases across states and time to look at whether that impacts the employment of undergrads in labs. Then we use that as an instrument for exposure of undergrads to science — their lab experience. We use that to link to career decisions of undergrads.

I also wanted to mention this paper because it uses UMETRICS data. It’s an exciting data set, but it hasn’t been used as much as it could be in our field, because you have to get approval to work on it. It provides very micro-level data — grant-level expenditures at major US universities. Bitsy Perlman has some papers using UMETRICS. They link this to other things like publications, and we also now have longer-run education data. We can observe students who worked in a lab as an undergrad, and then see what they do after they graduate from college. The trade-off with UMETRICS is that it doesn’t have names, so we can’t match it to anything else that you might want to match it to.

Our empirical approach uses the recent minimum wage literature — staggered difference-in-differences, using the timing of when you have minimum wage changes, variation in that timing but also in the magnitude, and then an IV for exposure to lab work.

What we find is that when PIs are exposed to increases in the minimum wage, they do reduce undergrad lab employment. They also increase their graduate student employment a bit, so it looks like there is some substitution to graduate students. We also see that they seem to be budget constrained. That’s a feature of why it’s interesting to look at minimum wages in a university lab setting: people are constrained. You have your grants, and they’re fixed. We do find that labs are more likely to seek supplemental funding when you have a minimum wage increase. But unlike with firms, you can’t pass the increased cost through to consumers — unless you think of the consumer as the NSF or NIH.

Going back to framing for our community, the economics of science and innovation: we show that reduced lab experience does affect careers like your Jennifer Doudnas. Students who are exposed to those minimum wage increases work 8.2% fewer quarters in labs. Then we show that those students are less likely to pursue a PhD or work in the life sciences — even just working in that sector. We see smaller effects for students with federal work-study support, which is a subsidy to pay the wage of the student.

Here you can see one visualization, the event study, where the undergrad is the orange line. The red line is the timing of the minimum wage.

You see a decline in employment. The green is the graduate students; you see a bit of an increase. It’s a bit noisy.

This is not super compelling, but it’s just to show you the relationship — blue is students who didn’t experience any change in the minimum wage by graduation year, then we have the quarters that they worked in the lab.

We do this in a regression framework too, but it’s suggestive that this is our first stage: if there was no change in the minimum wage when you were in undergrad, then you were more likely to work in a lab. Then this is with doing a PhD. This is just supposed to be suggestive, and then we do it in a regression.

Minimum wage increase, you work fewer quarters. Then [in the right-hand column] this is the IV, that you are more likely to go on to do a PhD.

This is Richard Freeman. Richard had this piece that he called “five rules for empirical work in labor economics.” He wrote this a while ago, but I think it’s pretty good, especially for those of you who don’t have a dissertation topic yet — or maybe you do, but you’re not sure which way to take it. Some of these rules are nice to reflect on if you need some ideas. A lot of them, when he was writing this — which was probably 30 years ago — weren’t as obvious, but it’s interesting how much is still relevant today.

1. “Create your own variation through an experiment (field or laboratory) or study markets when they experience sufficiently sharp exogenous shocks to create ‘natural experiments’ or learn the institutional details of markets to find plausible sources of variation (policy shifts and institutional quirks).” This goes back to this focus on causality. But in our field, if you can try to run an experiment, it’s challenging — but if you also have a chance to go and work for a government agency and learn about some of the details, you can come across some interesting data. These are also ways to find some variation.

2. “Focus on fundamental first-order economic and behavioral principles (supply and demand; incentives; altruism and reciprocity).” Economics of science and innovation is not one of the typical fields. To get into a general-interest journal — what would a macroeconomist care about if you’re doing a very micro study on the scientists? That’s something that I often get in my referee reports or editors’ letters: “Why should I care about PhD students in chemistry?” Going back to what the big economic principles are is important.

3. This is now standard, “Probe the robustness of empirical findings with different data sets, different specifications, and across time and space — tension vs. pre-analysis plans, pre-specification and concerns with data mining.” The more ways you can convince readers that you have strong results, the better.

4. This is the one that Richard always pushed, and in the profession people have done this: “Don’t be satisfied with just standard and easily available data sets – be willing to do your own survey research, use the resources of the web to collect data (eBay; on-line newspaper archives; on-line school or arrest record data); or to work with relevant organizations to collect/gain access to administrative data (IRS tax records, matched employer-employee data, Scandinavian matched registry data, Social Security admin data, personnel data, …).” A lot of our data is patent data, so if you can get at some of these questions with new data or survey research that you’re doing, I think it makes it more exciting.

5. People do this, but it’s still not the case that we talk about it often in our papers: the importance of qualitative research. “Discuss issues and analyses with the participants in the markets under study – ‘In a field lacking decisive tests of hypotheses, it is worth listening to what eyewitnesses and participants have to say’ (”quarks can’t speak” but humans can tell you what they think is going on!).” In a field lacking decisive tests of hypotheses, it’s worth listening to what eyewitnesses and participants have to say. If you read the original piece that Richard wrote, he talks about how we’re different from physicists, because we can talk to people. He talks about how Richard Feynman can’t go talk to what he’s studying — quarks can’t speak, but humans can tell you what they think is going on.
   

I have some friends who are scientists, and they always know when I’m testing out a new idea, like with that minimum wage study. I was like, “When the minimum wage increases, what does that do to your undergrads in your labs?” Just as a check, because when we were first presenting that, I remember people were like, “Why are PIs responding to the minimum wage?” I had talked to several people who were like, “We have fixed budgets; if the cost rises, you’ve got to let someone go.” The more that you can do that — if you have some friends who are scientists, that’s always helpful.

This piece has a simple suggestion: Rather than letting those ideas sit in digital file drawers, we offer them to philanthropists who might fund them. Perhaps instead of spending another year modifying a grant and throwing it back into an application portal, a scientist could be funded immediately by a foundation that finds their idea exciting.

I think this idea, sharing unfunded grant applications with other potential funders, has a real chance of being implemented. It’s easier than ever to do technologically and I haven’t heard a compelling reason why it shouldn’t happen.2

About the authors: Sandeep Patel and Amit Gupta are co-founders of Catalyze. Stuart Buck is Executive Director of the Good Science Project (where we are cross-posting this piece). And Ian Banks is Director of Science Policy at the Foundation for American Innovation.

Every year, federal agencies assess over a hundred thousand proposals that they never fund. Treating those peer review as public infrastructure could help unlock much more philanthropic capital for American research.

The National Institutes of Health (NIH) and the National Science Foundation (NSF) together run the most prolific scientific evaluation operations on the planet. Every year, they collectively review 110,000+ proposals, spanning everything from fundamental biology to quantum computing. While there is not wide agreement regarding how reviews should be conducted or how funding decisions should be made, no other process exists that applies rigorous, standardized review on such a comprehensive set of proposals.

In 2025, NIH reviewed 62,592 research grant proposals and only funded 8,161 (13%). That leaves thousands of proposals that were deemed to be in the top quartile, but are going unfunded due to a lack of money rather than a lack of merit. NSF exhibits the same pattern across a far broader set of fields (NIH only funds biomedicine, whereas NSF funds many other areas of basic science, including materials science, engineering, computer science, basic biology, and the social sciences).

The cost to produce reviews at this scale is substantial. Taxpayers support infrastructure at federal agencies, which totals roughly $263 million at NIH alone. Thousands of expert reviewers each volunteer on average 4-8 hours per application. And applicants spend an average of 116 hours writing each application. The time from when a scientist writes a proposal to when they receive grant funding can be up to two years.

Subscribe now

Whether a proposal gets funded is largely driven by peer review scores, but the detailed record (such as the actual scores and reviewer commentary) isn’t transparent at all. Peer review data disappears into an agency database that is almost never made available to anyone else. We think that these data should be far more transparent, so that they can help inform science funding organizations and the public.

Our proposal: make peer review scores and comments on unfunded proposals available to private funders through a shared platform. The scientist should be able to opt in and retain some control over what to share, but then the advantages are two-fold: (1) extra opportunities for the scientists to get funded (few researchers would object to that!), and (2) funders can more easily find vetted work that is matched to their interests.

What is needed, in effect, is a way for the public to reap the benefits of the millions of hours of due diligence that they have already paid for.

The moment is ripe. American philanthropy is arguably about to enter its third great wave, beginning now with Silicon Valley wealth (much from AI) and intergenerational wealth transfer, driven by donors who intend to give while they are living, who think in terms of measurable outcomes, and who are likely to be more comfortable with risk and with novel mechanisms.

Even when put against the NIH’s roughly $48 billion budget, private philanthropy is already quite significant: The Gates Foundation gave away over $8 billion in 2024, while the OpenAI Foundation is capitalized at more than $220 billion and has announced an initial $25 billion commitment to curing diseases and bioresilience.

Subscribe now

Many new donors want to fund science, including relatively new fields like biosecurity. Our federal scientific institutions (even with budgets growing at a typical pace, let alone with budgets under attack as they currently are) cannot possibly fund all of the meritorious proposals they receive, leaving great ideas unpursued. The third wave is arriving with its checkbook open and limited immediately-available options to write the check. Federal peer review could easily be reused as a signal of what private philanthropists should fund.

A skeptic might reasonably ask whether scientists themselves want their unfunded proposals to be circulated more broadly. Over the past several months we have spoken with more than 250 researchers, institutions, and companies across the United States and around the world, including interviewing dozens of principal investigators (PIs) directly about this question. We have heard near unanimous willingness, if not eagerness, to participate in such a system if it existed, assuming that scientists still maintained some level of visibility and control over what was shared and with whom.

Several small-scale examples of a marketplace for federal peer review already exist. The NIH has attempted a narrow pilot of a similar program that received researcher support. NIH’s Online Partnership to Accelerate Research effort gave highly rated but unfunded applicants a second channel to private funders, but ultimately it was too administratively complicated and its funder base too narrow for widespread adoption. The NSF’s Collaborative Research in Computational Neuroscience (CRCNS) program, still in operation today, offers other funders access to NSF reviews in computational neuroscience, including other government funders and philanthropy. As part of a very specific, highly interdisciplinary field, this program has backed research behind several modern AI advances, including brain-computer interfaces and spiking neural networks.

Another objection rests on whether the federal government can lawfully share this information. In the lightest-touch version, no disclosure is required. Agencies can email each applicant a private code, and the applicant then shares their own materials with a new potential funder. Neither the Privacy Act nor the Trade Secrets Act (18 U.S.C. § 1905) are implicated here. Indeed, the federal government already draws this line elsewhere: NIH and NSF publish the title, PI, abstract, and amount of every funded award.

We believe consent should be explicitly investigator-controlled. A PI would be able to opt in, and control access to abstract-level summaries, full proposals, and any panel reviews. A reporting layer would make a project’s existing support (from the government or philanthropists) visible, enabling a project to receive multiple gifts and for philanthropists to see where funding is most needed. None of this would require the government to spend significant capital or to endorse a particular funder. This proposal only requires federal agencies to make available evidence they already produce.

We can further reduce friction via two more expansive options. One option is to add an opt-in box to the NIH or NSF application, authorizing the agency to transmit the full proposal and reviews directly to the platform on the researcher’s behalf. Because that disclosure rests on the applicant’s written consent, it falls within the Privacy Act’s consent exception (§ 552a(b)) and would require no new system-of-records change. The only procedural step needed would be to modestly revise the agency’s approved information collection under the Paperwork Reduction Act.3

A second option, adopted by the NSF CRCNS program, is to disclose in the Request for Proposals itself that proposals and their reviews will be shared with outside funders.

Subscribe now

Two additional concerns should be addressed. The first is substitution: if philanthropy funds what the agencies are meant to support, does that hand Congress a rationale to cut public research budgets further? The best answer is that philanthropy and government largely have different views on prioritization, meaning there is natural complementarity to their co-existence. Rather than replacing public funding, philanthropy will most likely remain additive capital, reaching work that government agencies failed to fund.

The second concern is that simple metrics like percentile scores are noisy and are not reliable marks of quality, especially for a diverse set of funders with diverse motivations. This is correct, which is exactly why the summary of the agency assessment, along with detailed reviewer commentary and disagreement, should be included alongside any top-level percentile scores. In fact, we believe that sharing peer review information can uniquely enable more diverse prioritization of research by allowing funders to see with greater clarity how they fit into the larger funding ecosystem.

Our primary goal is to attract more capital into research. Through our proposed platform, philanthropy could complement federal funding and broaden the range of what gets funded. Where funders might normally duplicate each other’s evaluations in private, they could instead build on existing peer reviews and coordinate with each other through a shared platform. Besides bringing in more funding, treating peer review as public infrastructure would make the ecosystem more legible to everyone involved.

Shared infrastructure could enable the work of federal peer review to compress the timelines of other funders. Today, a proposal can wait the better part of a year for a funding decision, and many of the best and most ambitious ideas remain unfunded. In general, we should be building toward a world in which all the best ideas are funded, and the time from generating a very good idea to getting it funded goes to zero. This proposal could bring us one step closer to that distant goal.

This would not require the federal government to lower its standards in peer review, nor change what it chooses to fund. We should simply stop discarding the output of the largest and broadest scientific evaluation system ever built.

Subscribe now

In this lecture, Chad discusses how innovation and technology has shaped economic growth in the past, and how technology — in particular artificial intelligence — may influence growth in the future. I found the section toward the end of the lecture on the potential growth effects of AI and automation fascinating; one of the big insights from Chad’s modeling efforts is that the growth effects may lag considerably. So you might get profound abundance, but not until 50-100 years in the future. See below for a hint of what you’ll get in the lecture.

Subscribe now

Chad will have a chance to dig deeper into the economics of AI while influencing the technology’s future. In June, he announced that he was taking a leave of absence from Stanford University to conduct research on the economics of AI at Anthropic.

Here are the readings that go with this lecture:

• Bloom, Nicholas, Charles I Jones, John Van Reenen, and Michael Webb. “Are Ideas Getting Harder to Find?” American Economic Review 110, no. 4 (2020): 1104-1144.

• Jones, Charles I. “Growth and Ideas.” Handbook of Economic Growth 1B (2005): 1063-1111.

• Jones, Charles I. and Christopher Tonetti. “Past Automation and Future A.I.: How Weak Links Tame the Growth Explosion.” Stanford unpublished manuscript (2025).

• Jones, Charles I. “The Past and Future of Economic Growth: A Semi-Endogenous Perspective.” Annual Review of Economics 14, (2022): 125-152.

• Romer, Paul M. “Endogenous Technological Change.” Journal of Political Economy 98, no. 5 (1990): S71-S102.

Subscribe now

Thanks to William Higbie and Harry Fletcher-Wood for their support in producing this video and the transcript.

Subscribe now

Lecture Transcript:

The Past and Future of Economic Growth: A Semi-Endogenous Perspective

I’m going to present two sets of slides. First, an overview of how I see economic growth with some applications. Second, on AI and our economic future — a review of the recent work I’ve been doing.

The first thing I want to do is give you my view of how the economic growth process works. Here’s my favorite graph in economics.

This is a chart of GDP per person — average living standards in the US for the last 150 years. Once you plot it on a log scale, it’s a straight line with a slope of 2% per year. One of the big questions of the economic growth literature is, “How is sustained exponential growth at a stable rate for 150 years possible?” There are a bunch of growth models — a bunch of answers. But I want to give you my preferred answer.

I love simple models. Once you claim you understand something, it ought to be possible to strip out a bunch of ingredients that are unnecessary to show the core insight as simply as possible. You definitely don’t start there, there’s an arc — but it’s great to end up there.

• I want to do that.

• I want to talk about some growth accounting with this model.

• Then, I might skip through the future of growth: why it might speed up or slow down.

• That will take us naturally into the artificial intelligence material.

What’s the simplest possible model you can write down that gets at the core insight of why sustained exponential growth is possible? The key insight here is Paul Romer’s Nobel Prize for what I like to call “the infinite usability” of ideas. Romer pointed out that ideas are different from almost every other good in the economy. Most goods are objects. iPhones, airplane seats, surgeons, me, you, my computer, my office. Objects are rival. If I’m using it, you can’t use it at the same time. That rivalry is what gives rise to the fundamental scarcity at the heart of economics — economics is the study of how we allocate scarce resources.

Ideas are different. They are nonrival. I don’t like defining it as the negative of something else, so I prefer “infinitely usable.” Once you invent an idea, it’s technologically possible for it to be used by any number of people simultaneously without any depletion. You don’t run out of a given idea. Think about calculus, or HTML — the code that underlies web pages — or the chemical formula for a new drug. We invent the COVID vaccine and eight billion people can benefit simultaneously without any depletion. Contrast that with a barrel of oil or an apple: you or I can eat it. But we can all use calculus, computer code, or the latest ChatGPT. The code is nonrival.

The essence of Romer’s insight was, when you write down the production function — suppose A is the stock of ideas: the total number of ideas ever discovered. If we think of ideas as recipes in a cookbook, A is the number of recipes. Why, when you include ideas in a model, do you want the A outside the constant returns rather than inside?

This is a profound question. Your bias should be that everything goes inside the constant returns to scale. If we’re adding land, or splitting labor into skilled and unskilled, or splitting capital into equipment and structures, all that goes into the constant returns to scale. The default is that ideas should go in the constant returns to scale just like everything else, and you have to have a good reason for it not to. Griliches, when he was putting ideas in a model, put knowledge capital inside the constant returns to scale, exactly as he should have done if he didn’t have a good reason not to. Romer taught us the good reason not to, and why you want to pull the ideas outside the constant returns.

Let me talk about the constant returns to scale. Suppose we’re looking at Steve Jobs and Steve Wozniak in the garage in the ‘70s making the first personal computer. It’s the two of them, some screwdrivers, computer chips, and a keyboard. The reason you want to write constant returns to scale is because of the replication argument. If Jobs and Wozniak can make seven computers a week in their garage, one way to double production is to replicate it: create an identical garage, get two people identical to Jobs and Wozniak, and get the same computer parts. If we’ve replicated everything, we should produce the same amount. We’ve doubled our inputs and therefore doubled our output. That’s constant returns to scale. The replication argument is why I say the presumption should be constant returns to scale.

Yet once you think about ideas, that breaks. Why? Back to the Jobs and Wozniak example: there’s constant returns to scale after they’ve discovered the idea. The blueprints for building a computer are written down on a sheet of paper. The piece of paper is an object that needs to be replicated, but you don’t have to reinvent the design. You photocopy the design — you double the number of pieces of paper. But you don’t have to pay the years of work that went into getting the blueprint for the first computer. You have constant returns to scale to objects — that’s the replication argument — and therefore increasing returns to objects and ideas.

If you read the Romer 1990 paper, this core insight is in the first five pages, and he expresses it in some math just like this.

If A is the stock of ideas, you’ve got constant returns to K and L — they’re objects. Therefore, if the marginal product of knowledge is positive — if adding a new idea increases output — then there must be increasing returns to objects and ideas together. It is that simple. The replication argument and nonrivalry, the infinite usability of ideas, gives you constant returns to scale to objects and therefore increasing returns to objects and ideas. That’s the Nobel Prize. Everything else after that is dotting the i’s and crossing the t’s. That’s the essential insight in the way I view growth.

Now I want to write down a simple model to illustrate this core insight and how it delivers a theory of growth.

Notice I’ve dropped capital. Why? Solow taught us that capital is a bit of a sideshow — you don’t need it — so I’m going to take it out. In the first equation, there’s constant returns to labor — the only object — and A is the stock of ideas. With σ, there’s increasing returns to labor and ideas. The parameter σ measures the degree of increasing returns in the production of goods.

Then we also need a production function for ideas. Where does knowledge come from? This black box of where ideas come from is a big mystery. It turns out from the growth perspective you don’t have to solve that mystery. You can write down a standard production function for ideas and that works fine. There’s a fair amount of flexibility for how you write it down.

In the second equation, the number of new ideas — Ȧ, the flow of new ideas — depends on the number of people searching for ideas, R, and potentially on the stock of previous knowledge that’s been discovered, A.

Now I’ve got this φ exponent. What we’re saying is past knowledge could help or hurt you in producing ideas. There’s some flexibility there.

φ = 0 is the simplest possible model: one researcher discovers one new idea, and the model works great.

φ > 0 is standing on the shoulders of giants: past discoveries like calculus and information technology make me more productive at producing ideas.

But all graduate students are convinced that φ is negative. Why? φ < 0 is, “All the low-hanging fruit has been picked.” It says the more discoveries in the past, the harder it is to find new ideas today. It would be great to be an economist in the ‘50s — Solow, Samuelson, and Arrow had it easy. There was all this low-hanging fruit. Nobel prizes were hanging there. You just had to pick the apple. Graduate students today are climbing up the tree looking for the last remaining fruit, out on the branch precariously balanced, just to get the smallest of apples. It feels like ideas are getting harder to find in a strong way and that φ is negative.

It turns out the model works fine whether φ is positive, 0, or negative, which is a nice feature.

In recent work I found it helpful to rewrite this equation in terms of the growth rate of knowledge and define β to be 1 − φ. β captures the rate at which ideas are getting harder to find. Sometimes I like all my parameters to be positive. Both ways — the φ way and the β way — have lots of merits, but I go back and forth.

The next equation is the resource constraint. We’ve got L people in the economy. There’s population growth — I’ll come back to that. The main decision in this economy is how you take your number of people and divide it into production workers versus researchers. That’s the allocative decision. We could:

• Let markets decide.

• Write a utility function and let the social planner figure out the optimal allocation.

• Do a rule-of-thumb allocation — the one I’m going to follow here, and the easiest.

I’d encourage you, whenever you write down a model, just do rule-of-thumb allocations to start; you can do the optimal allocation or let markets allocate later. The thing that makes the model easiest to understand is some simple rule of thumb. Suppose we put a constant fraction s̄ — say s̄ is 10% — into research and 90% into producing goods. Then if you count equations and unknowns, the model’s closed and we can solve it. We can ask in particular: does this model generate sustained exponential growth at a constant rate, at least eventually?

[Student question.] Should I think about this rule of thumb as meaning skilled workers are unable to become unskilled workers?

In the simple model, there’s no such thing as skilled versus unskilled — there’s just a homogeneous labor. You could add skilled versus unskilled, and that would complicate the model, but the core result would still be the same. Like I’m forgetting about capital, I’m forgetting about human capital as well. In richer models — I’ll show you one at the end — I incorporate human capital. But for the simplest model you don’t need it, so I’ve left it out.

Remember when I said income per person grows at 2% per year — that’s about little y rather than big Y. In many growth models, we use lowercase letters to denote per capita variables. Little y is income or output per person. If you use my allocation, output per person depends on the stock of knowledge raised to some power times 1 − s̄.

[Student question.] I had a quick question on how to think about implementation costs within innovation. It seems a lot of implementation of new ideas are rival, and I’m struggling to think about something like calculus, which can be instantly transmissible to anyone who wants to use it, versus something like a new semiconductor innovation, which less developed countries wouldn’t be able to necessarily implement.

Even with calculus, it’s not like you show me a calculus book and I suddenly know how to do it. There’s a bunch of human capital — education — that has to be developed. It’s absolutely true that using ideas can require scarce resources. Imagine we’re making the COVID vaccine. It requires an incredibly sophisticated biotech factory and chemists to make it. Just because it’s technologically possible for many people to use the design simultaneously, doesn’t mean it’s not constrained by the scarce factors that are complementary to implementing it. In the simple model, those considerations are left out, but in the real world they’re definitely important.

[Student question.] Could you think about them being included in either β or σ here — just for conceptualization within the simple model?

I don’t think so. I think they’re different. Implementation is about rival stuff, and the φ in the production of ideas feels different to me. You’d have to complicate the model rather than use this one. But the first equation — income per person depends on the stock of ideas — Romer’s Nobel Prize is already there.

If we want the growth rate: on a balanced growth path, Ȧ/A is going to be constant. Whatever that constant is — along a balanced growth path, you can solve the idea production equation Ȧ = R·A^(−β) to get that A* (where the star denotes the balanced growth path) is some constant × R^(1/β). Where do ideas come from? From people. In the long run, the stock of knowledge is proportional to the number of people raised to some power, where the power is the inverse of the rate at which ideas are getting harder to find. A big β says ideas are getting hard to find quickly; a small β says they’re not getting as hard to find.

Now we put these two equations together, substitute the second into the first, and you get two useful results.

The first is that income per person is some constant × R^γ, where γ is my notation for σ/β. γ is a key measure of the degree of increasing returns to scale.

Notice what this equation says: income per person depends on the total number of people. Why? Income per person depends on the stock of ideas. The stock of ideas depends on the number of people. And we’re off and running.

Let me jump ahead quickly.

The equation I just derived says income per person depends on A to some power. The Solow model would say income per person depends on k to some power. Suppose I make both powers ⅓: α is ⅓ and σ is ⅓. The Solow model fails to deliver exponential growth, but the Romer model succeeds. Why? It has to do with the key difference between these two letters. A is the aggregate stock of ideas, and k would be capital per worker or per person. This is the nonrivalry of knowledge: each person benefits from every idea we create, whereas with machines you need a machine for every person.

Output per person depends on the number of ideas; the number of ideas depends on the number of people; so output per person depends on the number of people. If you take growth rates of this equation — logs and derivatives — the growth rate of output per person is the growth rate of the number of researchers × γ. So γn, where in the long run the number of researchers grows with population growth.

What this last equation says is the growth rate of income per person in the long run is a constant. This constant could be 2%: maybe n is 1% and γ is 2. I’ve given you a model that could give us constant exponential growth for 150 years. It has some interesting features — namely, population growth is important. If the population growth rate were 0, the exponential growth rate would be 0. We wouldn’t grow.

Why is population key? There are two ways to see it. The first is a narrow mathematical way. Romer’s key insight is that ideas are infinitely usable, and that gives rise to increasing returns to scale. What does increasing returns to scale mean? Bigger is more productive. Population growth makes us bigger, and the degree of increasing returns to scale turns the bigness of the population into per capita income. The growth rate is the product of the degree of increasing returns to scale, γ, × the rate at which scale is growing. The degree of increasing returns to scale is the ratio of how important ideas are in producing goods, σ, to the rate at which ideas are getting harder to find, β. γ is an overall measure of increasing returns to scale, including goods and ideas.

The other interpretation is that people produce ideas. We need the stock of ideas to grow. Getting aggregates to grow is easy; Solow taught us how. People produce ideas, so the more people, the more ideas. People produce computers, so the more people, the more computers. But it’s computers per worker that matters for productivity, whereas with ideas it’s the total stock. The COVID vaccine, calculus, or faster computer chips — you invent it once, eight billion people benefit.

You add one new idea and you can make an unlimited number of people more productive or better off, because the ideas are infinitely usable. Income per person depends on the total stock of knowledge, not ideas per person. It’s easy to make aggregates grow — population growth makes aggregates grow. That’s the essence of how Romer’s insight of nonrivalry leads to growth in a way that other models don’t.

Let me turn to this paper with Nick Bloom, John Van Reenen, and Mike Webb: Are ideas getting harder to find? This paper provides evidence for the view of growth I laid out. It documents a new stylized fact: there’s a robust sense in which ideas are getting harder to find.

There’s a word equation that many growth models have that says the growth rate equals the number of people looking for ideas times their research productivity times the rate at which they turn ideas into growth.

In this paper, we say, “Economic growth and the number of researchers are pretty easy to measure. So we can do growth accounting with the idea production function.” We measure growth and the number of researchers, and we can infer research productivity — the total factor productivity (TFP) of the idea production function — as a residual.

When we do that, everywhere we look — agriculture, Moore’s law, health innovations — economic growth is pretty stable or maybe even declining slightly. But the number of researchers is rising incredibly quickly. This says that research productivity — the rate at which we turn people into economic growth — is declining very robustly. That’s a sense in which ideas are getting harder to find. The way I wrote down the growth model is exactly this: growth is on the left-hand side times the number of researchers, and the TFP of the idea production function is A^(−β). For β > 0, ideas are getting harder to find. The evidence we document in lots of different places very much looks like the model I wrote down.

Let me show you some of that evidence. First, for the aggregate economy, what does TFP growth — the blue line — look like?

It was high in the ‘40s, ‘50s, and ‘60s, and it’s been declining ever since.

What about the number of researchers? That’s on the right-hand axis. If research productivity were constant, growth rates would be proportional to the number of researchers — the original Romer, Aghion-Howitt, and Grossman-Helpman models said that. They assumed research productivity was constant. What we’re showing is that research productivity is falling. That’s why I’ve made the scales similar: the model with constant research productivity says these two lines should lie on top of each other. Instead, growth is flat or declining, and the number of researchers has grown by a factor of 23 in the United States since 1930. If research productivity were constant, growth rates would have grown by a factor of 23, but they haven’t. Research productivity at the aggregate is declining — by more than a factor of 23, because the growth rate is declining.

My favorite example is Moore’s law. Moore’s law is a straight line on a log scale, very much like US income per person.

It’s been behind most of the rapid growth we’ve had in the last 25 years. It says the number of transistors we fit on a computer chip doubles every two years. For the last 50 years that’s been true.

Where does Moore’s law come from? Notice: constant slope on a log scale — constant exponential growth, doubling every two years. The growth rate is 70/2, or 35% per year. Moore’s law is not a law of nature — it doesn’t happen regardless of what we do. It’s viewed as a target by the industry, and we’re throwing more resources at these problems to stay on the line. If you measure the number of researchers devoted to pushing Moore’s law forward — Intel, AMD, Samsung, TSMC, and ASML, but also IBM, AT&T, and Motorola historically, or National Semiconductor and Fairchild Semiconductor in the ‘70s — it’s growing incredibly rapidly.

It takes about 18 times the number of researchers today to generate the doubling of chip density as it took in the ‘70s.

Again, growth is constant, the number of researchers is rising like crazy — so research productivity is falling and ideas are getting harder to find. We have to run faster — throwing ever more resources at these problems — to maintain constant growth. That’s what the growth model I started with says. We look at agriculture, medical innovations, Compustat data, the aggregate economy — you see this fact everywhere. We haven’t found a place where you don’t. You have to double researchers every decade to maintain constant growth. If we ever stopped doing that, growth would presumably slow down. That’s what the model says.

Let me do some growth accounting — this gets at the earlier question. I’ll give a broad overview. I can add capital, human capital, and misallocation, M, and do growth accounting with a model that includes the idea production function.

I’m using the balanced growth path version, where A = R^γ and R = sL — s is the fraction of people engaged in research, and L is the total population. When you write down this richer model, in the long run all growth is proportional to population growth, so it’s γn again, even with all these other ingredients. But along a transition path you can grow faster than γn.

Historically, how does growth accounting look in this richer model?

I’m writing out the terms. There is:

• A capital-output ratio,

• Educational attainment,

• The employment-population ratio — what fraction of people are working; and then,

• TFP growth.

TFP growth in turn has a misallocation and an idea term. The idea term has the research intensity and population growth.

In the long run, all of these terms, other than the last one, are zero:

• Educational attainment — what fraction of your time you spend in school — is bounded by 100%.

• What fraction of your population you put in research — the s variable — is bounded at 100%. It has to stop growing eventually.

But historically, s has been rising, and educational attainment has been rising. Historically, not all growth was due to population growth.

Let’s do the accounting. What you get is summarized in these pie charts:

• The capital-output ratio is pretty constant.

• Human capital per person rises at about a year per decade in the 20th century. Each year [of additional education], labor economists tell us, raises wages by 6%. It’s actually risen by less than a year per decade, so each decade we’re adding 5% to wages. Wages are output per person. If you divide by 10 you get 0.5% per year. Rising educational attainment accounted for a quarter of our 2% growth in much of the 20th century. But it’s already leveling out.

• Similarly, rising female labor force participation, the employment-population ratio: 2 percentage points. But the employment-population ratio has to level out at some point — it’s already leveled out.

These historical sources of growth are going away.

TFP growth is the residual, 1.3.

That’s due to research intensity, declining misallocation, and population growth. Rising research intensity is half of TFP growth, but it can’t rise forever. Declining misallocation — I make up this number from a paper on the allocation of talent with Chang-Tai Hsieh, Erik Hurst, and Pete Klenow. But if you get rid of all misallocation, you can’t get anything else. The population growth component, since the ‘50s — my estimate is it’s only 15% of growth, 0.3 percentage points.

What that means is when all these other sources come to an end — human capital is already slowing down, employment-population ratio has already come to an end, research intensity is still rising, misallocation is hopefully improving — population growth is the only source. There are lots of forces pushing down growth in the long run. Maybe our long-run growth is much slower than our historical growth.

The future of growth might be slower for various reasons — population growth is also slowing. It might be faster for two reasons. The rise of China and India. How many talented people in the ‘80s — because China and India were poor and far from the frontier — couldn’t push the frontier forward? Now that China and India are getting richer, that’s 2.5 billion people we can start throwing at these hard problems. AI is another: if machines can produce ideas, we don’t need people to produce ideas — maybe good things can happen.

Let me stop on that. These are great questions for future research.

• How large is the degree of increasing returns to scale? I was assuming it’s a third, and I have some reasons for that, but I wish we had better estimates.

• What is the social rate of return to research?

• Better growth accounting to look at the long and variable lags through which increases in research translate into productivity growth in the future.

• Automation has been ongoing for 150 years, but growth is slowing not rising: why?

AI and our economic future

I’ve written several papers on AI and I’m spending lots of time on this — it’s the most important thing happening right now. I have a paper called AI and Our Economic Future that summarizes the work I’ve done. These slides are loosely based on that.

The first paper I wrote was with Philippe Aghion and Ben Jones, called AI and Economic Growth. We identified two themes.

First, AI is the latest form of an automation process that’s been going on for centuries: the textile revolution, the steam engine revolution, electric power. Maybe we can look at history to learn something about what AI might do in the future.

The second important point is that AI may be limited by bottlenecks, weak links, or Baumol’s cost disease. In the early paper we talked about Baumol’s cost disease and bottlenecks; in my new paper I’m talking more about weak links, but we mean the same thing by all those phrases. Economic growth is constrained not by what we do well, but by the things that are both essential and hard to improve. It’s very much like Michael Kremer’s O-ring paper. The space shuttle Challenger explodes in 1986 because one $20 part fails. In a model with 5,000 parts, if one of them fails it can be disastrous, because of the weak-link problem — your production has an elasticity of substitution less than one. Everything has to go right or bad things can happen. Production functions feel that way. The computer chip manufacturers we talked about are like that. Weak links are the source of scarcity, and therefore earn high returns.

• There are a ton of papers by Acemoglu and Restrepo — every year they write another top-five paper on automation and growth. Half of what I know about automation and AI I learned from their work.

• There’s a foundational paper by Zeira.

• A nice paper by Hemous and Olsen that was around very early on.

• Ben Jones has done some work I’ll touch on.

Automation and Weak Links

First, let me show you the canonical model — the right combination of simple and insightful.

You’ve got a CES production function with elasticity of substitution less than one, σ < 1. That’s what we mean by weak links. Every input is essential: if any input is zero, output is zero, and if any input is infinite, output is finite. σ < 1 gives you that. The chain is only as strong as its weakest link. You don’t have to go to Leontief — σ < 1 is enough. Output is a combination of a continuum of tasks, yᵢₜ.

Each task can be produced with machines or with people as perfect substitutes. People have productivity ψ_L and machines have productivity ψ_K. In the paper with Chris we put the i’s there as well (ψ_Li, ψ_Ki) — different machines can be good or bad at different tasks. If the task has been automated — from 0 to β, where β is the fraction of tasks we’ve automated — then you produce with capital. If it hasn’t been automated, you’re not allowed to produce with capital; you haven’t figured it out yet, so you have to produce with labor and you get ψ_L⋅L. This implicitly assumes machines are sufficiently productive and cheap that you’d rather use machines — that’s an easy assumption to justify.

Capital accumulates.

• You’ve got a fixed number of machines, K, that changes over time, but at a point in time it’s a stock you have to allocate across tasks.

• You’ve got a given amount of labor to allocate across tasks. Y = C + I.

• Investment, like Solow, is a constant fraction of output, so I don’t have to talk about prices if I don’t want to.

• The model’s symmetric, so you’ll naturally put the same amount of capital and labor on each task, and the allocation is straightforward.

An important feature of this model is that it’s got both complementarity — the weak link, σ < 1 — and substitution in the production of tasks, with perfect substitutes. What gives rise to the richness of the predictions of these task models is this interplay between complementarity and substitution. If you allocate capital symmetrically across tasks, so from 0 to β you put K/β units of capital, and from β to 1 you put L/(1−β) units of labor on each task.

This production function — K/β units of capital, L/(1−β), on a β fraction of tasks.

There’s a green β and a purple β. The purple β dominates when σ < 1. There’s this dilution effect: as you automate more tasks, your given stock of capital gets spread more thinly, and your given stock of labor gets concentrated. Labor per task goes up, capital per task goes down.

You can collect these βs and write this as a reduced-form CES production function.

Output is a CES combination of BK and AL, where B = (ψ_K/β)^(1/(1−σ)) and A = (ψ_L/(1−β))^(1/(1−σ)). An increase in β causes B to go down and A to go up. You might have thought automation is capital-augmenting or labor-augmenting. It turns out it’s neither. Automation — an increase in β — is simultaneously capital-depleting and labor-augmenting. Capital is spread more thinly, so your weak links are weaker. Labor is concentrated, so your weak links are strengthened. That’s why B goes down and A goes up.

How does automation work?

A simple way to think about it: you automate a constant fraction of the tasks you’ve not yet automated every period. 1−β is the set of tasks that haven’t been automated. Suppose you automate 2% of those every year, so β̇ = 2%(1−β). x̄ is the automation rate — what fraction of tasks you automate every period. If you look at this law of motion, it says β → 1. Eventually you automate an arbitrarily large fraction of the tasks. It only goes to 1 as t → ∞.

What happens to 1−β? It falls to zero, but at a constant exponential rate if the automation rate is constant. The growth rate of 1−β is -x̄. The set of tasks doing labor is falling exponentially at the automation rate.

Put these things together.

Let me turn off the ψ_K and ψ_L — pretend those are constant; I’ll relax that in a second. Then B = 1/β and A = 1/(1−β). B is going to be a constant eventually, and A is going to grow at a constant exponential rate, because 1−β is falling at a constant exponential rate, x̄, so 1/(1−β) rises at a constant exponential rate. The growth rate of A is constant.

This is a model where you’ve got As and Bs moving around, yet in the long run all technological change from automation is labor-augmenting. The B becomes a constant and you get Uzawa’s theorem satisfied. You can get a balanced growth path, and the capital share, importantly, is between 0 and 1. The capital share can be a third here, even though β → 1. This is the weak-link phenomenon: you’re automating all these tasks, but you’re constrained by the stuff you’re bad at. Labor is the stuff you’re bad at, so it gets a high return. With elasticity of substitution less than one, the scarce factor gets the share. Labor is always the scarce factor. Capital’s plentiful — you’re adding machines left and right, but you’re not adding labor, so labor keeps its share because it’s so scarce. This is a way of saying: don’t be so sure to assume that automation has to raise the capital share.

The Jones and Liu paper takes the framework I just laid out.

Aghion, Jones, and Jones didn’t have the ψ_K and the ψ_L; Ben [Jones] added the ψ_K, capital-augmenting technological change, which he called Z — faster computers. In that case the capital share is proportional to the ratio β/Z. Automation, increasing β, raises the capital share, but faster computers lower the capital share — again because of the weak-link phenomenon. You’ve got infinite machines, which means labor’s the scarce factor, so labor gets the share. Ben showed how you can have a balanced growth path that’s not asymptotic and a constant capital share.

An interesting thing about it: he had to rig it so that it worked, because β → 1. If Z kept growing — if our computers kept getting faster — and β → 1, then the capital share would go to zero. They had an interesting argument for why that might not be the case, but it’s an assumption. With σ < 1, the fact that computers are getting cheaper might drive its factor share to zero.

This led to an interesting question, this is in my new paper with Chris: what’s happened to the factor income share of computers? Just like the labor share, you could break it into skilled versus unskilled labor — what’s happened to the college-educated share versus the non-college share. Or you could break capital into the structures share versus the equipment share. You can look up, in the Bureau of Labor Statistics multifactor productivity tables, what’s happened to the factor income share paid to computers.

If computers are getting better, and getting better quickly, and σ < 1, it could go down. But it’s tricky, because computers are everywhere — you use them much more broadly, and that tends to raise the computer share. That’s the automation effect, the β. But there’s also the Z. It’s a race between these two things. Which way has it gone?

Here’s the data.

In the late ‘90s, the dot-com boom, the factor income share paid to computers went from 2.5 to over 3. But since 2000, over the last 25 years, the factor income share paid to computers has fallen by a third, from 3 to 2. Computers are everywhere, but the rapid price declines dominate. That looks a lot like evidence for σ < 1. If σ > 1, the computer factor income share would be going up, but because σ < 1, the price effect dominates. When we’re thinking about this AI future where computers are doing everything, it’s not obvious that the capital share goes to 1. It could, but be careful.

AI and Our Economic Future

There’s a good chance that AI is the most transformative technology in your and my lifetime. We’ve had lots of transformative technologies before — electricity, semiconductors, the internet — but I think AI could be more. The argument is: what if, in the next 50 years, machines — AI for cognitive work, and AI running robots for physical work — can perform every task a human can do, but more cheaply? What does it look like to live in that world? What does the transition path to that world look like?

Let me lay out two scenarios that are both extremes. First, AI dramatically accelerates growth. The other extreme: AI is just business as usual. The truth is somewhere in between, but it’s helpful to look at these two extremes.

The first scenario, where AI dramatically accelerates growth — this is what people in Silicon Valley talk about a lot.

The first stage is AI automating software, and we’re already seeing evidence that AI is very impressive at that. The AI people for the last 10 years have been saying this is coming, and we’re right on path for what they were saying. They’re getting some credibility for these predictions coming true.

When Anthropic hires a software engineer, they give them a two-hour take-home exam, and it’s a hard exam. They gave the same exam to Claude Opus 4.5, the model before their latest one, and it performed better than any human in history. If you look at the horizon over which software models can perform tasks: two years ago it was four minutes, and now ChatGPT 5.3 Pro is up to six-plus hours — with 50% accuracy it can work for 6.5 hours to perform a task. Is it plausible that in the next decade, and maybe sooner, we’ll have AI agents that can automate most coding? Anthropic says that’s by the end of the year for them. It seems very plausible.

Once you have that, you ask your AI to do more experiments, design better algorithms, test them out, figure out what works, design better agents, get AI agents that can run a computer. Claude Code can basically run your computer now — not as effectively as it should, but it’s getting better at an astounding rate. Is it plausible that in the next decade your AI agent can do anything you can do on a computer? That’s not outside the realm of possibility.

Once we have that, we have these virtual remote workers. You can scale them up — they’re nonrival, infinitely usable. You run billions of these virtual research assistants, each running 100 times faster than us. This is what Dario Amodei called “a country of geniuses in a data center.” Once you have a country of geniuses in a data center, you ask them to:

• Produce new ideas.

• Invent better computer chips and robots. Build a virtual world to figure out the right way to build a robot.

• Invent better medical technologies. Cure heart disease and cancer. AlphaFold on steroids.

Once you’ve designed better robots, we let the AI run the robots in the real world, and that automates physical tasks as well.

I don’t know that this has to happen, but in the next 25 or 50 years, if not sooner, each of these steps seems possible. For the last 10 years, people have been claiming we’d get to the end of the first set of bullet points, and we’re getting awfully close to that now, so maybe they’re right about the rest of the steps. If all this were to happen, it would raise growth rates substantially. I’ll show you some simulations in a second. That’s the dramatically accelerating growth scenario. I think this has a lot of merit. I might not 100% buy in, but is there some substantial probability of it? Yes.

What’s the other scenario? I’d also put substantial probability on business as usual.

Automation has been going on for 150 years and growth hasn’t sped up. Remember the graph I showed you at the beginning.

In 1870, electricity barely existed — we were just starting to apply it to lighting and then to factories. The electricity revolution was the first half of the 20th century. Then we invent:

• Transistors,

• Vacuum tubes,

• Semiconductors,

• Information technology,

• Antibiotics,

• The internal combustion engine;

all these monumental general-purpose technologies, and yet straight-line 2% growth.

What’s going on? What might be going on is ideas are getting harder to find. Within any technology class, there are sharply diminishing returns. It’s like a gold mine. The steam engine runs out of steam. If you don’t invent the next general-purpose technology — if you don’t discover the next gold mine — growth would slow down. We don’t see the counterfactual. The counterfactual is, if you don’t get anything after the steam engine, growth slows. But electricity let us have 2% growth for another 50 years. Each of those great ideas — we kept discovering new gold mines, and that’s what kept growth from slowing. Maybe AI is the latest gold mine, the latest great idea that gives us another 50 years of 2% growth.

That’s the most pessimistic view. I probably think it’s more than that, but that view has the advantage that we need to understand this graph. Automation’s been going on and growth hasn’t sped up, so you need to fit those facts.

The second lesson from economic history: you all know Paul David’s article about the steam engine and electricity — the dynamo, the gradual adoption. Or Erik Brynjolfsson’s work on information technology. Solow: we see computers everywhere but in the productivity statistics. All these things do transform the economy, but that transformation requires complementary innovations, diffusion, applications, and organizational changes. The transformation happens over 50 years rather than overnight. There’s a good bet AI is going to have that same feature — that it takes longer than some of the more optimistic people think.

Let me show you some simulations.

Chris Tonetti and I have a new paper, where we take the Aghion, Jones, and Jones view of the world, take the data very seriously, and use historical evidence to calibrate a growth model and run it forward. The key finding from the historical evidence is, what is automation? Automation is replacing slowly improving humans with rapidly improving machines on an ever-expanding set of tasks. We estimate machines are getting better about 5% per year faster than humans are getting better, at least. Substituting more tasks to rapidly improving machines is the quantitative source of growth in automation.

Then we build a model and calibrate it to the evidence from the first half of the paper. It’s an idea-based growth model: you invent new ideas and that helps you automate. When you automate that helps you invent new ideas. Both the goods production function and the idea production function get automated endogenously. The model has these two key ingredients from the two scenarios. On the one hand, there’s the positive feedback loop — automation gives you new ideas, which gives you more automation, which gives you more ideas. There’s a flywheel effect. But there are also weak links, which limit the effect of automation. Imagine you’ve got a chain with 20 links, and I make 10 of the links infinitely strong. That’s good, but your chain is still limited by the weak links. Those weak-link effects can be very important.

We simulate the future with three scenarios.

• In one, all tasks can be done by humans in finite time — there’s no task a machine can’t do someday. If it’s full automation, you eventually get an AK model where the A is growing because machines keep getting better, so the capital share goes to 100%.

• There’s an incomplete automation scenario where we say 3% of tasks can never be done by machines. Then 97% of tasks, you get infinitely strong because your machines keep getting better, but it’s a weak-link model — you’re constrained by those 3% of tasks and the rate at which humans get better. The labor share goes to 100% and the capital share goes to zero, just like the computer graph.

• Then there’s a case in between where the capital share remains constant.

What do we find?

Running the model forward starting in 2020:

• In the full automation case growth explodes.

• In the incomplete automation case, in the long run the growth rate is the rate at which humans get better, which we estimate to be very slow, 0.5% per year.

• Interestingly, in the baseline case, where the capital share remains constant, growth also explodes. That surprised me. You’ve got this weak-link model, capital share constant — I thought maybe growth is constant, but no. You’ve got the flywheel effect automating stuff, and the weak-link effect, but eventually you automate all the weak links, because the baseline case says you can automate everything eventually, though it takes infinite time. That’s enough to give you explosive growth.

Let me show you the levels.

You can see growth exploding in the purple line. The interesting thing about the levels is that even though growth rates for the next 75 years are rising up to 5% per year, all three cases that look very different in the long run look very similar for the next 75 years. And, for the next 75 years, it’s remarkably small. Relative to the constant growth trend, we’re only 4% richer 20 years from now, and even 40 years from now we’re only 19% richer.

Growth accelerates, but it accelerates incredibly slowly. Growth explodes, but the explosion is slow. This is exactly the weak links: you get really good at some things, but you’re always constrained by your weak links, and that slows down the explosion. The explosion’s coming — you see it in all three cases. In the long run, in the incomplete case, it goes back, but growth accelerates incredibly slowly. In the next 50 years, we’re not learning too much about which of these three cases we’re in.

Growth explodes, but surprisingly slowly. We may have more time than I thought to handle the problems. I meant to have time to talk about the problems — jobs, what we’re going to do, inequality and meaningful work, and existential risk.

Existential risk

I’ve written two papers on existential risk, and I think these are problems we really do want to take seriously. Let me do one slide on existential risk, because people don’t talk about this enough.

There are two versions. The first, the bad-actor model, I think people are amenable to. Imagine ChatGPT 10, improving exponentially, incredibly good — this oracle that can answer any question. If we haven’t figured out how to keep it from answering any question, then a bad actor could ask it to design a virus more lethal than Ebola, that takes three months to display symptoms, and kills an enormous number of people. Nuclear weapons were manageable because they were rare — only two people had the red button, and we’ve managed so far not to press it. If eight billion people had access to a red button, can you make sure no one presses it? That seems like a problem. That case is easy to say yes, we should worry about.

The alien intelligence case is a bit more science fiction, but totally something we should worry about. Suppose we found out this afternoon there’s a spaceship on its way to Earth, passing Pluto. How would we feel? We’d be excited. We’d learn a lot. Yet on Earth, when advanced people or species encounter less advanced, it often doesn’t end well for the less advanced. Think about the Old World and New World, or horses and people. Maybe this would be dangerous as well. How sure are you this alien intelligence is going to be fine? We’re growing alien intelligence in these AI models right now, and we don’t understand them. Stuart Russell, a CS professor at Berkeley and co-author of one of the top machine learning textbooks for graduate students — I was on a panel with him and he asked this question: “How do we retain power over entities more powerful than us, forever?” That last word is the kicker.

To conclude.

I like to think of AI in terms of the internet. How much did the internet change the world in the 30 years from 1990 to 2020? It’s hard to see in the statistics — again, straight line — but the answer is a lot. You don’t know the counterfactual. How much will AI change things between 2015 and 2045, more or less? How many internets — two, five, ten? I think the answer is a lot. Just because the changes take 30 or 50 years instead of five years doesn’t mean the changes won’t be profound. Because of the weak links we have more time than I had thought. But we should absolutely use those intervening years to prepare for the changes that are coming. The political economy, the inequality, the jobs, and the catastrophic risk are all things we should be worried about.

How do you make sure you get your share of the pie? Historically, labor is the main asset many people have, that they trade for consumption. In a world where AI automates many tasks and therefore many jobs, and can do everything a human can do, whether cognitive or physical — maybe your labor asset isn’t worth very much, because it falls to the price of the machine. Maybe the pie is huge — how do you make sure you get your share? Something like give every newborn child a share of the S&P 500. That should work, but let’s acknowledge there’s a big political economy problem there. We need to think hard about that, and the transition could be very hard.

The costs are very important to consider. A sad thing is that in the world we live in right now, AI is exploding, and the thinking about catastrophic risk, or labor market effects, or how we redistribute, is behind the ball. We definitely need more good thinking there.

All these slides and papers are on my web page if you’re interested in seeing more.

We were happy to see this inclusion, as their description of metascience units is similar to what we proposed in a response to an OSTP Request for Information in 2025.

As is often the case, though, the devil is in the details (and the implementation). Metascience units will only pay off if set up intelligently, pointed at an agency’s most important questions, and given the power to generate decision-relevant evidence.

So we wrote a piece for the IFP website explaining our vision for metascience units: why we should create them, what they should do, and how they should be organized. We have never cross-posted with the main IFP website before, but we think this article is particularly important given interest from the White House and beyond.

Summary

The US government is the largest funder of basic research in the US and the second largest funder of applied research after industry. Despite this predominant role, federal science agencies have limited information on how well that funding is being spent and how to spend it better.

Ideas about how to improve science funding abound, from shortening application timelines with fast grants to uplifting reviewers with AI tools. But most ideas lack strong evidence of efficacy, especially at scale or within the constraints of government. To find out how these ideas perform in practice, agencies need to be able to run experiments within their own walls, applying the scientific method to science itself.

Science agencies can accomplish this by creating metascience units: nimble teams of government staff and academic experts housed within agency leadership offices. Using the tools of metascience, these units would run institutional experiments and analyze the treasure troves of internal agency data to provide evidence about how to most effectively advance breakthroughs.

These experiments and analyses would allow agency leaders to identify and quickly scale up improvements to program design. Metascience units could lead to agencies funding higher quality projects, cutting review timelines, and closing gaps in the scientific ecosystem.

Subscribe now

Motivation

The federal government spends nearly $200 billion a year on research and development. If we could make the process of identifying, distributing, and managing those funds even 5% more effective, we would effectively unlock another $10 billion a year for innovations that benefit Americans. For context, that is slightly more than what we spend annually on NASA’s entire science budget or on the National Cancer Institute.

The federal scientific enterprise is vast and decentralized. This allows agencies to develop a diversity of approaches to funding and conducting science, from small grants and innovation prizes to ARPAs and National Labs. But knowing which approach will work best to further an agency’s mission is not a solved problem. There is much we do not know about where scientific progress comes from.

When agencies make decisions about how they structure, fund and conduct science, they typically lack high-quality evidence. What federal agencies do know about the efficacy and efficiency of their science investments comes from ad hoc external program evaluations, idiosyncratic internal analyses, or academic studies with limited, largely historical data. Often, these products are neither timely nor focused enough to influence program decisions. At the same time, they don’t make sufficient use of the agencies’ internal data, even though that data could provide some of the clearest signals about program strengths and weaknesses.

Inefficiencies in federally funded science are well-documented. These include slowness in funding grantees, an overwhelming volume of paperwork for academics, limited funding for early career scientists who might have the most disruptive ideas, risk aversion in peer review, and difficulty in coordinating within and across agencies.

We know the problems, but we need evidence on how to solve them. The best way to improve science investments would be for agencies themselves to conduct experiments, analyze their treasure troves of data, and scale approaches backed by their results. This will not happen organically. Existing agency staff do not have the mandate or bandwidth to do such research — especially with considerable staffing shortages in science agencies — and external researchers and consultants lack internal data and institutional knowledge. We need dedicated, empowered spaces within agencies to do this kind of work.

Subscribe now

Solution

Establish metascience units within federal science agencies

To generate the decision-relevant evidence and analysis they lack, science agencies should establish metascience units: proactive, nimble teams akin to internal think tanks. Units would analyze existing programs, conduct internal research, and develop better ways to fund and conduct science. Above all, the units should be anchored around answering the question: how should the agency invest its resources — whether budget, staff time, or the time of the reviewer community — to most effectively advance breakthroughs and further its mission?

While new to the US, the idea has found footing in the UK, which launched its own Metascience Unit in 2024 to conduct experiments and share insights across the country’s science ecosystem. Since then, the UK Unit has seen early successes, including a pilot that sped up grant application decisions by three months.

Since we advocated for the establishment of metascience units last year, the idea has also picked up steam across the federal government. The National Science Foundation (NSF)’s Fiscal Year 2027 Budget Request called for a metascience unit, and the White House Office of Science and Technology Policy (OSTP) Report Science: A New Golden Age made the same recommendation across science agencies more broadly. The National Institutes of Health (NIH) is in the process of launching the Office of Research Economics, Planning, and Analysis (OREPA), which will conduct economic research and facilitate replications.

Metascience units should generate evidence about improvements to program design

There are numerous possible research questions about how to structure, fund, and conduct science. But metascience units should start with questions they are best suited to answer with the levers they uniquely have: administrative data and prospective experimentation. This suggests the units should focus on post-hoc analyses, pilots, and controlled experimentation to improve funding mechanisms and award evaluation processes.

Problems should also be relevant to program design. They should therefore be sourced from divisions and programs that would actually implement them, and agency leadership can prioritize metascience experiments that provide the most decision-relevant information.

Improvements that a metascience unit could research include:1

• Fast grants, to move from application to funding in weeks rather than months (or years). The NSF effectively implemented fast grants during COVID-19, and other research funders have honed similar models. Metascience units should test how fast grants perform relative to conventional grants, using tools like cost-benefit analysis. They can gather evidence on when agencies should be using fast grants versus generally speeding up the conventional grant process.

• Multi-stage review, to address the challenge of applicants and reviewers spending substantial amounts of time on applications that will not be funded. Multi-stage review requires applicants to submit a brief concept note to start, and then only invited applicants submit full applications. This approach is already used in some agencies, but a 2025 National Academies of Sciences, Engineering, and Medicine report proposed expanding it government-wide. While this is a promising direction, metascience units should conduct experiments that compare multi-stage review to conventional processes to see how it impacts both efficiency and the quality of research.

• Golden Tickets, to address the problem of conventional review panels often rewarding consensus and punishing novel ideas. Golden tickets allow individual panelists to champion applications for funding even if other reviewers rated them poorly, potentially encouraging high-risk, high-reward projects. Metascience units should conduct experiments to compare golden tickets to more conventional panel review approaches. They can also use historical data to simulate what applications might have won if a panelist who was alone in giving a high score had gotten their way. A related approach would be testing the effectiveness of instructing reviewers to consider the benefits of higher-risk projects (as NIH has instructed panels to do) or encouraging reviewers to reward novelty.

• Person-based grants, to learn whether betting on researchers’ longer-term agendas rather than specific proposals yields better results. Science agencies already fund some person-based grants at research institutions, such as NIH’s R35 Investigator-Focused Awards. Evidence suggests that person-based grants can free scientists to conduct high-impact research rather than focusing on grant writing and reporting. Person-based grants should be compared against standard project-based grants.

• Longer grant durations or higher award values, to test whether giving researchers more time or money leads to more ambitious and creative work. New program types, like NSF’s recently-launched X-Labs program, can combine flexibility with award sizes that encourage scientists to take on ambitious projects that require sophisticated infrastructure and multidisciplinary teams.2 We think this will enable more high-impact research. Before expanding these pilots, metascience units should carefully analyze them to understand what’s working.

• New AI-enabled tools, to identify the most appropriate ways to leverage the technology for funding decisions. Metascience units are well-suited to measure whether AI input could improve reviewer accuracy through controlled “uplift” studies. Uplift studies are used to compare how people perform tasks with and without access to an AI tool, and they can help metascience units assess whether the tool provides meaningful assistance to agency staff. Promising applications include AI-assisted proposal scoring, automated extraction of proposal characteristics for portfolio analysis, and reviewer matching based on AI network analysis. UK Research and Innovation, where the UK Metascience Unit is housed, has already begun studying AI as a supplemental tool in grant proposal review. This would be best approached as a test environment where metascience units iterate on AI applications faster than an agency’s usual processes allow, building on the agency’s past data and grounding the applications in meaningful correlates of scientific impact.

These pilots and controlled experiments test alternatives directly. Alongside those activities, post-hoc analysis of how existing programs have actually performed are also useful. These efforts can reinforce each other. Retrospective analysis of existing data may be one of the cheapest ways to identify which questions are worth the cost of an experiment.

Subscribe now

Metascience units should make experimentation and internal data analysis easy

Metascience units can overcome the current difficulty of experimenting with new approaches and analyzing internal data. They can do this at a high level by making agency-level decisions about questions to tackle, cultivating talent in-house, shortening feedback loops between generating evidence and making decisions, and tapping into the institutional knowledge of agency staff in a way that external researchers or consultants could not.

But more importantly, metascience units should build the infrastructure for experimentation and internal data analysis as a core part of their mission.

To reduce the operational burden of running experiments, units can work with enterprise systems management and Chief Information Officers to build testing features into proposal management systems. Every improvement to this infrastructure lowers the cost of all following experiments.

Without careful attention, experiments could become burdensome for programs to partake in. Metascience units should build light-touch experimentation into default systems and provide program offices with incentives to participate. In some randomized controlled trials of new approaches, metascience units could provide additional funding for control groups, so programs don’t have to sacrifice portions of their budgets to generate evidence.

For internal data, the metascience unit should coordinate with system owners to capture and structure the data that makes evidence building possible. Much of the unit’s early value will be in organizing administrative data on grant applications and selection processes, especially non-winning applications, that is largely inaccessible to external researchers.

But the unit should also go beyond traditional output measures like citations and patents, tracking a wider range of outcomes: measures of scientific breakthroughs, grant turnaround times, staff time per award, and other process costs. Centralizing data on turnaround times and reviewer feedback quality in particular would give science agencies a clearer picture of how their processes perform over time.

Subscribe now

Implementation

Metascience units will only pay off if enabled to answer a science agency’s most important questions. Without sufficient staffing, support for cross-program experimentation, or the determination to scale the approaches that work, the promise of a metascience unit may end up unrealized.

Rather than being buried in bureaucracy, metascience units should be housed in an agency leadership office and entrusted with the authority to mandate pilots or experiments across agency programs. This ensures alignment with agency-wide goals, protects against capture by any one program, and provides a clear home for experimentation, rather than having the unit serve an advisory role.

To execute on the science agency’s metascience vision, leaders should prioritize recruiting talent to staff the offices and building strong relationships with stakeholders. These units should be staffed with a mix of civil servants and rotational experts like academic economists brought in via the Intergovernment Personnel Act (IPA). Career roles will hold institutional memory, while shorter rotational roles can bring in new ideas from the field.3 These staff must pay attention to the unit’s relationship with the metascience community and not assume it naturally maintains itself.

Metascience units should not function as traditional grantmaking or program evaluation offices

Academic metascientists have contributed strong research on some of the questions outlined here, and agencies should continue to support these researchers with external grants. But metascience units should not become grants or contracting offices for metascience. This would distract from their distinct comparative advantage: conducting internal analysis and supporting decision making.

For example, the programs on Science of Science & Innovation Policy (SciSIP) and Science of Science: Discovery, Communication and Impact (SoS:DCI) at NSF have funded foundational metascience research. But these sorts of programs do not focus on NSF’s own performance, nor should they. Tasking a metascience unit with external science-of-science grantmaking would distract from its goal, forcing the unit to develop extensive grant and contract management capabilities in addition to its core metascience capabilities. The metascience unit should maintain awareness of the external literature to inform the experiments it runs, but it should not be primarily responsible for the oversight of those awards.

To develop actionable research, agencies need proactive and nimble analysis directed at decision support. The evaluation offices that responded to the Evidence Act pushed agencies to make better use of their data, but did not produce the actionable evidence needed to make decisions. Evaluation offices often included a mix of managing traditional program evaluation activities (often conducted by third party contractors), reporting to Congress and regulatory agencies, and requesting and organizing data from offices across their agency. Unfortunately, in many cases they were overly burdened with compliance exercises and underdeveloped in key capability areas, which complicated their success. Metascience units must learn from those lessons. Agencies should resist the urge to combine metascience units with these and other program performance oversight functions. These units should be entrepreneurial and forward-looking, not compliance and backward-looking.

Subscribe now

Which agencies should have metascience units?

Not all science agencies need a metascience unit. To begin with, metascience units should be placed in the agencies that are the largest science funders. The NSF’s Fiscal Year 2027 Budget Request includes funding for a metascience unit, and the NIH is in the process of standing up OREPA to perform some functions of a metascience unit. The NIH has substantial internal capacity already, including a science of science scholars program, but due to the complexity of NIH’s 27 Institute and Center structure, OREPA could still unlock more effective cross-agency experimentation if given sufficient authority.

Additional agencies that should consider them include the Department of Energy (DOE), Department of War (DOW), and NASA. Given the prominence of Genesis Mission and massive ongoing infrastructure investments in the National Labs, DOE has unique metascience questions that won’t be answered by other agencies. DOW and NASA invest in diverse and complex R&D programs using a wide variety of innovative financing approaches, and both agencies could focus a metascience unit’s attention on industry-specific, translational questions.

Setting up metascience units for long-term success

Metascience units will likely produce decision-relevant evidence faster than external academics, and the experiments we suggested above could inform program design now. But the social return on scientific research is realized in long timelines (often over a decade or more), and some metascience unit projects will take time to pay off, likely crossing presidential administrations.

Those long-horizon payoffs require not treating these units as disposable, short-term efforts. Instead, metascience units should receive stable, sustained investment even as individual experiments resolve quickly. The White House and Congress have important roles to play to that end.

The White House has been supportive of metascience units, as exemplified by the Science: A New Golden Age report. To ensure the units’ success, they should support agency leadership in staffing and funding requests for metascience units, as well as providing flexibility and recognizing that the units may need to shift focus.

Metascience units provide benefits regardless of which party is in power by providing insights on science agency effectiveness and opportunities for improvement. Congress and the White House should therefore ensure that metascience units remain apolitical, truth-seeking offices and, to that end, support the sharing of findings publicly. A coordination body, perhaps convened by OSTP as a National Science and Technology Council subcommittee, could help in coordinating between metascience units, facilitating information sharing, and encouraging the external publication of results.

It’s also important to remember that metascience units themselves are a new institution in science — and are an experiment. As such, they too should be evaluated and improved based on evidence. And if they succeed in delivering more efficient, higher impact science, metascience units could be their own kind of breakthrough.

We just wrapped the third year of the Economics of Ideas, Science, and Innovation online short course. Targeted at economics PhD students, the course features many of the luminaries of innovation economics. This time around, we recorded the lectures and will be releasing one every other week to Macroscience subscribers (for free!), starting today. Regular Macroscience essays will continue on alternate weeks.

The reason we’ve held three iterations of this course and are publishing the recordings is that we believe more people should study the economics of innovation. Let me suggest five reasons.

Subscribe now

1. Impact

The best way to increase human flourishing in the long run is via technological progress, the speed of which is in part determined by social systems. The economics of innovation is the study of the social systems that drive technological progress — markets, governments, non-profits, universities, etc. There is no reason to think these systems already operate as well as they could, so better understanding could help us design better ones. Because ideas and discoveries are public goods and technological progress is cumulative, even small increases in the rate of discovery have enormous long-run effects on human welfare.

2. Government Demand

We need more people who understand this material. Scientific expertise is in short supply in some parts of government (a small share of legislators have STEM backgrounds), and this is especially true for social science research about science and innovation. While the US government has infrastructure for bringing in scientific expertise (nearly every major agency has a chief scientist and the American Association for the Advancement of Science places 250+ science policy fellows per year in government) and employs many economists, the intersection of these two sets is surprisingly small. In short, there are scientists and economists in the government, but few economists focused on science policy. Yet people with this kind of expertise are needed in science agencies, the White House Office of Science and Technology Policy, congressional offices, and in the Metascience Units that are emerging in the US and abroad.

3. Research Opportunity

For those interested in a research career, this is an exciting time. Scientific progress often accelerates when new tools for research come online, and AI seems poised to be just such a research tool. Modern AI is especially good at working with text, which means it’s very well suited to the economics of innovation, as scientific and technological progress generates a lot of text (scientific papers, patents, and software code).

Subscribe now

4. Creative Destruction

New AI tools make this an exciting time to study the economics of innovation for another reason. AI will likely change the way we do science, and it’s not that unlikely that we’ll need entirely new theories and empirics to design optimal policy. The results derived prior to AI may become like economic history, with questionable relevance to optimal policy design.

5. Inherent Mystery

Finally, the economics of innovation is, in itself, fascinating. Cumulative improvements in toolmaking — technological progress — is one of the things that sets humans apart from the rest of the animal kingdom, and this special feature of ours mostly stems from our social systems, rather than our biological capabilities. Species close to biologically modern humans existed for many thousands of years without strong technological progress. And we still don’t fully understand so much of how this system works. What a great thing to study!

If that intrigued you, I hope you’ll check out the Economics of Ideas, Science, and Innovation PhD short course. The course consists of 11 roughly hour-long lectures. Each is taught by a different expert — truly a who’s who of innovation economics:

1. Introduction to the Economics of Ideas, by Benjamin Jones

2. Idea-Based Models of Economic Growth, by Chad Jones

3. The Supply of Innovators, by Ina Ganguli

4. Open Science as an Economic Institution, by Pierre Azoulay

5. The Direction of Science, by Kyle Myers

6. Science and the Returns to R&D, by Matt Clancy (me)

7. AI and Innovation, by Kevin Bryan

8. Innovation Policy, by John Van Reenen

9. Immigration and Innovation, by Michael Clemens

10. Competition and Innovation, by Mitsuru Igami

11. Patent Policy, by Janet Freilich

Subscribe to Macroscience for an introduction to each course, along with recommended reading. If you find this topic interesting but aren’t yet looking for a PhD-level course, check out the Metascience 101 podcast, also produced by IFP.

Subscribe now

We’re launching the course with Benjamin Jones’ introduction to the Economics of Ideas. In this lecture, Jones discusses ideas as unique goods, with a particular focus on how market failures and spillovers can justify government support for the production of ideas.

Here are the readings Jones assigned to the PhD students (truly some of the foundational pieces on the economics of innovation):

• Arrow, Kenneth. “Economic Welfare and the Allocation of Resources for Invention.” In The Rate and Direction of Inventive Activity: Economic and Social Factors (1962). Princeton, NJ: Princeton University Press, 609-625.

• Jones, Benjamin F. and Summers, Lawrence H. “A Calculation of the Social Returns to Innovation.” In Innovation and Public Policy, University of Chicago Press (2021).

• Bloom, Nicholas, Mark Schankerman, and John Van Reenen. “Identifying Technology Spillovers and Product Market Rivalry.” Econometrica 81(4) (2013): 1347-1393.

• Jones, Benjamin F. “The Burden of Knowledge and the ‛Death of the Renaissance Man’: Is Innovation Getting Harder?” Review of Economic Studies 76(1) (2009): 283-317.

To paraphrase Jones in his presentation, if the social returns to R&D are high, the social returns to studying R&D are very high.

Go forth and produce some social returns!

A full transcript is available here.

Subscribe now

• I want first to discuss why we need a course specially about the economics of ideas.

• We’ll talk about the nature of ideas, which is part of the answer to that question. Ideas are special and unusual goods that have very different properties from other kinds of goods.

• I’m going to close by helping us think about spillovers and market failures.

This is an area where our free market welfare theorems tend to break down. That means that there’s going to be a distinctive role for policy in the innovation space — for particular institutions to try to overcome some of these market failures. In addition to innovation driving dynamics in the economy, this is also an area where there needs to be an enormous amount of policy. That gives you a whole different perspective, and there’s so much to be discovered still about what kinds of policies are effective.

Why a course on innovation?

The advance of ideas is a profound, propulsive force for rising standards of living, increasing health, and longer lives. This is showing rates of increases in income per capita, in real terms, going back to the 11th century.

In the 19th and 20th centuries, we see a much more rapid expansion. This course begins with the Industrial Revolution. AI is the latest of a series of general purpose technologies that have been part of this explosive, ahistorical — from a prior perspective — expansion in standards of living.

You can try to get at why we think ideas matter here, and you could get into macro residuals. My colleague at Northwestern, Joel Mokyr, who just won the Nobel Prize this fall, has a book, Lever of Riches, which I read during my PhD and recommend to you strongly. It’s a beautiful, technology-by-technology take about how advancing ideas, often embodied in physical capital equipment, have greatly advanced productivity. It’s a very direct take on how important ideas have been throughout this period and particularly in the Industrial Revolution.

More powerful arguments for why we should think about ideas come from thinking about a particular sector of the economy and how it’s changed.

• Transportation: In the middle of the 19th century — before trains, automobiles, and airplanes — if you wanted to get places, there was livestock; riding, or being pulled by, a horse. In the United States, if you wanted to go west you might be being pulled by oxen, who are going to be both your propulsion and possibly your food, because to get from Illinois, in the middle of the country, to California in the west, took something like six months. It was slow, dangerous, and you had to bring your own food. Today you can make that journey in less than six hours on a jet-powered airplane. Six months to six hours. How do we make that huge improvement in transportation efficiency? It has to do with a lot of new ideas that didn’t exist in the middle of the 19th century.

• Farming: We used to do a lot of hand labor. Go back to the late 19th century in the United States: 80-90% of all people were farmers. Now it’s something like 2%. We’ve had incredible improvements in productivity including mechanized agriculture: machines that can sow, harvest, and process grains.

• Computers: If you’ve seen the movie Hidden Figures, it’s about a team of women at NASA who were, as a job, called “computers,” because that’s what they did. They would be computing, by hand and with slide rules, orbital trajectories for launching satellites. They were replaced by what we think of today as computers, which are machines. At first they were replaced by an IBM mainframe. We know how much the advance of computing has done in terms of the breadth of effects across society.

We didn’t get to our standard of living today by saving and doing more of what we knew how to do in the 19th century. We’re not richer because we have more wagons per person. There’s something very different going on — we’re coming up with better ways to transport, harvest, and compute. Those are driving massive increases in output per hour, and advancing standards of living.

In those pictures we’re not seeing the ideas — we’re seeing implementations of those ideas: machines, for example.

Much of economics is built around the thing on the left — chairs. We think about production functions, prices, markets, and firms. But the thing we’re talking about in this course is on the right: it’s an idea. It’s the quadratic formula. One way to encapsulate that distinction is that there’s a knowledge production function that might be quite different from a production function for ordinary goods or services, like chairs.

To put it in formal terms, pretty much everything you’ll hear over the next 13 weeks is going to fit into this equation.

• What is the output on the left-hand side? A is some stock of knowledge, or maybe a measure of productivity or health, like how long you live, or survivability from a disease. We’re trying to improve A over time. The output of ideas is an improvement in the standard of living, through increases in productivity or health.

• On the right-hand side, in that function q, I have the natural inputs.

  • H is human capital. There’s a lot of human capital going into ideas production, which might be distinctive from the human capital that goes into ordinary production of a chair.

  • K is physical capital, like machinery, where you’re going to use a lot of specialized machinery like telescopes, or microscopes, or particle accelerators, or a lot of Stata or R doing statistics in economics.

  • Z would be the institutions that are going to shape incentives: rewards for creating ideas, which are very distinctive in science and R&D from other institutions in society.

  • Then there’s the current state of knowledge itself, A, which may have an implication for the ease with which we can continue to improve our understanding.

I’m not going to talk about AI today because I’m doing an introduction. But if you want to think about AI, AI is K — physical capital. A necessary input into idea production through history has been human minds — without that, you can’t make any progress. You can think about AI as shifting some of that to K. Maybe it’s really high quality K. To be determined. But that’s one way to think about AI in a coherent framework.

Thinking about supporting innovative activity, a wide range of institutions and policies are directed at this peculiar activity of idea production. You’ve got:

• The US Patent and Trademark Office: the patent system supporting private incentives to invest in research and development.

• Grant funders for science like the National Institutes of Health, or the European Research Council.

• Laboratories where things are actually happening: multinational collaborations like CERN for particle accelerators.

• On the bottom, that’s my university. I’m sitting along that lake right now. Lots of labs here doing things.

• Argonne is not a university but a US national energy laboratory.

• The Howard Hughes Medical Institute is a philanthropic, very high-end health research lab.

• Government funders for more solution-oriented process like DARPA, which is within the US Department of Defense, trying to come up with defense solutions to various problems.

• Venture capital funders like Bessemer.

• Philanthropies like Sloan.

• Private sector corporate labs like Abbott, which is trying to come up with new drugs for medical devices.

There’s a wide range of private, nonprofit and public institutions coming up with new ideas, funding them, or coloring the incentives with which we do things.

To encapsulate the first question for our course, why have an economics of ideas? First, because the advance of ideas informs central phenomena. It’s an incredibly powerful, first order force in understanding the path of economic prosperity, which would include income, health, and also how inequality evolves between labor and capital, and between different kinds of labor. It’s about the dynamics of how markets, industries, and international trade work. It’s about the role of specific institutions and policy that are suited and designed particularly for this peculiar space of idea production.

We talked about the chair versus the quadratic formula — there’s something about ideas that’s distinctive, and I’m going to try to clarify that in an introductory way today. It’s from those distinctive features that you can begin to build out this more complicated, rich understanding of idea production and all of its features. The related issue, which I’ll come to very quickly now, is that because of these special features of ideas, it’s natural to think there are very large market failures. The market is not going to produce ideas correctly, left to its own devices, and that’s why we need public policy and places like Institute for Progress to help think about how we can do that policy better.

I want to talk about the nature of ideas, and I’m going to come to several features: non-rivalry, excludability and cumulativeness. I’m not going to talk much about uncertainty in the interest of time, but I can mention it. These are features that are going to underpin the market failures associated with ideas, and in some sense this course.

Non-rivalry

This is a somewhat odd concept when you first hear it, and then it becomes very natural. Most goods are rival goods: chairs, cars, or computers. If you’re sitting in the chair, no one else can use the chair at the same time. You’re rivals. If someone is driving a car, you can’t be driving that same car somewhere else at the same time. If you’re getting a visit with a dentist, they can’t provide that service to somebody else’s teeth at the same moment. Ordinary goods and services are rival: if one person’s getting the service or using the good, nobody else can at the same time.

But with ideas, at the core of it all is this concept — like the quadratic formula. Let’s say I’m using the quadratic formula right now to solve some algebra problem. Andrew could be using it at the same time. So could Matt. So could all of you. There’s no sense in which my using the quadratic formula is at all influencing your ability to use and get value out of it. We’re not rivals. If I’m driving the car, you can’t drive the car. But if I’m using the quadratic formula — no effect on your ability to use it.

Ideas have this non-rival property: algebra, a biomedical concept like the germ theory of disease, the use of an assembly line. Once Henry Ford puts in an assembly line to make cars, every other car company’s like, “Maybe I could do that.” Or you’re making something else, scaffolds, or chairs — I could use an assembly line. A chemical process. Regression: if I run Ordinary Least Squares, you can too. Jennifer Doudna comes up with CRISPR. Now that might be patented and be a source of some debate, but when the patent expires, people can use it for free. There’s nothing fundamental about the idea that would say that your use of CRISPR would make it harder for me to use it.

That property, which is essential, is going to have very important implications. It’s going to mean that there’s a lot of value in ideas because value spills over. If one person creates it, it can spill over to other people. That’s going to lead to market failures. This course is going to suggest that markets are going to underinvest in new ideas. If the markets underinvest, we’re not getting enough of this incredibly important thing. So we’re going to want to care a lot about policy.

If it’s non-rival, let’s say one of you comes up with an idea, and you put it out in a paper and everyone’s like, “Wow, it’s a new method,” or insight, or fact. Other people will be like, “I’m going to start making use of that.” But because everyone else is getting value from it, that means that you, in creating it, are not getting the whole value of it to society. You’re getting the value you use it for yourself, and then its value is spilling over through non-rivalry to all these other people. So there are positive spillovers.

But where it becomes more problematic from a market investment point of view is as follows. The first time you come up with the idea — let’s say you’re going to write a book, or record a music album, or come up with a new pharmaceutical that fights a certain kind of cancer. Often the act of creation, like writing a book, is quite expensive. It takes a lot of time and effort. Coming up with a new pharmaceutical can cost a firm hundreds of millions of dollars to find it, show that it works, and that it doesn’t have bad side effects. So there’s this huge fixed cost to producing an idea. But if you show the pharmaceutical works, other people — if they could access the pharmaceutical — could make copies of the pill, not having to bear any of that fixed cost. If you make a book and everyone could make a copy and read it as a PDF. If you recorded an album at some expense and then everyone else can listen to the music and make copies and we all can listen to the same song at the same time for free. It’s non-rival. Then what happens? A market’s going to make it hard for this work.

Why? Here I’m showing you the units of ideas produced along the x-axis. But there was a fixed cost to come up with the idea. For the person who created the idea — they’ve got this fixed cost of coming up with a pharmaceutical and they’re going to make more copies of the pill, so their average costs are going to decline. But because of the fixed cost, their average cost is always going to be above the marginal cost of making a pill, because they still have to pay off this fixed cost of producing the drug in the first place.

Any entrant who wants to make copies of the pharmaceutical, or the song, doesn’t have to pay that fixed cost — they just make copies. Which means that if competitive entry happens, they’ll come in, and they can price just above their marginal cost and make money, but they’re going to be pricing below the average cost from the idea creator. Who’s going to win the market? Not the idea creator. They can’t charge that low because they have to pay themselves off for the fixed cost of creation. It’s the entrants who come in and copy with the non-rival ideas that can serve the market at the lowest price. So the person who actually did the invention goes out of business, from non-rivalry alone, if you let that be the dominating force.

In a context of non-rivalry, it’s very hard to invent. You might do it because we’re creative. I like to write a book; it makes me feel good. But you’re probably not going to marshal hundreds of millions of dollars to make a drug, because you’re going to lose all your money even if you’re successful. This is the first key market failure, and we need to solve it. To solve it, we have to get to a second concept, closely related to non-rivalry, but it’s more of a choice of society, which is excludability.

Excludability

Ideas themselves are not typically naturally excludable. They’re non-rival. If I heard about the quadratic formula, I’ll just use it. If I’m going to come in and use it for free — and I might undercut you who created the idea — in order to create a private decentralized incentive for someone to create that idea, we need to let them exclude others from using it. In other words, we need to give you something like a patent. If you did a pharmaceutical, and I could come in and undercut you and there was nothing you could do about it, you would have no incentive to invest. But if I give you a patent, I make you a monopolist — that gives you the right to exclude others from using your pharmaceutical unless you want to let them. You could charge a license or a royalty to let them do that. If it’s my pharmaceutical and I have a patent, I can decide who gets to produce it. Now I can charge a much higher price, and then pay off my fixed cost of investment. That’s a legal solution relying on creating intellectual property.

In general, whether you have excludability is going to depend on things like the policy institutional space in a country and the strength of the legal system. It can also depend on the technology itself. The patent system is the most obvious form of creating excludability. Think of it like this. Ideas are non-rival. But you can choose to make them excludable by allowing patents. Other intellectual property would be copyrights, trademarks, non-compete agreements, or trade secrets. There are other ways to do it which cover different kinds of creations. It’s up to a society to decide how much legal protection we want to give to certain ideas and creations, how long the patent right should be, how long you want a monopoly around these design questions.

The other way to go is not necessarily relying on intellectual property. A company or a person might get excludability in other ways. One thing is you can make it really hard to copy. The idea itself is non-rival, but you might encrypt it. An example would be satellite radio. If you’re driving around, the satellites are sending down the radio signals to all cars. Everyone is getting the signal, but you can’t listen to it unless you pay a subscription, because that is going to give your decoder in your car the decryption ability to listen to that channel. So there we don’t need patents. You could play a video game online peer-to-peer on some server and they don’t have to have a patent or intellectual property — because they can control your access to the server, and you can only play the game if you have access to the server. If you can control some necessary gatekeeping thing, or keep it secret, and people can’t figure out because it’s an internal process to your firm, maybe you can get excludability, despite the non-rival nature of things. This is why we’re going to have things like the patent system, because we need to create some incentive where the market without that property right would not succeed.

Uncertainty

Uncertainty is an absolutely fundamental feature of ideas. It’s not just that I’m going to flip a coin: is it heads or tails? I’m not sure. There’s that element where you’re not sure. If you start a drug program, 9 out of 10 pharmaceuticals that go into the first phase trial don’t make it through. They think they might work, but until they try it — they’re going to fail most of the time. Venture capital firms — 9 out of 10 bets are losers. They’re going to lose most of the time, and hope that some bets are really successful and on average carry the portfolio.

But it’s not just that you have a 1 in 10 chance. It seems to be that we can’t even foresee the reason an idea will be valuable or not. Before the internet, there’s NSFNET; before that, ARPANET. ARPANET was created by DARPA, a research agency in the Department of Defense. What were they doing? Let’s say we’re trying to communicate with each other across our operations, but someone knocks out one of our communications towers. We want to have a robust system of communication so we can still communicate. The idea was that you would divide information up into packets, and send it on many different routes through a network. If you lose one node of communications, information can still be rerouted across different nodes and people can still be in communication. The early idea of the internet for them was robust communications when you’re under attack. Were they thinking about Amazon, Uber, or e-tail? They were thinking none of those things. ARPANET was around for a while before anyone began to think, “Maybe this could be useful for something else.” So it’s very hard to know which ideas are going to take off.

Cumulativeness

The other feature of ideas that’s very important is cumulativeness. You create a spillover. It’s non-rival. If I come up with a song, other people can listen to it. If you read a book, other people can read it. But it’s not just that they can imitate or use the idea. It’s that that idea has an influence on future idea creation. Sometimes math achievements look like, “What’s the use of that?” But then later they prove important because they unleash further applications. An advance in physics might lead eventually to a new material, and that changes how we design airframes. So there’s an intertemporal spillover. Another version is you create a smartphone platform. Having created that, people can write apps for it. So there’s an intertemporal spillover on the ability to do software applications.

This intertemporal part seems very fundamental to knowledge. The most quoted line in all of innovation studies goes back to Isaac Newton, who said, “If I have seen further it is by standing on ye shoulders of giants.” He wrote this in a letter to a contemporary of his, a physicist, Hooke. He’s saying, “I can take a step forward because I have the benefit of all the knowledge that’s already been accumulated, from which I can step forward.”

If knowledge is cumulative, if there’s much more that is known now and that’s an input into future knowledge creation, does that make your lives as young economists easier or harder? Does it make it easier to do research today and come up with a big idea?

In one sense, there’s a lot more known and a lot of those things are tools. We can search the internet, look at Google Scholar, and read lots of papers. AI is a new tool. If you want to do econometric analysis, it’s easier doing it with modern computers than with punch cards on a mainframe, or before that trying to do it by hand, writing down the matrices and inverting them — that’s a pain. So certain tools make it more productive. Also the constituent matter that we can build on is vast; so we have lots of opportunities to combine things that are new.

On the other hand, and the thing I’m going to go in the direction of, there’s so much more known, so it’s harder to get to the frontier, because you have to learn something of what’s come before. Every year, there’s more that has been discovered. In some sense that’s an accumulating mass of knowledge. That’s what I call the “burden of knowledge,” which has the effect of making people narrower.

This is one of my earlier ideas in research. I hope the burden of knowledge is an evocative way to think about it. The question is, what happens if there is an advancing stock of knowledge, from the perspective of human capital accumulation and ultimately innovative potential? We’re born knowing nothing. You haven’t been taught anything. You know certain things instinctively and then you have to go learn. But you’re being born, with every passing year, into a place with so much more knowledge — biological, statistical, mathematical, knowledge of the known universe. How can you respond to that? You’re born knowing nothing and there’s a bigger mountain of knowledge that you’re confronting.

You have two options:

1. You could spend longer in school, spend more time training, as there’s more to know.

2. You could say, “I can’t possibly know everything, so I’ll become much narrower.” Maybe going back in Plato’s time, the inspiration may have been the same, and Aristotle could make all these contributions across the waterfront of knowledge at the time. But you can’t really be a leading biologist even anymore. Biology is so many sub-areas. A physicist, meaning what? You do cosmology, or quantum, or solid state physics and materials. These are very different math, and sets of ideas. So people get much narrower.

It may be that computers are making us more productive, and there’s lots of great ideas and they make us more creative and inspiration is quite the same in terms of a process. Yet if we spend more of our life cycle training, we have less time, other things equal, to innovate. We see this in the data. A bachelor’s degree used to be the ultimate degree in electrical engineering. Then back in the early 20th century there’s a master’s degree, then a PhD, now they do postdocs, so you spend much longer in your early years training.

More importantly, we’re narrower. If you’re narrower, that means the constituent matter that you can be inspired from is quite narrow. You’re doing some combinatoric search, based on things that you know — that are within your specialty. But also, your ideas are probably applicable to your specialty, and may not be as broad in their impact, because you yourself are narrow. So maybe we’re all branching into these niches. Our ideas are important in those niches, but there’s so many niches, and your collective impact — on productivity as a whole and on the economy as an individual researcher — is low.

You’re going to hear from Chad next week about idea-based models for economic growth. One of his main findings, early in his career, is that we seem to be putting a lot more effort into R&D — more people, more money — and getting less out in terms of Total Factor Productivity growth per person or per dollar spent. You could think of this narrowness effect of the burden of knowledge as one way to explain that phenomenon.

I had that idea when I was getting my PhD, and I went out using patent data — which is what we had — and looked at various patterns.

This upper picture is the age at which people get their first patent, and it looks like that’s going up. Another version of this in another paper is the age at which people do Nobel Prize winning work — not when they win the Nobel Prize, but when they do the research for which they would win it. Or people who are known technological inventors, like Thomas Edison and the light bulb, or Bell with the telephone — how old are they when they produce these signature contributions? You can see that the age at which people make those contributions is going up. If you unpack all this, you’re going to find that it’s going down in your 20s. You can still produce Nobel Prize-winning insights in your 20s. It’s just much less common now than it used to be, and people are typically peaking in their early 40s.

If you look at the lower rows, the one on the left is a measure of narrowness. It’s looking at an inventor and saying, let’s get two patents in a row that they do and ask if they’re likely to jump between one technology class on the patent to a new technology class. They’re less likely to jump with time, suggesting people’s ambit of research is more narrow.

On the right is one of our classic responses, which is — if everyone’s getting super narrow, what we might do is work in teams to pull together complementary forms of expertise. You see that too across all forms of patenting and science. Not only that, you see that teams are increasingly the source of the higher impact work. It used to be, in fields like math, if you wrote alone, you would do better than if you worked with a co-author or in a team. But now we see that reverse. Even in math today you’re going to have higher impact work if you work in a team than if you’re working alone.

Maybe AI is going to make you guys very productive. On the other hand, there is this training niche challenge that you have to overcome, and our classic solution to that is collaboration. Something you want to be thinking about, if you’re really good at econometrics but you aren’t a theorist, or vice versa, rather than bite your head against the wall of the other thing, can you collaborate with someone who brings those complementary skills together with you and produce a stronger paper more quickly.

Spillovers

The last piece I wanted to talk about was spillovers, to start bringing us around to the idea that this may be an incredibly high return activity and that we may greatly underinvest in it. That’s going to open up the policy and institutional space to us.

I had two papers I was going to talk about. One was Bloom, Schankerman, and Van Reenen, which asks whether firms’ R&D spills over to other firms, and what the returns are. A firm investing its own money in R&D is trying to come up with better products or services, or lower cost production methods, and they would capture the value of that themselves. But to the extent that they’re putting in place new discoveries, products, or services, other firms could learn about that and then be inspired for spillovers to do it themselves. That would not go to the private return of the firm. In fact, it might go against the private return of the firm because the other firms would now compete with them with imitative ideas. So this paper tries to pull apart key dimensions in those spillovers in a very clever and successful way. Given the time I’m going to go to the second paper, also because you’re going to hear from John Van Reenen directly later in the quarter.

Let me ask you a different question about spillovers. There are a number of spillovers in ideas:

• There’s non-rivalry. You produce some new idea and get some value out of it, but then I see it. I can use the idea if it’s not excluded through a patent, and then I get value out of it. Even if it was excluded through a patent, I could probably be inspired by your thing and come up with my own version, which doesn’t quite violate your patent, but still you inspired me and I got some of the value. That’s what we call an “imitative spillover.”

• There’s the intertemporal component. By producing a new mathematical idea, the next person can say, “That inspires me to something else,” and they’re going to capture the value in some intertemporal sense.

• If you make something new and you sell it in a market, people are willing to pay for it. They’re paying because they’re getting value in excess of what they’re paying. The consumer of your ideas is getting a consumer surplus benefit from your creation that is not going to be captured by the creator.

So there are several reasons to think that idea production would be undervalued. The private inventor, or the scientist, is getting less value personally than the social value that is created.

On the other hand, there are a couple of forces that might go the other way. One feature among private sector players is what we call business stealing. Let’s take Amazon. They’re going to come up with a new business model based on buying and selling things online and delivering them to you — originally books. They’re going to kill a lot of existing brick and mortar bookstores that find they’re not able to compete with Amazon. There’s going to be a huge shrinkage in the bookstore sector. If you think about the private returns to Amazon, they’re doing something that is valued by consumers and there’s some social gain. On the other hand, the private return of Amazon is also a transfer from the existing bookstores to Amazon selling them instead. That isn’t a social gain. It’s just a transfer. They are stealing business from the existing bookstores. That could mean that the return to the investors in Amazon is too high, because a lot of what they’re getting is stealing other people’s business but not really adding much that’s more efficient to society. So business stealing can be a negative spillover of innovation. It’s part of the creative destruction idea that you’d see in Philippe Aghion‘s work, who also won the Nobel in the fall, working with Peter Howitt.

Then the last one might be a racing phenomenon. If we’re all different research teams spending money and time in a race after a superconducting material, or a particular new drug, it’s like a tournament. One of us gets the patent, or the discovery, then the effort of all the other teams may be partly or fully wasted. In racing for certain opportunities, we may overinvest because we have a congestion or crowding externality on each other, or duplication externality from the perspective of the research spend.

In reality, there’s stories for positive and negative spillovers. It’s very much an empirical question, and papers like Bloom, Schankerman, and Van Reenen looking at business behavior net it out among the businesses in the sector and say, “It looks like on net a very positive spillover.” But even that is quite limited. What firms are doing is not really science. What about science like math or physics, and what about all that’s going on in universities? That’s a trickier question, because for a lot of things the spillovers are very diffuse.

Here are three examples.

• On the left, that equation is general relativity — that’s Einstein in 1915 and 1917. You might say, what is the benefit of general relativity? It’s a curiosity about the curvature of spacetime, and you might think it’s intriguing and neat, but not really useful. But it turns out that it’s incredibly useful. GPS satellites all work on atomic clocks that are very accurate. Those clocks have to be adjusted according to that equation, because they’re at lower gravity up in the air, and they’re moving at a speed compared to the surface of the earth. It turns out that if you don’t correct for general and special relativity, the whole GPS system doesn’t work. So you don’t get Uber — where we use GPS for drivers to find you and to navigate — unless you have the GPS system. You don’t have the GPS system unless you have relativity. That’s a hundred-year delay.

• That equation is an application of Riemannian geometry from the 1860s I think. Riemann came up with a geometry in math that seemed like it had no application. He was wondering about the geometry of spaces in more than three dimensions which have curvature. This non-Euclidean curving geometry in high dimensionality seems completely useless, because we live in three dimensions, and things aren’t curved. We can do everything in orthogonal planes or whatever. But it turned out that when it got to time and space there was a fourth dimension, and it turned out that mass was curving them, so the appropriate mathematics 60-70 years later for Einstein was Riemann. That’s going to lead to GPS. In other words, if I were to ask you the question, what’s the social value of geometry or general relativity, it’s a hard question to think about because of these really long lags. Obviously you need it. It’s necessary, but it’s not the only thing that’s necessary to the GPS system or Uber. So how would I allocate what the value of all that mobile internet I could put back on these equations? It’s very hard to think about.

• The one in the middle is Taq polymerase. That’s a protein that is the heart of Polymerase Chain Reaction, which is the heart of all modern biotech. It allows you to make millions of copies of the same strand of DNA, which we use for any kind of genetic engineering, or forensics, or PCR, as you probably know from COVID tests. That protein was discovered by a couple of biologists in a hot spring in Yellowstone. The organism it’s in is a bacteria that lives in really high temperatures, and at the time it was just a biological curiosity. That was in the 1960s. It turned out that that is going to unleash, a decade or two later, the entire biotech industry, and without that protein you can’t do it. What’s the social value of that research? This is now stuff that’s being funded by a government science agency.

• I put the iPhone up there because — try to imagine enumerating the social value, or cost, of the smartphone. It’s complicated, because there’s so many things it touches. How would I begin to think about these spillovers and all these different margins from that one device? So it’s very hard to do social returns in some adequate way. The more important the technology, the wider its use, probably the harder it is to measure, and we don’t want to miss those.

I want to close with one idea on this. There’s a big literature on how to calculate the social returns to R&D or to science. I’m going to give you a conceptual take that’s more macro, which I think will help motivate why what we’re doing in this course is so important and why we need policies. If we believe ideas are central to advancing prosperity — which I think there’s a huge literature coming from many angles telling us — without which we don’t really grow among advanced economies over the long run, could we harness that insight to think about the social returns from a macro perspective?

This goes to a paper I did a few years ago. The idea was to say — under the thought experiment of innovation and our modern understanding of it, as well as endogenous growth, growth theory, and macro in general — if you think of macro the variable is A, the state of knowledge, and if we don’t improve it we stay the same. If we just had all the ideas we have now, things would keep breaking and we’d have investment because I have to fix the computer I’m on, your car will break down, your house will need the windows repaired and the roof redone eventually. You’re going to do investment just to keep things going, but we wouldn’t be getting better at stuff. There’s diminishing returns to capital investment. So we can’t just rely on savings and investment. We need to improve the ideas. We didn’t get faster at transportation because of more wagons. We needed something much better than wagons.

What you can do as a thought experiment is say, “Imagine we just stopped innovating.” We would replace what breaks and stay where we are. You can use that insight to create an overall calculation of the social returns to R&D under some assumptions that you can then play with, that reveals an average return to that investment. I’ll show you the very simple baseline and then you can try to generalize it to confront a lot of other forces that you might think are important to deal with beyond the baseline. Then you could also think about how it relates to micro evidence, or other kinds of macro evidence.

Here’s the thought experiment. It’s got non-rivalry, temporal spillovers, imitation — it’s netting it all together.

Look at the upper row. Imagine you’re investing, on the green line, a certain amount of your total GDP into R&D — or innovative activity, however you want to measure that. R&D is 2-3% of GDP in most advanced economies. You’re following the green line. Then for one year, you’re going to go on to the black line, where everyone stops. We stop the course. You guys go work somewhere else for a while. The labs — we just put some sheets over the equipment. A year later, we come back, we pull the sheets off the equipment, and keep going on exactly the same projects that we were doing before. We just had a year-long delay. We don’t change the pathway of discovery. It’s the same projects and the same intertemporal set of projects. They’ll inspire the same things, but we just delay it a year. We save a bunch of money by not having to do it for a year. That’s what we invest in innovation. That’s the top line.

The next line is what would happen to growth in a simple model. If we don’t advance the state of knowledge, we stay at the same standard of living. So we’d be growing at — most advanced economies 2% per year — and then we wouldn’t grow for a year in the middle row, and then we just grow again on exactly the same path. To put this in income per capita terms, in the lowest row, you’d be growing at a log scale like 2% per year going up, and then for a year you wouldn’t grow at all. You wouldn’t get rich or poor, you just didn’t advance ideas, and then you would advance ideas again at the same rate as before.

What’s the cost and what’s the benefit?

f you invest 2-3% of your GDP in a given year, you would be 2% richer forever, because you would grow that year and you’d keep it, you don’t forget the ideas, they don’t depreciate, we still know algebra. So the value of that is this green wedge in the lowest row. In every future period, you’re going to be 2% richer if you invested in those ideas than if you didn’t. What’s the present value of being g% richer in every period? Its present value is g/r, where r is some discount rate. In other words, you invest a share x/y of your resources. That’s the cost in R&D for a year, and you get to be g/r in present value terms. That’s the thought experiment. There’s maybe a lot of imitation, there’s business stealing, all sorts of things are going on in a given year of R&D. Firms are destroying each other, some are winning, some are losing. Many drugs fail, some win, but on net, we get 1.8-2% richer per year. That gives us the net of all of that effort.

The present value of the benefits is g/r. The cost of investment is x/y from a share of resources. That’s the social benefit cost ratio. Now think about this intuitively. x/y is 2-3% of GDP, but you’re 2% richer next year — and the year after forever. That’s an amazing thing, because if you push up the level of ideas quite a bit today, given our R&D investment, and you get that benefit forever.

What do you get from that calculation?

g is something like 1.8%, 2% more accurately in the US. x/y is 2.7%. You can pick your discount rate, but I’m going to take a 5% discount rate. Look at that table below, you’ll see it says 13.3. That means, for every dollar we spend on R&D, we get $13 back in present value, which is incredibly high. It’s like a machine: you put in a dollar, you get all this money back. The logic that’s driving that is because you push up the level, and people generations from now are going to be smarter, because there’s Riemannian geometry. They’re able to figure out how to use it. The ideas don’t go away. You push things up and it stays there. And it turns out we don’t spend that much.

Maybe you think R&D formally measured is more than 2.7%. Maybe there’s a lot of innovation happening other ways and it’s badly measured. So this whole paper is about making this baseline calculation and then trying to kill it by making all sorts of different assumptions. You could put an adjustment factor I call β in front of that ratio.

Why would my $13 per dollar be too high?

• It might be because there’s lags: I invest now but I don’t get the growth right now, I get it for science sometime in the future.

• Maybe I have to embody the ideas in physical capital — they have to go into a smartphone, I have to spend on the smartphone.

• Maybe other sources of innovation are out there — not just people doing formal R&D.

On the other hand, there’s a bunch of reasons that that baseline might be too low.

• In macro we think that inflation is actually lower than we say because there’s a new goods bias if you follow that literature, so actually growth is higher than 1.8% in real terms. Maybe it’s more like 2.3% or 2.5%. That makes the returns to R&D even higher.

• We don’t actually value health directly in GDP. A lot of research is trying to make you healthier and live longer, not to make higher income per capita based out of productivity on the job.

• International spillovers. If the US does R&D, other countries can learn as well. So we’re a net exporter of ideas — the gain in standards of living can be far beyond your own borders.

You can balance these forces out. The conclusion is that it’s really hard to get the number of the benefit to be small compared to the cost, however you try to attack this. We actually said that on the very conservative end, you get $4-5 out per dollar. So this is a method where you’re trying to calculate the average return.

Where would that leave us? This is one version of the social returns. There’s many papers about the social returns to R&D doing different approaches, including a couple of very new ones. One of which, Fieldhouse and Mertens, is really about public investment in R&D alone and science, and it’s showing incredibly high returns. Not every paper agrees, but what you tend to see across a wide range of methods is repeatedly calculating very high social rates of return to idea creation. The paper I just showed you gives you some of that logic, because you push things up in a meaningful way that you benefit from in a future stream from your investments today.

In light of that, it seems like there are really high social returns. A lot of these papers are going to show higher returns on the margin. If we did another $1 billion of basic science, it’s going to be really valuable for society. So that means that both we’re going to have a bunch of institutions that are going to try to push on this, but also probably that they are currently way underfunded compared to their social benefit. Even conditional on their funding, do we have a good idea about the best way to design a science grant, or the patent system, or to run a university, or to create incentives for human capital like the tenure system, or how we give credit? All of these things are very open.

The questions in the bigger picture are:

• What are those policies?

• What are the right institutional structures?

• How should we design and fund these various systems?

There’s a big policy question. Then there are a lot of questions about understanding idea creation that are really interesting from a personal and a private sector perspective:

• Who captures value?

• How do you capture value from data?

• Where do you get excludability?

• Who gets credit?

• How does uncertainty affect our ability to produce and exchange ideas? There are some really interesting breakdowns in the market for ideas because of uncertainty between buyers and sellers. There are a lot of strategic interactions in markets in the production of ideas.

Then there are really big questions:

• Where do big ideas come from?

• Who produces them?

• How do they come up?

• What is one scholar doing versus another, versus an inventor, versus an entrepreneur?

• Does someone have a better process? What is that process?

• Are there better incentives? What are those incentive systems?

• What is the role of AI in all this? It’s coming at us very fast. Is it going to fundamentally change how we do things?

Let me close, trying to motivate you a little:

1. I tried to get across how important this subject is. The social returns to R&D are very high, because if you can make this system more efficient, the returns for society are vast.

2. It’s not an area where we’re just studying it because it’s curious and markets get it right. It needs interventions, and there are so many kinds. It’s a very complicated space, and there’s so much to be learned at a very micro level, to more industry sector levels, to macro levels, about how to do that well. That’s going to require great research.

3. This is a growing area with amazing data. When I started out doing this long ago, there was patent data. There was no data on papers and scientists. There was no real good data on entrepreneurs. There wasn’t really data on funders and who’s funding what. We didn’t have the full text of anything. Between the data sets that have emerged, and the tools that have emerged in parallel, both our classic econometric tools and compute that we have, but also now all these AI tools, in addition to our causal identification and structural tools, it is a vast opportunity, and there’s theoretical opportunities as well.

I have gotten to the point where every single data set I touch, I’m like, “I could spend the next 15 years on this data set,” and then I have to go over to something else. There’s so much data. Another way to say that is, our scarcity right now in the economics of innovation is not opportunity or data. It’s people. We need more people on that mountain figuring it out. I think there are great opportunities to find your own niche that is of great interest to the whole community on the mountain.

I hope this course, in addition to giving a taste of what people are doing at the frontier, and some of the core questions, can inspire you to realize how much great work there is to be done. I hope we can attract you into this research community.

The biggest challenge in intentionally driving breakthroughs through initiatives like the Cancer Moonshot or the Apollo Program is that, while we might want to focus on things that are important to society, it’s difficult to know what science is actually ready to be exploited. Kris and her co-authors developed a fascinating approach to predicting breakthroughs by using early signals in the volume of academic publications.

Kris is President and Founder of the Woodley Park Institute. This blog is cross-posted with Stuart Buck’s Good Science Project (subscribe here).

Picking winners or hedging bets?

Federal science agencies have long wrestled with the question of how to ensure their billions of dollars of grants and contracts result in maximum benefits for Americans.

On the surface, it might seem that a goal-oriented, interventionist style of management would be the best policy: prioritize our most pressing problems and distribute funding accordingly.This top-down strategy creates a unique hazard, though: bad choices can send resources in the wrong direction, resulting in slow progress that impedes the delivery of tangible benefits to citizens. Moreover, if decision makers pursue an applied advance when the underlying scientific principles remain poorly understood, they risk costly failures and a loss of support for the research enterprise at large. Examples include Nixon’s 1971 War on Cancer, undertaken on the premise that a cure could be achieved in less than a decade, the drive to build an operational commercial fusion plant by the mid-1990s, and the National Plan to Address Alzheimer’s Disease, which failed to produce the effective treatments and prevention strategies promised by 2025.

The contrasting laissez-faire approach is to hedge your bets. If the path to an advance is impossible to predict, then the wisest choice might be to distribute funding to as widely as possible. The main risk of this strategy is that spreading funding thinly may limit the resources available to the most promising research, resulting in the same negative outcomes as the interventionist approach: limited support for the most promising topics, slow progress, and delayed returns.

Subscribe now

In reality, at science agencies like the National Science Foundation and National Institutes of Health, program managers navigate between these two extremes, setting some funds aside to target high-priority areas and using the rest to cover the widest possible range of meritorious proposals. This middle path seems like the most rational way to proceed, but it still requires making choices. How much should be set aside for high priority topics? What should those topics be? How should we define merit? Expert opinion can provide a helpful guide, but leading experts can have remarkably different opinions about what matters and what should be prioritized. Moreover, experts and administrators alike can be risk averse, giving an edge to high-profile, well-established concepts.

What’s really needed is a data-driven framework to help guide decision-making, one that doesn’t simply repackage prestige or incumbency.

Recognizing breakthroughs years in advance

Many breakthroughs fail to attract funding or recognition at the earliest stages of development. Katalin Karikó, for example, spent years struggling to win grants for work on mRNA that would eventually win her and Drew Weissman a Nobel Prize. Similar stories can be told about Jim Allison’s groundbreaking studies leading to the development of cancer immunotherapy, Stan Pruisner’s demonstration that prions can self-replicate without DNA, or Robin Warren and Barry Marshall’s proof that Helicobacter pylori, not stress, causes ulcers.

Yet at some point, the scientific community recognizes and responds to transformative breakthroughs like these. How does that change happen? And what if science funders could identify the areas that are on the cusp of a breakthrough?

Those questions led my collaborators and me to study how scientists responded to past breakthroughs. Based on patterns in that data, we developed a means of predicting future breakthroughs, using biomedicine as a proof of concept.

It’s worth taking a moment to describe how our approach works. Briefly, our prediction process begins by grouping the 18 million or so peer-reviewed papers in PubMed, the authoritative database of biomedical research, into roughly 50,000 unique topics. A small subset of these topics are so widely acclaimed as breakthroughs that they have been recognized with a major prize like a Nobel or a Lasker Award. We theorized that studying the development of these outliers over time, especially before they gained acclaim, was the key to identifying future instances of them.

After defining topics, we chose 21 representative breakthroughs and asked what they looked like before they won prestigious prizes. To answer that question, we re-ran our topic-mapping algorithm repeatedly, winding back the clock one year each time, stringing together the years to follow the progress of each field as it grew and advanced.

We found that in each case, the years leading up to a discovery followed a pattern: a burst of papers on a new, rapidly evolving topic, many of which quickly became influential. These characteristics—the percent of papers that are new, the percentage of papers that are brought in from other topics, and the influence of each individual paper—can all be measured separately for any given topic. Together, they act as a predictive signal that is detectable an average of five years before a transformative discovery is made, and up to thirty years before the discovery receives a major prize. Scanning the current research landscape for topics that display this signal allows us to predict what work will likely produce a future breakthrough. Each year we analyzed includes four or five signals, a number that appears to have remained constant across a twenty year time frame.

In their classic work on the sociology of science, Bruno Latour and Steve Woolgar established that the desire to participate in a major discovery is an important motivator for scientists as they consider whether they should take up (or abandon) a research problem. Although further work is needed, the simplest explanation for our results is that this behavior is widespread enough to detect at scale. A foundational advance, or something like it, draws the attention of scientists and causes them to change the direction of their research. They vote with their feet, staking their own reputation and careers by publishing on and citing the new idea. That upswell of interest is the foundation of our breakthrough signal.

Subscribe now

What kind of research does our model identify as a breakthrough?

After building a model based on the patterns in our 21 preselected breakthroughs, we tested it by identifying all early signals of discovery from 1994 through 1997. Of the 18 signals we found in that time period, 17 can be traced forward in time to a breakthrough. Examples include super-resolution fluorescence microscopy (see the image above), the directed evolution of proteins and enzymes (Chemistry Nobel, 2018), and sequencing the human genome (National Medal of Science, 2008).

The discovery of the role of leptin signaling in obesity is a particularly interesting case study. Around 1950, scientists noticed the existence of a type of lab mouse that suffered from a unique inherited form of obesity. When given the freedom to choose their own diet, these mice ate more than normal and preferred food that was high in fat. Researchers had few clues as to why the animals overate; early experiments pointed to the existence of a soluble ob factor found circulating in the bloodstream that regulated appetite, but its identity was unknown.

In 1994, Jeffrey Friedman and colleagues identified and sequenced the ob gene, noting that the protein it encoded appeared to fit the profile of a circulating factor. A year later, a trio of high-profile publications demonstrated that ob was a hormone that regulated body weight and fat deposition by regulating appetite. One of these was co-authored by Friedman, who christened it leptin from the ancient Greek word for thin. Using our model, the data from 1996 make it clear this advance would become a breakthrough. Scientists had flocked to study the biology of the new hormone, generating the flurry of new papers and citations required to produce a signal. One of these demonstrated that serum leptin concentrations reflected the amount of adipose tissue in the human body, providing clinical validation of the earlier mouse studies. Before the identification of leptin, scientists were unable to point to a specific molecule that controlled appetite and adiposity, so that it was impossible to rule out non-physiological causes of obesity; its characterization fundamentally changed our concept of weight gain. In 2010, Friedman and fellow pioneer Douglas Coleman were awarded a Lasker for the discovery.

Our work also identifies breakthroughs in clinical practice and behavioral and social sciences, even though these areas are less likely to draw the attention of major prize committees. One example that falls into this category is the introduction of endovascular aneurysm repair, a minimally invasive surgical procedure for the treatment of abdominal aortic aneurysm, a leading cause of death among Americans over the age of 65. Relative to the previous standard of care, it reduces in-hospital mortality by almost four-fold. In 2017, Timothy Chuter received the American College of Surgeons’ prestigious Jacobsen Innovation award for his multiple refinements to the technique.

Another under-recognized breakthrough identified by our model is the development of standardized survey instruments to assess the quality of life for HIV⁺ patients. The antiretroviral cocktails introduced in the mid-1990s reduced AIDS mortality, but treatment came with serious side effects. The new evaluations showed that patients whose disease progressed had worse physical functioning than Americans with other chronic diseases, while those who were asymptomatic maintained a health-related quality of life on par with the overall US population. This evidence made it possible for patients and activists to argue that even in the absence of a total cure, and in spite of the side effects, alleviation of symptoms was a worthwhile priority for clinicians. My colleagues and I have failed to identify any significant recognition for the physicians and health policy experts who pioneered this advance, although in 2016, researchers called for the World Health Organization to set targets for good health-related quality of life as part of its framework to end the AIDS pandemic.

Subscribe now

From forecasting to funding

How should science funders use our predictions?

Any serious discussion of what we should do with this new capability needs to begin with an understanding of its limitations. We can’t (and shouldn’t) invest all of our resources in the small number of topics predicted to produce breakthroughs. Innovative new fields are born from existing ones; they rarely arise de novo. If funders don’t maintain a diverse portfolio, the breakthroughs of tomorrow will have no antecedents. Think of this as eating the seed corn, or killing the goose that lays the golden egg.

There are also multiple goals of funding beyond basic scientific discovery, including translating discoveries into clinical practice, developing new technologies, supporting economic development, and training new scientists. The investments needed to accomplish these goals are likely different from those required to support emerging breakthroughs.

Finally, we must be aware that we may not capture every instance of a breakthrough. The concept is fuzzy, and not every significant discovery is recognized with a major award. Further, breakthroughs are rare, meaning that our dataset is, by necessity, small. This makes formal estimates of accuracy challenging.

Although our overall success rate appears to be high, any one signal may turn out to be a false positive. We found one apparent example: the development of the third generation COX-2 inhibitors, Vioxx and Celebrex, which were much celebrated in the early 2000s as effective but non-addictive pain relievers, meets all the algorithmic criteria of a breakthrough. Five years after it received FDA approval, researchers demonstrated that Vioxx was associated with serious cardiovascular complications, and the manufacturer voluntarily pulled the drug from the market. Celebrex remains available, although like other NSAIDs, it carries an FDA warning for increased cardiovascular risk. Decision makers who were overly focused on the breakthrough potential of this research might have wasted substantial resources nurturing an area that ultimately proved to be a failure—but so would those who relied on expert opinion at the time.

That said, the potential benefits of funding the next mRNA vaccine or cancer immunotherapy years earlier than we might have are large. When we can’t assume the outcome of an action, it’s worth conducting an experiment. Can targeted support for areas that are poised to produce a breakthrough increase the return on scientific investment without cannibalizing future advances?

With the necessary administrative infrastructure in place, conducting this experiment would be straightforward. First, identify all the breakthrough signals between 2018 and 2023, select half at random, and commit to a decade of investment. For biomedicine, a six year window should yield about 30 topics with the potential to produce a breakthrough. Providing strong support to half of these might be accomplished for $300 million a year. That’s roughly half of the recent annual budget of the NIH Common Fund or around 20% of the fiscal year 2025 budget for ARPA-H. After five years, then again at ten, compare to see which group — the one that was actively managed, or the control — produced more breakthroughs, and on what timescale. Although we used biomedicine as an example, the same experiment could just as well be run for physics, materials science, or any other discipline.

Subscribe now

Any attempt to undertake such an experiment should keep three points top of mind. First, we can predict the topic of future breakthroughs, but the people and vision for how to move the work forward are still important. In practice, this means success requires active program managers with the expertise to develop the science and the authority to do so, including by recruiting investigators. Second, we would be wise to learn from the failures of past attempts to fund transformative research. High among these are funding modestly repackaged work, confusing the genuinely novel with the merely unfamiliar, and relying too much on incumbent investigators. All of these risks can be mitigated by thoroughly and intelligently integrating high quality data into portfolio management. Finally, we need to have patience. Even with strong funding support, producing a breakthrough takes time.

If supporting breakthroughs is the best way to nurture the birth of new fields, we also need to accurately describe and properly manage the other two stages, so that we invest wisely throughout the productive lifetime of ideas and divest once they reach the point of diminishing returns. We would benefit from more study of all of these phases, especially the transition from one to another. More research could answer questions like:

• When does a breakthrough become an established field?

• How do the availability of funding, the size and characteristics of the available workforce, and the accumulation of evidence that disagrees with prevailing models affect a field’s decline?

• Exactly how do old ideas give birth to new ones, and can we speed up the process?

The more we understand about the dynamics of scientific progress, the better we can maximize the odds that our curiosity leads to results. I’d call that finding a way to make our own luck.

Subscribe now

Two things can be true at once:

1. Science is the ultimate public good and has resulted in massive positive spillovers, making R&D one of the most important investments the federal government makes.1

2. The systems we use to fund, select, structure, and conduct that science have serious problems and could work much better.

The federal government spends nearly $200 billion a year on research and development. If we could make the process of identifying, distributing, and managing those funds even 5% more effective, we would effectively unlock another $10 billion a year for higher expected value science and technology.2 For context, that is slightly more than what we spend annually on NASA’s entire science budget, or on the National Cancer Institute.

This insight has driven a series of “science reform” movements over the years, some within existing science policy circles and some from other vantage points. Metascience is the study of the scientific enterprise, aimed at generating evidence about what increases the reliability, productivity, and impact of science. But different science reformers have distinct core hypotheses about what has gone wrong and what is most tractable to fix. How do the people building new Focused Research Organizations fit in with those working on the replication crisis? Are they all doing metascience? And how is metascience different from science policy in general?3

If policymakers and agency leaders are going to change how they spend billions of dollars of taxpayer money based on metascience advice, they should have a clear picture of the landscape of metascience and its various goals, perspectives, and terms.

This is our attempt to map the landscape. We’ve split the territory into six recognizable metascience ‘camps’. Each camp advances a distinct hypothesis about which parts of the scientific enterprise most need attention and how we should test what works.4

Subscribe now

A few caveats before we share the landscape.

• These camps are permeable. Many individuals and institutions move between two or three, and the borders are intellectual emphases rather than walls. Each is grappling with a different part of the same elephant.

• For this landscape, we are also limiting ourselves to metascience rather than the broader landscape of science policy. The line is debatable, but we think of science policy and metascience as overlapping circles: metascience asks how the scientific enterprise functions and how to make it work better, where evidence about outcomes is the primary arbiter. Science policy includes many metascience questions but also broader normative, political, diplomatic, and national security ones. Not all metascience is science policy, and not all science policy is metascience, but the two have plenty to say to each other.

• The methods we list for each hypothesis reflect dominant practice, not a logical constraint. In principle, most questions could be pursued with multiple methods.

With those caveats, here are the six camps and some examples of their associated research and initiatives.

Research Integrity

Hypothesis: Poor statistical practices and publication bias have often resulted in publishing inaccurate or overstated claims, leaving researchers to spend time following unproductive paths of inquiry. Science could be more reliable and current institutional incentives perpetuate the problem.

Theory of change: Fix methods and publication norms → fix the literature → more reliable science.

Policy relevance: This is the version of metascience that NIH Director Jay Bhattacharya has spoken about directly and launched initiatives to address. The “replication crisis” showed that in fields like behavioral psychology, a large fraction of published results don’t hold up when repeated (see examples below). A representative policy question: should we require replication studies as a condition of large grants?

Evidence and methods: Common methods include large-scale replication studies, meta-analyses, statistical power analyses, and adversarial collaborations to replicate empirical results. The inference that science needs stricter standards is supported by the pattern of failures rather than by randomized controlled trials (RCTs) testing the standards themselves.

Examples:

• “Why Most Published Research Findings Are False,” which argued that most published findings are likely false when studies are small, effect sizes are small, and many teams pursue the same question. This is the foundational text of the replication crisis literature.

• The Reproducibility Project: Psychology, which attempted to replicate 100 studies from three top psychology journals and found that only 36% of replications produced statistically significant results, even though 97% of the original studies had reported significant findings.

• The Many Labs project series, which has run a series of replications of classic and contemporary psychology findings with replication results varying widely (from 30% to 75%) across iterations (2014, 2018, 2016)

• Data Colada established the foundational case against “p-hacking” and has since surfaced evidence of data fabrication in high-profile research. P-hacking is manipulating data to artificially produce a statistically significant result.

• Registered Reports, a publishing format in which a study’s design and analysis plan undergo peer review and receive in-principle acceptance before any data are collected. This structure was designed to reduce publication bias and selective reporting of positive results.

Open Science

Hypothesis: A lack of transparency enables bad practices and slows progress. The case for openness is both that transparency is good in itself and that science advances faster when it’s open by default.

Theory of change: Increase transparency → increase accountability → faster, more reliable science.

Policy relevance: This perspective shows up in transparency requirements in agency grants and in open access mandates, such as the 2022 White House Office of Science and Technology Policy memorandum requiring publications (and underlying data) to be available without an embargo and Plan S, a mandate for immediate-publication access from a coalition of European research funders. It has produced practical infrastructure that many researchers now use regularly, such as preprints, preregistration, and registered reports. A representative policy question: should agencies mandate open data, preregistration, and open access for federally funded research?

Evidence and methods: Historically, their approach has been normative and values-based, with some empirical analysis. The movement’s strength comes from a combination of advocacy-driven policy reform, platform development, and community norm-setting, increasingly paired with empirical evaluation of outcomes.

Examples:

• The Open Science Framework, a free platform used by hundreds of thousands of researchers to host preregistrations, protocols, data, and study materials.

• The Public Library of Science (PLOS), founded in 2000 through an open letter signed by ~34,000 scientists who pledged to publish only in journals making articles freely available, and which has since grown into one of the largest open-access publishers in science and demonstrated that fully open-access publications are commercially viable.

• Preprint servers such as arXiv and bioRxiv, which allow researchers to share manuscripts before peer review and have dramatically accelerated dissemination in physics, mathematics, biology, and adjacent fields.

• Open data citation analyses, including a multivariate analysis that found papers making their data publicly available received more citations than comparable papers without publicly available data.

Subscribe now

Metrics of Science

Hypothesis: Science is a complex system with discoverable regularities. Large-scale data can reveal how breakthroughs happen, how careers unfold, and what predicts impact.

Theory of change: Understand patterns in how science works → more informed future program design → higher impact future research.

Policy relevance: This body of work provides evidence on optimal team size, career stage effects, and the novelty-versus-convention tradeoff. A representative policy question: should government programs fund more small teams, given that they are more likely to produce disruptive work than large teams?

Evidence and methods: Common methods are largely descriptive and include computational analysis of bibliometric data and machine learning on large scientific corpora. Some scholars track social dynamics (teams, citations, networks); others study materials used in research, arguing that new methods and tools are the primary engine of breakthroughs.

Examples:

• The “Science of Science” research program, which synthesized large-scale quantitative analyses of scientific careers, citation networks, team dynamics, and knowledge production, providing the canonical statement of the field’s research agenda and methodological toolkit.

• Disruption index research, which uses citation network patterns to distinguish papers and patents that consolidate fields from those that disrupt them and to measure whether disruptive work has been declining over recent decades.

• Hot streaks research, which identifies periods within scientific, artistic, and film careers where individuals produce their highest-impact works in concentrated bursts, with no accompanying change in productivity, often following a phase of exploration that transitions into focused work.

• Team size and innovation studies, which find that smaller teams are systematically more likely to produce disruptive work while larger teams develop and extend existing fields, suggesting that funding portfolios should support a diversity of team sizes.

• Atypical combinations research, which finds that the highest-impact papers tend to combine conventional foundations of prior knowledge with unconventional mixtures of ideas (e.g., drawing on research from two fields that rarely go together).

Innovation Economics

Hypothesis: Funding structures and incentive design causally shape the direction, quality, and quantity of science.

Theory of change: Understand incentives and causation →redesign programs → better science and translation.

Policy relevance: If a policymaker needs to know what might happen when you pull a specific lever, this is where to look. A representative policy question: do person-based grants produce more innovation than project-based grants and, if so, should more of the R&D portfolio be invested through person-based mechanisms?

Evidence and methods: Both causal and descriptive empirical methods are used, including RCTs, natural experiments, and other quasi-experimental analyses. This body of work relies on research designs that isolate cause and effect, which makes its evidence attractive for specific mechanism questions. It focuses on testing questions where credible counterfactuals exist.

Examples:

• Howard Hughes Medical Institute versus NIH comparisons, which find that scientists funded under HHMI’s “people, not projects” model, where grantees receive freedom to change direction and tolerance for early failures, produce more high-impact papers and explore more novel topics than comparable NIH-funded scientists.

• Intellectual Property (IP) protection and follow-on innovation studies, including evidence that a brief, non-patent form of IP held by the firm Celera over portions of the human genome reduced subsequent research and product development on those genes by 20–30%. Later research suggests that gene patents, by contrast, had little effect on follow-on innovation.

• Studies of expertise versus bias in peer review, including a quasi-experimental analysis of NIH grant evaluations suggesting that though reviewers are more biased about projects in their own subject area, the benefits of their expertise still weakly dominate the costs of bias. This implies that policies seeking to limit bias by using impartial evaluators may actually reduce decision quality.

• Small Business Innovation Research (SBIR) Program evidence, including a quasi-experimental study of Department of Energy SBIR applicants showing that an early-stage award roughly doubles the probability of subsequent venture capital investment and produces large positive effects on patenting and commercialization.

Subscribe now

R&D Management and Implementation

Hypothesis: Organizational design and management practices shape what science gets done. The same funding mechanism can succeed or fail depending on how the agency implements it. Beyond evidence about which mechanisms should work, you also need evidence about how to make those mechanisms succeed in practice.

Theory of change: Understand how R&D organizations work in practice→ design more practical reforms → more sustainable adoption.

Policy relevance: This approach answers the implementation questions generated by policy recommendations. A representative policy question: how should we design the organizational structure for a new agency metascience office, i.e. where should it sit, what authority does the director need, and how do we make sure it exists long enough to produce results?

Evidence and methods: Common methods include institutional analysis, comparative case studies, program evaluation, mixed-methods research, process tracing, and interviews with agency officials.

Examples:

• Comparative institutional analyses of NSF, NIH, DARPA, and ARPA-E, which document how organizational features such as program-director authority, hiring flexibility, and active project management distinguish ARPA-style agencies from traditional grantmakers.

• Studies of “open” versus “conventional” innovation procurement, including a quasi-experimental study comparing the outcomes of the awardees of Air Force SBIR “open” topics and conventional topics competitions. Winning an Open award increased subsequent non-SBIR Pentagon contracts, venture capital investment, and high-originality patents, whereas winning a conventional award increased the probability of winning a future SBIR contract.

• Analysis of ARPA-E’s active program management, where program directors retain authority to expand, contract, or cancel awards based on performance. Research found ARPA-E-funded cleantech startups filed patents at roughly twice the rate of comparable firms.

• NIH intramural technology transfer impact analysis, which traces how NIH-licensed inventions move from intramural labs to market. The study found that, in contrast to the simplistic linear “pipeline” model of innovation that agency policy often assumes, a chain-linked model in which research and product development continuously inform each other is a better fit to what NIH technology transfer offices actually do.

Institutional Entrepreneurs

Hypothesis: Existing institutions are too ossified to reform. The best path forward is to build new entities that demonstrate radically different models.

Theory of change: Generate hypotheses for new organizations → create existence proofs → competitive pressure and policy templates for incumbents.

Policy relevance: These experiments generate real-world evidence about what is possible. That evidence can be unusually persuasive to policymakers because it is demonstrated rather than hypothetical. A representative policy question: How much of a funder’s portfolio should fund new institutions?

Evidence and methods: Institutional entrepreneurs conduct real-world organizational experimentation, but these institutions are more proofs of concept than formalized experiments. The work resembles entrepreneurial iteration more than social science hypothesis testing. Its methods carry high persuasive power for demonstrating feasibility but make limited claims to generalizability.

Examples:

• “Reinventing Discovery: The New Era of Networked Science,” an early and influential argument that internet-enabled collaboration tools could fundamentally transform how scientific discovery happens, providing an intellectual foundation for many of the open and collaborative experiments that followed.

• Arc Institute, founded in 2021 with roughly $650 million in private funding, which provides investigators with eight years of unrestricted funding and lab support for up to 20 people in lieu of project-based grant applications.

• Convergent Research and the Focused Research Organization (FRO) model, founded in 2021, which structures mid-scale engineering projects as nonprofit, time-limited startups (typically $30–50 million over 5-6 years). FROs aim to build tools and datasets that fall in the gap between academic labs and industry.

• Arcadia Science, founded in 2021 as a for-profit research institute combining basic research, tool-building, and translational development under one roof, as well as experimenting with publishing outside traditional journals to release research products earlier and more openly.

Subscribe now

Charting the map

The camps in this landscape agree on a surprising amount! Federal R&D investment having a positive ROI overall is a longstanding consensus across metascience and science policy. In addition, metascience scholars and policy entrepreneurs largely agree that the current system of science funding is suboptimal, peer review can suppress risk-taking, administrative burden is excessive, career incentives distort behavior, and evidence should inform system improvements.

But those shared premises can lead to very different reform recommendations, depending on the theory of change, method, and intellectual tradition.5 This metascience landscape reflects the diverse priorities held by the metascience community. Some camps, particularly Innovation Economics, R&D Management, and Institutional Entrepreneurs, converge on a similar diagnosis of the funding system’s biggest problems. Others approach the same enterprise from different angles: Research Integrity worries about the reliability of published findings; Open Science prioritizes transparency. This intellectual diversity is healthy and is a sign that the field of metascience is maturing.

Why a map matters

We are in a critical window for R&D reform. The Trump administration’s proposed funding changes and reorganizations to federal science agencies create both pressure and opportunity to revisit the structure of federally funded research programs. Multiple kinds of metascience evidence can inform that work, and policymakers are recognizing it — expressing demand for economic evidence to drive system reform, R&D management knowledge to ensure reforms actually work, and examples of new institutions they can use in legislation and program design.

NSF’s recent launch of X-Labs is an indication of that appetite. X-labs will award institutional block funding to independent organizations with operational autonomy, with milestone-based awards to encourage team-based, infrastructure-heavy, and long-horizon approaches to research. This program structure was informed by metascience and is a new experiment in portfolio diversification.

We are also excited to see the evidence-generating and evidence-translating communities depicted in this map continue to grow. We believe the most durable reforms tend to draw from several perspectives at once. For example, a reform inspired by new institution existence proofs (institutional entrepreneurs), backed by causal evidence (innovation economics), embedded in an organizationally realistic design (R&D and innovation management) is more likely to survive first contact with reality than one informed by any single tradition alone. Building more connective tissue between metascience communities would help ensure that our perspectives and methods do not stay siloed and that our reforms are more robust.

Subscribe now

Building the connective tissue

Some funders have already prioritized evidence-building into their research programs, but this remains all too rare. Even the Congressional mandate to have an evidence-to-practice pipeline at Federal Agencies, including research agencies, under the Evidence Act did not produce a swell of metascience evidence.6

One of us (Jenn) experienced the dynamic of adapting evidence to program design firsthand while leading early-stage innovation programs at NASA. For program managers in the federal government, the demands of satisfying institutional requirements and culture while confronting resource limitations (people, IT, etc.) can make implementing changes to program design feel like hand to hand combat at best and insurmountable at worst. It helps when there is external evidence to point to, but that is often insufficient when proposing changes to institutional norms could introduce protest and oversight risk, create confusion for proposers and reviewers, and not pay off for many years.

In short, even if they want to, most program managers don’t have the time or the institutional cover to implement a routine evidence-to-practice pipeline that could in turn provide real-world experimental evidence.

Further, program managers often have their own hypotheses about program design and impact they’d love to evaluate. But they either are unaware of the metascience community eager to study these same questions or face significant barriers to working with the researchers (data access restrictions, timing of research findings not aligning with program reform windows, etc.).

Our map of the metascience landscape can help agencies recognize that they don’t need to embark on evidence production alone. For example, questions they’re already required to ask under the Evidence Act have intellectual homes — communities with methods, scholars, and accumulated evidence that could dramatically improve the quality of their answers.

At the same time, better coordination with the external metascience community cannot fully substitute for internal metascience capacity. Agencies’ experience with the last six years of learning agendas and evaluation plans developed under the Evidence Act demonstrates this: even when agencies have identified the right questions, they have lacked the in-house staff with metascience expertise to design and run high-quality evaluations, and the most policy-relevant data (proposals, reviewer scores, panel deliberations, and selection decisions) often cannot be shared with outside researchers for confidentiality and statutory reasons.

We believe that an important first step to drive more metascience understanding and improve the social ROI on R&D is to institutionalize metascience capacity inside the government through dedicated metascience units. In 2024, the UK established a Metascience Unit and the Administration’s FY 2027 budget request included a proposed Metascience Unit at NSF, which is a good start. These units should draw on research questions, methods and external expertise from across the metascience landscape.

Policymakers are hungry for evidence-informed ideas and answers. If we applied even a fraction of the funding we deploy for object-level science to studying what works and implementing those practices at scale, the payoffs could be enormous. Science needs more evidence about itself, and there is a broad metascience landscape poised and ready to help.

Subscribe now

As I said to the congressional staffers, I think there are meaningful differences between new scientific institutions, such as Focused Research Organizations and X-Labs, and the National Labs. That the Labs are massive, old, and federally-owned really is important when comparing them to institutions that are small, new, and free from regulatory and bureaucratic buildup. And it’s good to try new things; when the National Labs were set up in the middle of the 20th Century, they too were new scientific institutions.

I hope that some of the proponents of new institutions referenced in this piece (Caleb Watney and Ben Reinhardt) and anyone else with a take or a stake will respond. What can new institutions learn from the Labs? What can the Labs learn from new institutions? And how can metascience better integrate the Labs into our collective mental model of science?

Disclaimer from the author: This piece represents my personal opinions, and not the opinions of either of my employers.

Metascience proponents have produced a flurry of think pieces arguing that new institutions are needed to accelerate innovation in R&D. They contend that academia, government, and industry are “ill-equipped to solve” important challenges because they “don’t provide the engineering teamwork needed to produce the platforms, datasets, or tools” required.

I agree with metascience’s emphasis on continuously reevaluating whether we are maximizing scientific progress. But many of these think pieces fall prey to a common pitfall in academic R&D: novelty bias. While it’s understandable to be excited about novelty, this tendency can lead to reinventing the wheel, where we ultimately recognize we had the tools we needed all along.

I hesitate to pick on my hosts, but the X-Labs described by Caleb Watney sound a bit like the National Labs. To quote Watney, “X-Labs would fill a longstanding structural blind spot in the US research ecosystem: work that is infrastructure-intensive, team-based, exploratory, or oriented around critical bottlenecks.” They “would provide stable, full-time roles for staff scientists, engineers, and technicians, enabling institutional memory and technical depth more typical of industrial R&D.” Sounds familiar.

Though they differ in size, nimbleness, and specificity of mission, Focused Research Organizations (FROs) are pitched with similar features and an added dose of Bell Labs nostalgia.1 While we might want to recreate the creative spirit of Bell Labs in new institutions like FROs, the infrastructure and personnel structure actually survives within the National Labs.

Does the call for new institutions describe what the National Laboratories already do? As I discuss in this article, the answer is that sometimes metascience writers are describing what the Labs already do and sometimes they’re describing what the Labs aspire to (but don’t always achieve). It’s worth revisiting the role of the National Labs. The metascience discourse neglects them in favor of new models for understandable reasons, but we risk leaving massive value on the table if we fail to make National Labs a central component of efforts to accelerate progress in R&D.

Subscribe now

What are National Labs, exactly?

DOE National Labs are part of the broader category of Federally Funded Research and Development Centers (FFRDCs).2 Their mission is both to advance fundamental science and to occupy a missing middle: research that is too complex for a single principal investigator grant, but too risky or oriented on public goods for a startup or a large industrial incumbent to take on. National Labs shine precisely when the scale of a challenge demands approaches that are infrastructure-intensive, team-based, and motivated by critical bottlenecks, to borrow from Watney.

National Labs are a significant portion of the federal R&D budget. As of fiscal year 2024, the government invested over $31 billion in R&D at all FFRDCs, and DOE National Labs are a substantial portion of that investment. DOE contributes nearly $18 billion to the Labs. This figure is significantly larger than the budgets of other R&D funders, such as the intramural investments by the National Institutes of Health (~$5 billion) or the entire National Science Foundation (~$9 billion). Granted, ~40% of DOE’s overall spending is in the National Nuclear Security Administration (NNSA), whose programs in national security and non-proliferation makes the separation between “fundamental science” and “weapons maintenance” fuzzy. But NNSA directly manages three of the largest National Labs (Lawrence Livermore, Los Alamos, and Sandia), which house some of the flagship large R&D infrastructure and carry out plenty of fundamental science, so they cannot be dismissed simply because of their nuclear weapons maintenance function.

By design, National Labs house a lot of infrastructure. It is economically inefficient for every university or firm to have a particle accelerator, so that sort of equipment makes sense as a long-term federal investment. This practice holds true across fundamental–to-applied capabilities and scale, ranging from massive facilities to decentralized advanced instrumentation scattered across an institution’s buildings and campuses. The best-known examples of the former are the various X-ray light sources (synchrotrons) or the exascale computing facilities. On the NNSA side, they include the National Ignition Facility, which is important given renewed federal interest in fusion energy.

On the applied end of the basic-to-applied research spectrum, we find examples like facilities to accelerate manufacturing scale-up and nuclear test reactors. Many of these facilities have a mission to attract external users, which allows tens of thousands of scientists outside the National Laboratory complex to use them for their own research.3

Subscribe now

Complex, multi-year engineering loops require experienced, permanent staff who are not splitting time with coursework or dissertations. National Labs are explicitly built and staffed for this kind of long-term team science. Their emphasis on problems that span from basic to applied forces cross-disciplinary collaboration and removes silos that are typical of academic institutional structures. This structure builds multidisciplinary density. It co-locates, say, domain scientists, computational experts, and process engineers and organizes them into a stable, long-term team to solve large challenges. Staff permanence reduces loss of tacit knowledge inherent to the passing of generations of students.

While universities have departments that cover a vast constellation of topics, National Labs are more narrowly focused on energy technology (albeit broadly defined). And while universities have a dual mission of education and research, National Labs are primarily focused on research. As a result, universities have a high proportion of graduate students, who are on a roughly five-year clock, compared to the primacy of the permanent (not to be confused with tenured) staff scientist model of National Labs.

What does this structural design buy?

The structure of National Labs allows them to perform many of the functions associated with new institutions for science: taking on ambitious, focused initiatives; collaborating on pre-commercial technology; and crafting the future of AI for R&D.

National Labs are arguably the original FROs. There are many examples of “time-bound, technically ambitious efforts” at National Labs designed to unblock whole domains of technical progress (the Manhattan Project might be the most famous one). And although it’s not their primary focus, the National Labs played important roles in biomedical research: the Labs were crucial to the Human Genome Project, for instance, by hosting unique, large omics facilities that were accessible to users nationwide. In addition, the Protein Data Bank, the foundation on which AlphaFold was built, was founded at Brookhaven National Lab and contains data measured at the US synchrotrons.

Subscribe now

National Labs also offer an alternative to academia for scientists who need access to “equipment, skills or status” to advance their pre-commercial idea. In “Unbundling the University,” Ben Reinhardt writes that “it’s very hard for individuals and organizations to avoid interfacing with academia if they have an ambitious pre-commercial technology idea.” They can interface with the National Labs instead! Universities largely focus on basic generalizable principles and industry focuses on commercial deployment. National Labs are uniquely positioned to bridge this gap because they house both basic and applied R&D programs and can facilitate collaboration on pre-competitive research between academia and industry. Some of the applied programs at National Labs are explicitly tasked with taking a concept derived from a lab and piloting it, including through partnerships with industry.

The National Lab model is also uniquely suited to using AI and autonomous experimentation to accelerate science. Many of these facilities are centralized and operate year-round, so they can generate the massive, standardized datasets required to train foundational models, something a decentralized network of university labs cannot do. Moreover, the National Labs can host the repositories needed to curate large sets and make them widely accessible.They are also good hosts for large cloud labs that require millions of dollars in investment and upkeep. It is much harder for universities to serve this role because you cannot build a self-driving lab or a long-term AI-supported research loop if context is lost every time a student graduates. Institutions with staff permanence hold the substantial tacit knowledge needed for initiatives like the Genesis Mission, so it makes sense that the National Labs are central to their implementation.

So are National Labs the answer?

Although National Labs have many of the capabilities and staff needed for, say, an X-Lab, I must partially agree with the metascience writers: while the Labs can play the role that FROs and X-Labs are intended to, they often serve different functions and therefore require distinct strategies to maximize our return on R&D investment.

More specifically, the institutional dynamics of National Labs deviate from the model of X-Labs and FROs in four ways:

1. The current National Lab model may lead to mimicking the academic approach to research, structurally and culturally, rather than fulfilling the goal of complementing and augmenting academia. There is a tension between serving users at a large-scale facility carrying out their own R&D mission, competing for broad-based grant calls, and seeking scientific publications.4

2. Access to large facilities at National Labs is not as smooth as many in the private sector would want, and it can be expensive. But there are ideas to address barriers to using facilities.

3. Intellectual property agreements and security clearances create friction points to starting close partnerships with industry. But the Labs remain the only entity with the scale and neutrality to play this role, making the bureaucratic hurdles worth navigating.5

4. National Labs are subject to high overhead costs, which inspire questions about whether they are the most efficient use of tax dollars to achieve a desired R&D outcome.

Some of these issues are a result of institutional design. After all, DOE is a government agency that also uses the National Labs to manage nuclear weapons. It would be tough to get a National Lab to operate with the nimbleness and scrappiness of a start-up. Nevertheless, the Labs really do have many of the characteristics that proponents of new institutions say we need, and they have achieved impressive scientific feats. National Labs can be optimized, and metascience analysis and reform should consider how to refresh these institutions, rather than only angling to build new ones.

Subscribe now

Direct metascience analysis at the National Labs

For infrastructure-intensive, engineering-heavy, decadal science, there is no substitute for the National Labs. No FRO is ever going to build a synchrotron or anything like the portfolio of delocalized, cutting-edge instrumentation that a National Lab offers. Metascience’s reform portfolio misses an opportunity to put the impressive National Labs infrastructure to better use. Rather than ignoring the Labs, we should use metascience approaches to investigate the following questions (to start):

1. How can we further activate the significant potential of the National Labs? What more can we get out of the existing ~$30 billion per year National Lab network? We need a better answer than just “National Labs are too slow.” How do we make them fast(er)?

2. What falls outside the National Labs jurisdiction and institutional design? Those should be the target for new institutions (X-Labs, FROs, BBNs…).

3. How do university and National Labs’ rewards for performance differ? Are the incentive structures at National Labs sufficiently different from universities to meet their distinct roles in the R&D landscape?

Attention to these questions will allow us to more tactically target interventions to fill the gaps that National Labs fundamentally cannot close. The new institutions envisioned by metascientists may differ from National Labs in some aspects, but National Labs certainly belong in the toolkit. It’s time to consider how to use them better.

Subscribe now

I enjoy when Substackers share what they’ve been reading (Brian Potter and Lauren Gilbert publish roundups I like); it’s a good way to find pieces I might have missed and gives me a glimpse at what other authors are thinking about. Now I’m paying it forward.

Here are four articles and one new website that have influenced my thinking recently. These pieces span a few questions I’ve been considering: how science is globalizing, how it’s shifting with new technology, and how we can better measure its benefits.

1. The Pop-Up Journal Initiative

The Alfred P. Sloan Foundation and Coefficient Giving launched this initiative to “curate and synthesize evidence around specific, policy-relevant questions” with the goal of delivering actionable evidence to decision-makers. This is intended to be a series of journals, and the topic of their first — what is the social return on R&D? — is neglected. Our knowledge about the returns to R&D has grown in recent decades, but we still need more granular evidence of what is driving returns, and in which sectors and areas of the world.

I’m intrigued by the project’s format. Like focused research organizations (FROs) and Renaissance Philanthropy’s time-bound funds, this pop-up journal will dedicate bounded time and funding to producing policy-relevant evidence. By providing new research funds, Sloan and Coefficient Giving will grow the community of researchers studying the returns to R&D (and those researchers will likely continue on that work after the journal, uh, pops down).

The journal is accepting applications for research grants through April 30, 2026 (I realize the deadline is very soon — but perhaps you have a good proposal lying around or can work quickly). They’ll provide $250,000 for studies that will “provide key empirical insight into the social and economic returns to R&D investment.” Larger requests may be considered for uniquely ambitious projects. Apply here.

Subscribe now

2. “Are we ready for a multipolar world of research?” (Carlos H Brito Cruz in the London School of Economics Impact blog)

Brito Cruz points out a blindspot in the debate on whether the US or China is the most scientifically advanced country: in the meanwhile, low- and middle-income countries (LMICs) have been building their scientific capabilities. For the first time ever, the majority of authors in the SCOPUS Abstract and Citation database are from LMICs. In my view, this piece is a story about (1) China, of course (still technically middle-income), but also (2) developing countries investing in science, and (3) technology — from Zoom calls with co-authors to Google Translate — making it easier than ever for LMIC researchers to publish.