Maybe this isn’t a surprising sentence to see at the top of a Macroscience article. But once governments decide to, how should they actually go about incentivizing innovation? This week’s Economics of Ideas, Science, and Innovation lecture is about the policy levers that governments have available to them.

Society gets three times more return on R&D than a firm is able to capture. As Flaubert put it, inventors “all die in the poorhouse. Someone else profits from their discoveries.” Or, as another Economics of Ideas, Science, and Innovation lecturer Matt Clancy stated in the title of an excellent blog post, knowledge spillovers are a big deal.

John Van Reenen, the Ronald Coase Chair in Economics at the London School of Economics, is one of the authors of the legendary (at least in my circles) lightbulb table that grades innovation policies on the evidence behind them, the payoff they have, and how fast they work.

Subscribe now

In covering the different policy angles for R&D, John brings us back to the economics underlying it all: how to get more effort into innovation, and how the previous lectures connect in the policy arena. From the supply of researchers, to who becomes an inventor, to tax credits that shape what research firms do, policy often sets the terms.

Earlier this year IFP released the Atlas of Innovation, which walks through a selected set of policy tools, and in this lecture, John goes deep on some of the same ones and many more.

Which levers actually work? Working at an innovation policy think tank, I ask myself this question nearly all day, every day, and I feel lucky we have the expert to tell us more.

Subscribe now

Here are the readings that go with this lecture:

• Akcigit, Ufuk, and Stefanie Stantcheva. “Taxation and Innovation: What Do We Know?” In Innovation and Public Policy, edited by Benjamin Jones and Austan Goolsbee, Chicago: University of Chicago Press (2022).

• Bloom, Nicholas, Mark Schankerman, and John Van Reenen. “Identifying Technology Spillovers and Product Market Rivalry.” Econometrica 81, no. 4 (2013): 1347–1393.

• Bloom, Nicholas, John Van Reenen, and Heidi Williams. “A Toolkit of Policies to Promote Innovation.” Journal of Economic Perspectives 33, no. 3 (2019): 163–184.

• Dechezleprêtre, Antoine, Elias Einiö, Ralf Martin, Kieu-Trang Nguyen, and John Van Reenen. “Do Tax Incentives Increase Firm Innovation? An RD Design for R&D, Patents, and Spillovers.” American Economic Journal: Economic Policy 15, no. 4 (2023): 486–521.

• Kerr, Sari Pekkala, and William R. Kerr. “Immigration Policy Levers for US Innovation and Start-Ups.” In Innovation and Public Policy, edited by Benjamin Jones and Austan Goolsbee, Chicago: University of Chicago Press (2022).

  Subscribe now

Thanks to William Higbie and Harry Fletcher-Wood for their support in producing this video and the transcript.

Lecture Transcript:

Innovation policy

I’m really pleased to be able to talk about innovation policy, because I think it’s one of the most exciting areas of innovation. I’m going to start off by asking why not do what we do with most markets and leave it up to the private sector? Then I’m going to get on to the meat of the lecture: assuming the government should intervene, how should it stimulate innovation?

Heidi Williams, Nick Bloom, and I have tried to frame the whole thing around ways government could intervene. We put them broadly into two buckets:

• Demand-side buckets: things which increase firms’ demand for innovation;

• Supply side: things which can stimulate the supply of inventors and innovation in the economy.

On the demand side there are direct subsidies, R&D grants, and subsidies. I’m going to focus more on indirect approaches, in particular talking about research and development tax credits and the way the tax system could influence innovation. On the supply side, I’m going to focus on what Raj Chetty, myself, and others have called the “lost Einstein” effect, which is how you could potentially create more inventors by exposing people to the possibility of becoming an inventor at a younger age.

Why should the government subsidize innovation?

Why should the government subsidize innovation, if at all? There are many potential market failures which could give reasons for governments getting into the innovation space, but the main one is the knowledge externality, arising from the fact that research and development is at least partially non-excludable. There’s a public-good nature of knowledge, which means that those who do the research and development only get a relatively small part of the social benefit; it spills over to many other agents in the economy. Therefore, the incentives a researcher has are much lower than the benefit that society gets.

Ideas and innovation are hard to hold. You try to hold them, they slip through your fingers. Knowledge builds on the shoulders of giants, and this creates spillover effects — this non-excludability reason for subsidizing R&D. This was best encapsulated by this quote from Gustave Flaubert in 1911, in his ‘Dictionary of Received Ideas’, where he defined inventors. They, “All die in the poor house. Someone else profits from their discoveries, it’s not fair.” That more or less summarizes this idea of the benefits from innovation being much wider than what accrues to the firms or inventors who put in the research effort.

That’s the primary market failure. But there are many other potential market failures, and broadly you could think of these as frictions in other markets. A particular market failure people have emphasized is in the financial markets. If you think about research and development, there’s an upfront research cost. It can often be large and uncertain; there’s asymmetric information. By definition, the person who’s doing the research knows a lot more about it than the potential investor who’s providing the research finance. The potential investor is always worried, “How do I know, when you’re spinning me the story about how great this invention is going to be, that it really is going to be that great?” On the inventor’s side, they’re worried, “If I tell an outside investor what I’m going to do with it, they may take the idea and run away with it.”

For a whole variety of reasons, from at least Arrow onwards, we think that there may be frictions in financial markets. We know there are in general, but it may be particularly a problem for an intangible investment like research and development, and this will mean that we tend to get too little R&D finance. This may be a particular problem for startups and small and medium-sized enterprises.

You could broadly think about the spillover effects being multiple.

I’ve emphasized the positive effects.

• Spillovers can occur because others can copy the inventions of a firm — that’s the Flaubert quote.

• There could be these intertemporal benefits: even if you have a patent which gives you some intellectual property protection, future inventors can build on your shoulders using the ideas from previous generations. Those benefits are not fully captured by the person who has the intellectual property. That’s a similar kind of spillover benefit.

• There’s also surplus captured by consumers and downstream firms, which won’t necessarily be taken into account when the firm is doing its research and development. That will all tend to be on the side of positive spillover — so too little investment in R&D.

On the other side of the coin, there can be some negative spillovers.

• In the literature we think of this as a business-stealing effect. This is where I get an idea, and it might be quite incremental or small. If the market structure is such that a very small difference in our cost or quality means I get the entire market, then I may have almost too much incentive, from the social point of view, to do invention. A classic example of that is when we think about pharmaceuticals. There are things called “me-too” drugs, which may only have a small therapeutic value over the previous drugs, but because typically insurers and other payers will often reimburse the best drug, even if your drug has a small advantage you get a huge amount of the market. Like an arms race, it can give you too much incentive to do R&D. This is the destructive part of creative destruction.

• There could be duplicative R&D — we’re both doing the same type of research and development, and it’s a waste of resources.

• There could be intertemporal costs: ideas could be fished out if there were a finite number of ideas. The more I take, the less is left for you.

So there are potential negative spillovers as well as positive spillovers. Which dominates is going to be an empirical issue, and it may be different in different contexts and industries.

Although theoretically these spillovers can be positive or negative, the empirical evidence strongly suggests that the positive effects dominate. A few of the papers here have tried to look at the net effect of these different types of spillovers.

The conclusion they generally come to is that the social returns to research and development are way larger than the private returns. A factor of three or more is a typical number. That implies there’s a lot of underinvestment in the private sector in innovation.

That’s the fundamental justification for why we might want the government to intervene.

There is a nuance to this, though. For a smaller country in particular you might say, “It’s great that we have more innovation in the world, but I’d much rather that the taxpayers of another country pay for it rather than I pay for it. So by all means, subsidize research and development in your country, create these ideas, so I get the benefits without having to pay the financial costs of doing the R&D.”

For a big country like the US, the European Union, or China, this is less important. It’s also much harder for advanced countries, which are near the technological frontier. For a developing country it may make a lot of sense to enjoy the benefits of the innovation happening at the frontier. The issue is — and this is the development process — as countries get closer to the frontier they may have to switch their model of development from the diffusion and imitation model toward more innovation. Japan would be an example of that, and now China might be another.

The third reason is that when we think about research and development, people sometimes say it has two faces. We often think of it as the innovation part: if I do more research, I can push the technological frontier. But there’s also some evidence that by doing research and development — by having engineers and scientists who can understand innovation at the frontier — this might help you catch up to the frontier as well. This is sometimes called “absorptive capacity.” To the extent that’s important, this is another reason why a country may have incentives to build up its R&D even if its neighbors are not paying as much as it is. There are answers to that policy challenge.

A toolkit of innovation policies

I mentioned this paper in the Journal of Economic Perspectives that Heidi Williams, Nick Bloom and I put together, where we try to summarize the existing evidence, as it was in 2019, on different policies. We divided things up into the type of policies, the quality of the evidence, and how conclusive it was. We did a rough benefit-cost calculation.

I’ve just come out of spending almost two years chairing the Council of Economic Advisers in the UK. If you talk to policy makers about what type of innovation policy you should be putting into place, they’re very alert to how long it’s going to take. Of course, innovation policy takes longer to have an effect on growth than a policy toward unemployment insurance or something which is more near-term. But within the class of innovation policy, some things have a larger effect in the shorter term — say over the life of a parliament or a president — than others do. From a political point of view, that is something which is very front of mind. The other thing that policy makers are often very concerned about is the effect on inequality — spatially and across people. That’s also something we discussed a bit.

The kind of policies that we looked at are:

• Direct R&D grants;

• R&D tax credits, which I’ll talk about in some detail;

• Patents and the patent box.

The reason I’m going to focus on the R&D tax credits is that there’s a lot of evidence that the benefit-cost ratios could be relatively high, and that they can affect over a relatively short run, like four or five years, rather than over a longer run like 20-40.

These are all demand-side policies: by trying to subsidize firms’ research and development activities, you can stimulate those firms to do more R&D. As I mentioned, there’s another group of policies on the supply side, such as:

• Encouraging skilled immigration;

• Changing incentives or the resources in universities

• Exposure policies — exposing people to the possibility of becoming inventors at an early age, and;

• Trade and competition.

Many of these also have some evidence, especially skilled immigration. Things like the exposure policies I’m going to talk about — there’s less evidence, and the effects are more in the long run. But the reason why supply-side policies are quite attractive is that demand policies stimulate demand, but if you don’t have an elastic supply side you’ll end up driving up the prices — the cost of doing research. Since most of the cost of research is people, the wages of scientists and engineers may go up a lot when you stimulate demand. That’s good for us as researchers, but not necessarily good for society and the taxpayer, if you’re driving up the price rather than increasing the quantity, which is ultimately what we want.

Taxation and innovation

This has been talked about for a long time. Joseph Schumpeter said, “On the one hand, taxation is an essential attribute of commercial society… on the other hand, it is almost inevitably… an injury to the productive process.” Schumpeter was a bit skeptical about the benefits of tax.

This wasn’t his most famous quote, of course. That was probably, “Early in life I had three ambitions. I wanted to be the greatest economist in the world” — very modest guy — “the best horseman in all of Austria, and the greatest lover in all of Vienna.” “Those who knew Schumpeter as an economist, lover or horseman” — this is a quote from an unnamed source — “presumed his skills were in the other two fields.” It’s probably a nasty referee of one of his papers.

When we’re thinking about tax and innovation, it’s a very broad question, so we’re going to start off relatively narrowly, with specific taxes around innovation.

The primary example of this is tax credits for research and development. But we could also have what’s sometimes called patent and innovation boxes, giving incentives for intellectual property. We’ll focus on the R&D tax credit. Then we’ll move on to a wider question, which is the impact of more general personal and corporate tax systems on innovation as a whole, and we’ll draw on this recent paper by Akcigit, Grigsby, Nicholas and Stantcheva.

There’s an even wider way you could see this. You could think of many policies as implicit taxes or subsidies on innovation incentives. If you saw some regulations as being like an implicit tax — some employment regulations start when firms grow beyond a certain size, and they’re like a tax on size. These size-dependent regulations can also affect your incentives to innovate. If you know that if you innovate and get very big you’d have to pay a higher tax because the regulatory burden was so high — it’s not an explicit tax, it’s an implicit tax, but this could have an effect on your incentives to innovate. We have some evidence on that in a couple of papers.

R&D tax credits

Ronald Reagan introduced the first US R&D tax credit. They’re a way of supporting R&D through the tax code. The basic idea is to change the tax system to make R&D more attractive than other kinds of spending. It’s increasingly popular all over the world. If you look over time, there’s been a shift from direct support — giving out direct R&D grants to firms — to indirect support by changing the tax system. This OECD report I looked at from last year showed that 34 out of 38 countries granted tax relief for R&D — that’s risen from 20 in 2000.

What’s the advantage of an R&D tax credit, as opposed to choosing which firms to directly give subsidies to through a grant system?

1. Because it’s performed in the private sector, it’s probably managed more efficiently than if it was done in a government laboratory.

2. Informational: if you set the parameters of the tax system, there’s no need for a government to explicitly choose projects and have to say, “I want to subsidize this particular new fighter jet, or chemical, or pharmaceutical.” It can say to the private sector, “You choose what you want to do,” and that economizes a lot on bureaucracy, and also the government’s information — the private sector probably has more information.

3. It mitigates the risk of political capture by a single firm or industry. The move away from direct subsidies being given to firms is that if firms know this, they start investing in lobbying and trying to get more money out of governments. By setting the parameters through the tax system, in principle that’s a much more neutral way, and is less subject to single-firm or industry lobbying.

The disadvantages:

1. It’s a pretty blunt instrument. If the main market failure is the knowledge spillover, what you’d want to do is to target your marginal research dollar on high-externality R&D, like basic research at universities. Firms left on their own devices are more likely to do activities in R&D which are nearer to the market — that’s where they’re going to get the biggest returns. If there were much lower or no externalities there, that’s going to be a much less attractive thing to do from the government’s point of view.

2. Firms might relabel existing activities to obtain the tax break. You can imagine that firms might want to say, “The money I’m spending on McKinsey consultants — this is actually research and development.” In order to get that tax break I can relabel activities as R&D to get the advantage of that tax break. They have some incentives to do that anyway, but the incentive might be greater the more generous the tax credit is.

3. The tax credit is like a reduction of your corporate income tax bill. But if you’re a startup or an SME, you make very little profit. You may be making losses. Having a reduction of your corporation tax doesn’t benefit you very much, because you’re not paying any tax to begin with. You can potentially overcome that. You can say, “Even if I’m not making profits today, I can build up the credits to use them in the future” — carry-forward provisions. Some of the tax credit systems say that if you’re not making profits you can still get some of the money. So there may be ways to overcome that, but it’s still an issue, because it means that the incentives are less the more you discount the future.

4. There are lots of perverse incentives it can create, depending on the particular design it has. There’s a lot of work on that.

5. As with any demand-side policy, you’re subsidizing R&D to increase the demand; if the supply side is not strong, you could end up increasing the price of R&D.

6. If research and development is defined quite narrowly in order to avoid the relabeling, it means you’re missing out on some of the potential innovative activity; and if it’s defined very widely, you might end up subsidizing lots of activity that would have already taken place — it’s not necessarily targeting the marginal investments.

What does the empirical evidence suggest?

There are two types of things that people have looked at. One is: do fiscal incentives increase R&D? The broad answer is yes, and we’ll go through some of the studies. A typical thing people will look at is the elasticity of R&D with respect to the tax-adjusted user cost: if you halve the user cost, how much do you increase research and development? Most estimates place that elasticity at least one, and usually greater than one. A 10% reduction of the cost of R&D through the tax system increases the amount of R&D by at least 10%, and usually more. There are even meta studies of this. The micro studies tend to find larger elasticities than the macro studies, and that is because, as this paper by Appelt and co-authors argues, take-up is much less than you might think by some of the firms who could have taken out these tax credits. But that doesn’t necessarily suggest the macro effect isn’t big; it’s just that not all firms necessarily take up the tax credit when it’s offered.

A second group of questions is whether fiscal incentives increase innovation. This is important because if there’s this concern that some activities get relabeled, then maybe what looks like research and development goes up but it doesn’t mean that innovation increases. This is a question which hasn’t been looked at as much as it should have been, because ultimately we don’t care directly about R&D; we care about the innovation growth and the welfare it creates.

Identifying the effect of R&D tax credits

What are the designs that people have used to try and look at the impact of R&D tax credits? I mention four designs here.

Federal tax credit heterogeneity in firm-level user cost

The first wave of studies — this is a bit US focused, but this is the same method people use in lots of different countries. The federal tax credit in the US generates a lot of heterogeneity in the firm-level user cost across different types of firms. You can use some of this firm-level heterogeneity. For example, in the US tax credit when it started, it looked at your past levels of R&D and then said, for your R&D spend this year, how much bigger is your R&D than that historical base? You got the tax credit based on that incremental amount. People could use the fact that different firms had different histories of R&D, and that would mean that the generosity of the tax credit depended on your history.

Another form of heterogeneity comes from the fact that if you’re a firm which is not making profits, you’re going to have less of an incentive, because you can’t write off your profits against the benefits of the tax credit system. Although the federal tax credit rules are the same across the US, that still gives you a lot of variation across different firms which you can use to try and identify the effect of the tax credit.

Bronwyn Hall had this early study where she used that variation and used standard econometric techniques — panel data techniques, using lags as instruments — to try and look at the impact of the tax credit on outcomes. That variation I’ve described is useful, but it’s not exogenous variation. Whether you’re earning profits, what your history of R&D was, isn’t solely exogenous — it can also react to expectations. For that she tried to come up with instrumental variables. These instrumental variables use the lagged values of some of these variables, and they suffer some problems: you have weak instrument problems, especially if you have serial correlation.

More recent work, which has used administrative data from the IRS which has actual tax credit receipts, has typically used synthetic instruments approaches. The classic paper here is by Gruber and Saez. The idea here is that you say, “Let’s hold firm characteristics fixed at their lagged values.” Firms will have different values of things like profits or R&D, and then you say, “Given those characteristics, how much does the generosity of the user cost change because of the changes of the tax rules on those individual firms?” You’re not using the actual changes of the firm’s tax liabilities, or the R&D, or anything else — endogenous components. You’re fixing that in the pre-policy period and then using the policy change to generate what the user cost would have been had you not changed any of your other choice variables. That is a shift-share analysis, if you like — a kind of Bartik instrument, if you know that — but in a very disciplined way. With a tax simulation model you can calculate for each firm what its tax rate would have been, given its prior characteristics, coming from the change of the policy. Some firms will be more affected than others, depending on whether they’re lucky enough to have characteristics which are conducive to the generosity of the tax credit system. That was Rao’s approach. Both of these papers found long-run elasticities well above one — relatively sensitive, and definitely positive, responses to the tax changes. That is one classic approach to try to identify the effects.

Cross-country variation in R&D tax credits

Because of these worries about endogeneity at the firm level, some approaches have taken cross-country variation. I did an early study with Nick Bloom and Rachel Griffith. There we used cross-country variation over time — firm country panels — and we again found elasticities about one. So smaller than some of the micro elasticities, but still positive. We did find that in the short run the effect was much smaller than it was in the long run. There are quite long adjustment dynamics for firms to adjust to the changes of tax credits.

State-specific tax credits

Wilson has a paper using different tax credits across different states. That’s a nice source of variation. You’re staying within one country, institutions are the same, but you’re using the fact that California may introduce a tax credit whereas Nevada doesn’t. Then you can use that variation to look at the impact. He emphasizes in his paper that there was a lot of cross-state relocation as a result of the state-level R&D tax credits. When California made its system more generous, it saw its R&D go up, but a lot of that was because firms were moving from, say, Nevada to take advantage of the tax credit. That may have been good for California, but it wasn’t so obviously such a good thing for the US as a whole.

Using non-linear design of tax credits to generate regression discontinuity designs

Some of the more modern stuff uses non-linear designs of the R&D tax credit system, using regression discontinuity designs (RDD). This Dechezleprêtre paper uses administrative tax data and firm accounts in the UK to look at the impact of R&D tax credits. It looks not just at R&D but also at patenting, jobs and productivity. It exploits a discontinuity in the generosity of the R&D tax relief at a particular threshold. In 2008, the government changed the R&D tax credit system, making it more generous for small and medium-sized enterprises based on, for example, their assets. You can use the discontinuity around that threshold to do an RDD. In doing that we found there were positive effects both on the subsidized firm itself, in terms of its research and development and its innovation, but also there were technological spillovers to other firms. This is an “RDD for R&D,” which Josh Angrist particularly likes, because it sounds good: an RDD for R&D.

This was the threshold.

Bigger firms, as measured by assets, do more research and development. But you can see that around this threshold, after the policy was introduced, firms which were just to the left of the threshold, which got the more generous tax credit, did a lot more R&D than firms to the right of the threshold, which had a less generous tax credit. This looked very much like there was a large and significant effect of the policy. You didn’t see this prior to the policy coming in. Firms seem to react.

Similarly, if you look at outcomes like patents, you see these effects.

Those took a few years to come through — there’s a lag — but there appeared to be both an effect on R&D and on innovation as measured by patenting. R&D more or less doubled as a result of this; quality of the patents increased by about 60%. These are larger than some of the past literature, and the argument the paper makes is that this is likely because these treated firms are smaller than most of the existing literature, and therefore more likely to be financially constrained.

You can also try and look at technology spillovers. The idea here is that within a technology class, the greater generosity that the smaller firms get spilled over to some of the larger firms, using this neighbor design.

Does general taxation affect innovation?

The more general type of taxation gets very political. I’m going to try to avoid politics to make general economic points. Higher taxes reduce returns to successful innovation. You could say, “It’s obvious that higher personal income taxes reduce innovation incentives, and the only question is by how much is that innovation reduced.” One proviso would be that it depends what you do with the revenues from that tax. Tax is generally not there for its own sake; it’s there to provide other services. If you use that revenue to do pro-innovation things, that might be a reason for doing it. But the tax in and of itself is going to reduce innovation incentives.

If innovation was about a lone genius model — in your ivory tower as an academic — and that was it, maybe there’s some effect, but not much. Much recent work has tried to test for this by looking at individual inventors — measured by whether you’re an inventor on a patent — and seeing to what extent inventors do or do not respond to changes in the expected tax rate.

There are a few reasons that you might think the aggregate innovation tax elasticity might be small in magnitude. If you think about a fall in the tax rate to induce more innovation, the people who are going to be affected first of all will be the marginal inventors. The really good inventors are inventing, so when you reduce the tax rate you’re going to maybe encourage the marginal inventors. They’re going to be lower quality, so that might reduce the aggregate effect, if innovation was forecastable.

What about if innovation is uncertain? It’s like buying a lottery ticket. If you think about individual incentives, if you have concave utility as we normally think — a dollar to a billionaire is worth less than a dollar to a poor person — then if you imagine you’re a successful innovator but you win the lottery for $1 million versus $5 million, that difference may not be so huge. Even for unforecastable innovation, the tax impact may not be quantitatively large. This is the optimal tax literature intuition.

The third factor is that the decision to be an inventor depends on information. If it depends a lot on information and early motivation or exposure, then that decision on the extensive margin may not have so much of an effect on whether you choose to have an inventor career. It may have a much bigger effect on people who are already inventors and are trying to decide, “Do I do more invention?” or “Which state or which country do I live in?” There may be more effect on that intensive margin.

This paper by Akcigit et al. tries to look at this directly. They take data from the US Patent Office from 1920 onwards, focusing on the 60-year period from 1940. They disambiguate the inventor names and they calculate state-specific marginal tax rates for corporations and for individuals. They focus on the 90th percentile of the income distribution compared to the average, with the assumption that inventors are more likely to be in the higher parts of the income distribution, and they use tax simulation models to work out the different levels of tax that those individuals are going to face.

They do some estimates at what they call the macro level, the state level, and the micro level, which is the individual level, looking at the effect of taxes three years ago on:

• Counts of inventors, including mobility across states;

• Number of patents;

• The quality of patents as measured by citations.

The result they get is that lower general taxes do encourage more innovation.

Broadly speaking, states that increased taxes had slower growth in subsequent innovation.

How do they deal with the fact that when the state changes its tax there could be lots of other things changing at the same time? It could be a new government; it could be a stronger economy. They do various things.

• Because they have individual information, they can include inventor-level fixed effects and look at how the change in the tax rate changes the probability of a patent for an individual inventor.

• Because they’re looking at the top tax rate versus the median personal tax, they can include state-by-time dummies and then identify this difference within a state year by looking across the income distribution.

• They can also use this kind of Gruber-Saez synthetic IV approach that I mentioned before, where you fix an individual’s characteristics and construct an instrument based on the changes of federal and state taxes.

• Finally, they do some event studies.

They look around states which had big changes of tax reforms and use an event study. They find effects on innovation in the states which made the tax systems different. An increase in tax is associated with — or causes — a lower amount of innovation.

They find an elasticity of patents with respect to the personal net tax rate of about 0.8, and the corporate tax rate of about 0.5. So smaller than some of the elasticities from the R&D tax credit, but still significant. A big part of the effect is driven by relocation, a bit like the Wilson paper on R&D tax credits. Inventors are much less likely to move to the high-tax states. For the corporate tax one, it’s almost all relocation. For the personal one it’s divided up: some is the personal tax, some is relocation, and some, they argue, is a general aggregate effect.

So it’s somewhat smaller than the tax credit effects, but there is some effect there at the individual level.

In terms of identification, one of the issues is whether or not they’re capturing the extensive margin. When you include inventor fixed effects, this conditions on people who are inventors: you only look at — if you’re an inventor, do you innovate more when tax goes down? What that misses out is people who could have invented but didn’t invent. What about the “lost Einstein” or “lost Marie Curie” effect — that it doesn’t necessarily take into account whether you become an inventor in the first place? That was the original motivation I started this with. There’s a second thing on the link to the R&D tax credit literature: it’s not obvious how well that’s controlled for.

They emphasized that there was a risk of “beggar thy neighbor” tax policy. They’re not necessarily arguing in favor of using the tax system to encourage innovation, because of some of the effects of relocation. They argue in the paper that building up amenities and research infrastructure may be a better way of reducing the risk of brain drain than cutting taxes. Nevertheless, this is a paper which does show that there may be more effect of changing income tax rates on innovation than would maybe have been previously thought by some.

Innovation and human capital

These demand-side policies, like fiscal incentives and direct subsidies, do seem to be effective. But there’s this critique — Paul Romer was the first to voice it — which is simply that if the supply side is completely inelastic, the main effect of this is to increase the R&D price.

There’s a paper by Austan Goolsbee who, using macro time series data, argued that if you looked at federal subsidies, almost all the effect was simply to increase the wages of scientists rather than the volume of R&D. I’m not sure I totally believe that 100% of the incidence is on wages, but certainly in the short run prices are likely to increase as you increase the subsidy, unless the supply side responds.

That’s why there are a lot of attractions to doing these supply-side policies. If you can increase the quantity of R&D workers, that directly boosts innovation, because people who are doing R&D do more innovation. But it could also have an equilibrium effect: by increasing the supply you reduce the price of R&D, you depress the wages of people, and this will give an indirect boost to innovation through a general equilibrium effect.

One problem is that you train more R&D workers — more people come out of MIT with a physics degree — and if they all go off and work on Wall Street, there’s a leakage of that talent to other sectors of the economy. Of course the effect might be slower than a subsidy. With the tax credit you get an immediate reduction of the price of R&D; in order to train someone to have a PhD in physics it takes several years, and then they have to go out and do the research. So it’s a slower effect than the demand-side policies.

[Question:] Is there some evidence about the elasticity of this supply of human capital? At least in the medium run, and even more in the long run, the wage of researchers is going to attract people into research. Is this supply quite inelastic?

There is evidence on this, and there’s dispute over the magnitude of it. There is clearly an elasticity. A lot of the estimates have been done on looking at elasticities of existing inventors — like the paper I mentioned, that was cross-state elasticities. Stefanie Stantcheva has a really nice paper looking at cross-country mobility of inventors, and they respond a lot to tax incentives.

In terms of the effect of higher wages on choosing to become an inventor, there’s a broader labor economics literature on that, looking at how people go into occupations and study as a result of prices. But the evidence there is much weaker, because it’s an occupational choice decision, and that’s a long-run decision. You’re looking not just at a small change in wages, you’re looking at your expectation over the whole future. There are some estimates of that, but I’d say it’s pretty unclear what the magnitude is of the response. I think everybody would agree that there is an effect as you increase the wages of different groups, but the magnitude of that is quite hard to identify. That’s an important parameter, of course, in terms of assessing some of these effects.

Universities and graduate supply

There are lots of types of human capital policy:

• Increase the supply of STEM-qualified people,

• Expand universities,

• Increase skilled immigration,

• This exposure, “lost Einstein” or “lost Marie Curie” type of effect.

There’s evidence of positive impacts of universities and entry expansion on GDP per capita at the subnational and national level. There is better-identified work focusing on universities and innovation, usually finding positive effects. There’s an old paper by Jaffe looking at state-level spending on university research, which seems to be associated with more corporate patenting. There are innovation surveys. There are nice papers by Belenzon and Schankerman and Hausman looking at patenting in universities.

One of the big issues is that university presence or expansions aren’t exogenous. Furman and MacGarvie have a nice paper using the land grant college funds to instrument for university location, and find that it seems to have a positive impact on corporate pharmaceutical R&D.

One question is: even if there is a positive causal impact of universities on innovation, what’s the mechanism? It could be faculty scientists at universities starting firms, doing research; it could be democracy and institution building; there could be demand effects as well — Andrews has a paper on this. All of those could be at play. There’s a lot of important work which could be done, especially now we’re in an era where universities, especially in the US, have had all their research funding cut. People are very interested in what the impact of that might be in terms of thinking about growth and innovation.

There is a series of papers which have looked at whether graduate supply has had an effect — that’s an obvious mechanism. There’s a nice paper by Bianchi and Giorcelli who looked at enrollment requirements changing for STEM Master’s in Italy, and they found that subsequent innovation increased, especially in biomedical and ICT, although there was some leakage into other sectors like finance.

There are also some papers looking at an increase in STEM (Science, Technology, Engineering and Math)-focused colleges. A paper by Otto Toivanen and Lotta Väänänen looking at technical schools in the 1960s in Finland found they had not just direct effects on patents but also indirect effects on the second generation, the children of those original people who went to university. There’s also some evidence in Norway. This is a growing area trying to get at one of the mechanisms, which is the supply of high-skilled workers, and there seems to be some evidence on that.

There is lots of good stuff on immigration: Kerr is my go-to reference.

“Lost Einsteins”

This is a stream of research that I began with Raj Chetty, Xavier Jaravel, Alex Bell and Neviana Petkova, looking less on what you do when you’re already an inventor, but whether you become an inventor in the first place. We matched patent applications in the US and patent grants from 1996 to de-identified tax records. For everybody in the US we could figure out whether or not they were on a tax record.

Then we focused on looking at who grew up to become an inventor. In particular, we were interested in this question of whether there are some groups who systematically are much less likely to become inventors than others. We documented how kids from low-income families, minorities, and women are very underrepresented in the inventor pool. Part of the paper was looking at, what is the reason for that? We found evidence that the majority of this effect wasn’t due to lower ability, but rather more to do with lack of opportunity or exposure to the very idea of becoming an inventor.

To show you a couple of stylized facts from this.

We looked at whether you became an inventor and we lined that up against your household income. I put those in percentiles here along the x-axis, from kids from the poorest families to kids from the richest families. You can see here that there’s an upward-sloping curve. It’s a bit convex. The patent rate if you’re in the bottom 50% of the income distribution is about 0.84 per 1,000. The patent rate if you’re born in the top 1% of the income distribution is 10 times higher than that: 8.3 per 1,000.

Of course, it may be that parents up here are much more able than parents down here. What we did was we matched in information on different measures of early human capital. This is one of the experiments.

We looked at the math test results that kids got in the 3rd Grade and split this up in various ways. Along the x-axis is your standardized math test score, in standard deviations.

We found that kids from poor families were less likely to become inventors than kids from rich families. But even when we look among the very smart kids, it’s still the case that the kids from the rich families are much more likely to become inventors than kids from the poor families. When we do a formal decomposition of the effects, we find that although ability can explain maybe a third of that gradient between income and invention, most of it is not explained by different measures of ability; it looks like it’s other factors.

Similarly with gender: if you look at the difference between men and women, there are a lot more men who are inventors than women.

If you look over time, that’s started to converge. For cohorts in the 1940s, less than 10% of inventors were women, compared to about 20% of the cohort born in 1980. But the rate of convergence is pretty slow — it’s going to take over a century to get full parity at current rates. If you look at measures like math scores or other kinds of measures of ability, men and women are very similar. That can’t explain these differences.

The thing which did seem to be more important was the exposure that you had to the possibility of becoming an inventor when you were young. We cut the data in various ways to look at this, but one way is to look at this by place. We broke up the adult invention rate by where you grew up.

If you grew up in a place where there was a lot of innovation happening when you were growing up, as measured by things like the patenting rates, you were much more likely to subsequently be an inventor than if you were in a place where there wasn’t so much innovation going on. In San Jose and Silicon Valley, a lot of people grew up to be inventors. Someone from, say, Brownsville is much less likely to grow up to be an inventor.

Of course, those are just correlations. To try and get at the causal impact of some of these place-based exposure measures, we try different strategies.

• We look at timing: we look at adult outcomes and child exposure. We can dummy out the current destination place — take two people living in Boston, where did they originally come from — to see what the effect was on innovation.

• We use the sharp discontinuity by technology class, with the idea that growing up in an area that specializes much more in software compared to medical devices — if there was a genuine exposure effect, it should mean that you’re much more likely to grow up to innovate in software than medical devices. That’s true even within very narrowly defined patent classes.

• Finally, we can use a kind of movers design, where we can compare families where kids moved at a young age compared to ones moving at a later age. This is all conditional on these being people who moved, because people who moved are not the same as those who stay, but the exact timing of how old the kid was when they moved — we can argue that’s not correlated with the overall chance of becoming an inventor, except through the mechanism of being more exposed. We found that if you moved when you were a kid, say when you were two, compared to moving when you were 15, to Silicon Valley, the kids who moved when they were younger were much more likely to grow up to become inventors.

All these designs suggest an important role of early exposure to innovation in determining who grows up to be an inventor. That suggests that when we think about the right model for pulling people into becoming inventors on the extensive margin, it goes much beyond the conventional tax measures that we think of. We might want to think about other types of policies:

• Educational interventions at earlier ages;

• Mentorship programs to give people experience working for innovative firms;

• Other ways of reaching out to give people who are not traditional inventors the idea, “Well, this might be a possible career route that you might want to think of.”

One of the problems with looking at whether you’re on a patent as a measure of innovation is that it’s quite narrow. Some of the work that Raj and I have been doing more recently is looking at entrepreneurship as a wider measure of innovation. This is work which is literally hot off the press — or slide deck, I should say. It’s not even a paper yet. We look at 10 million US entrepreneurs using the IRS data who founded employer firms — looking at stars who created lots of employment and revenue in particular. We document similar gaps in entrepreneurship rates by class, gender, and so on. One difference is that, although there are these effects of early exposure — and we also look at liquidity shocks as well — we find that a critical thing for entrepreneurs is whether you had early labor market experience of working in more entrepreneurship-friendly firms, like tech firms. We use port-of-entry type of instruments there: whether you graduated into San Francisco in 1999 when there was a dotcom boom, compared to 2000 when there was a crash. We find that this early labor market experience is important for entrepreneurship, and one of the things we’d like to do is also look at that for inventors.

Conclusion

Human capital policy acts on the supply side — it’s more attractive than demand-side tax subsidy policies — to reduce equilibrium cost, and also reduce inequality for some of these things as well. These are policies which can be good for equality and also growth. There’s much less of an empirical literature, and policies take longer to have an effect.

Some lessons. A lot of what we’ve done here is leveraging off the credibility revolution, using modern empirical work in innovation policy. It does pose special challenges when we think about using a lot of the Angrist-Pischke type techniques: spillover effects can violate the SUTVA assumption — your control group can be affected by the treatment. Because innovation is all about spillovers, this poses a particular challenge when you’re thinking about empirical work. General equilibrium effects are going to be critical, because we’re focusing on growth.

Despite some of these issues, there’s been a lot of progress made over which policies work better or worse over the last 10-15 years. Tax does matter — that’s certainly coming out of the literature. But the supply side is also critical. There are a lot of opportunities for very good work, especially on the supply side.

But the bottom line is that there are effective ways of increasing the growth rate and the innovation rate through policy interventions. A lot of people are unduly pessimistic about the future growth rate. People often take my paper ‘Are Ideas Getting Harder to Find?’ in a negative way, to say, “This shows you that there’s diminishing returns to R&D.” The point of that paper was that even if there are diminishing returns, if we want to have the rates of economic growth that we have become accustomed to, and we want more of them, then we can do it if we’re prepared to make policies which raise the rate of R&D innovation. That’s absolutely possible, and many of these papers show that there just needs to be the political will in order to make that happen.

In the past few weeks, metascience units have gained new prominence. First, the National Science Foundation (NSF) announced the Office of Metascience. This is, to our knowledge, the first metascience unit in the federal government. Then, the bipartisan House Build America Caucus agenda elevated metascience units as a top national innovation priority. And in Japan, Office of Science and Technology Policy (OSTP) Director Michael Kratsios announced a vision statement with 17 countries that included a call for “dedicated metascience capabilities in our governments,” which sure sounds like metascience units.

This was all welcome news to us, since we had published an article proposing metascience units at federal agencies in July. As we described them, metascience units are:

Nimble teams of government staff and academic experts housed within agency leadership offices. Using the tools of metascience, these units would run institutional experiments and analyze the treasure troves of internal agency data to provide evidence about how to most effectively advance breakthroughs.

The US government needs metascience units because (1) science agencies have limited information on how well science funding is being spent and (2) there are enormous opportunities to make science more efficient and increase the likelihood of breakthroughs.

However, metascience units are new in the US.1 If the US government is serious about creating these units across science agencies (as it appears to be), it’s important to get clear about what they should look like.

Since publishing our piece, we have received thoughtful comments from across the science policy ecosystem, including from people currently or formerly in government, in academia, and in think tanks.2

This feedback focused on eight important topics:

1. How different metascience units should be across different agencies.

2. Whether the units should fund external grants and contracts.

3. How independent the units should be from agency leadership.

4. How to make the units sustainable.

5. How the units should be staffed.

6. How the units should work with other agency analysis offices.

7. What sorts of evidence the units should focus on.

8. What they should produce, and how their findings should change what the agency does.

Subscribe now

Thinking through our answers to these questions helped us sharpen and update our views. For example, a few comments convinced us that metascience units must have enough independence, likely enshrined in congressional authorization, to share research results even if those results are uncomfortable for agency leadership.

To keep the conversation going, we’re sharing our responses to each question here.

Question 1: Should metascience units look different in different agencies?

We positioned this question first not because it’s the most important question, but because it helps explain why we say “it depends” for subsequent questions.

Metascience units will likely differ quite a bit based on their agencies’ missions and structures.3 For example, the Pentagon is naturally going to be quite different from NSF (it’s massive, has multiple agencies under it, conducts classified research, etc.), and some agencies may have somewhat different hypotheses about what should be improved (e.g., the National Institutes of Health (NIH) will likely focus its metascience efforts on replications, at least initially).

In a small, relatively centralized agency such as NSF, a metascience unit working in the Director’s office can likely collaborate with directorates without substantial challenges. In an agency made up of highly independent offices such as NIH, the unit might need a different approach — such as coordinating with liaisons in specific institutes or centers that are working with the unit. In the Pentagon, to be effective the metascience unit might need to coordinate across DARPA and the military research labs or decide to limit its remit to a specific part of the massive defense R&D ecosystem. In short, diversity between institutions implies diversity in metascience unit approaches.

As we wrote in our initial piece, we also don’t think every research agency needs a metascience unit. It makes sense that the first two agencies to launch metascience units are NSF and NIH (the NIH unit is likely to launch as the Office of Research Economics, Planning, and Analysis), both of which exist to fund and conduct research.

Subscribe now

Question 2: Should metascience units give out grants and contracts or just conduct research internally?

Science agencies should be able to run internal experiments to improve how they structure, fund, and conduct science. One way metascience units could do this would be to award grants, contracts, or other transaction agreements for third parties to conduct the experiments and evaluations, especially when the unit lacks the internal expertise to conduct specific types of analysis. This has the added benefit of signaling information to the research ecosystem and creating an expectation that research results will be published.

However, we think external grants for metascience research should not be a unit’s primary focus, for three reasons.

First, managing grants and contracts is labor-intensive, and the skills required to do so differ from the skills required to do metascience analysis. Under this model, it would be easy for a unit to lose its analytic capacity as it outsources its thinking to other groups. Second, grants and contracts can be expensive, they are often quite slow to set up and execute, and contractors and grantees can lack sufficient context to provide useful analysis. Finally, providing grantees and contractors access to agency administrative data can be difficult due to protections under the Privacy Act of 1974.

Separate from the work conducted by metascience units, agencies should also fund serious external metascience research that has no direct bearing on agency decisions. This kind of academic research should be funded by other departments or programs, as is done at NSF and NIH. At NSF, the Social, Behavioral, and Economic Sciences Directorate’s research on metascience would not be appropriate to bring into the NSF Office of Metascience (because it doesn’t focus on NSF’s performance). Similarly, at NIH, the Common Fund might fund external metascience research, but it is more appropriate that NIH’s proposed Office of Research Economics, Planning, and Analysis conduct the internal research on NIH programs. The structure of those two functions can look very different, but we need to fund both to grow our metascience understanding.

Question 3: How independent should metascience units be? How can they be incentivized to share bad news?

We heard some concern about the possibility that metascience units might keep uncomfortable or embarrassing evidence quiet. Why would an agency want to share information that makes it look bad?

There might be tension between independence and leadership relevance. If units lean too far into independence from agency political leadership and responsiveness to Congress, they might function largely as compliance offices — checking boxes on agency performance in response to Congressional requirements or inquiries without being useful to leadership decision-making. Conversely, if they’re less independent and function solely as tools for leadership, they may become politically captured and will have little incentive to share uncomfortable findings.

It’s challenging to define the appropriate level of independence for metascience units, and we don’t have a foolproof answer. There are, however, examples of offices where the evaluations are commissioned by leadership but released without leadership approval. Government statistical agencies release publications on pre-announced schedules, without prior review or approval by agency leadership. As an example of how this can look, the US Energy Information Administration makes requests from Congress and Cabinet secretaries public and then publishes the findings. Metascience units can also partially model the authorities of Offices of the Inspector General (OIGs), where the office has statutory independence, a right to access agency data, and freedom to publish findings.

These examples are not perfect analogues, however. A key difference between statistical offices and metascience units is that metascience units should provide policy advice to agency leadership. It’s often ambiguous what analytic evidence implies for decision-making and, since metascience units are explicitly designed to inform agency decisions, the units should both produce evidence and share perspectives on the implications of that evidence. There is also a critical distinction between any comparison of OIGs and metascience units: today most OIGs focus on ensuring compliance with rules via audits and investigations; metascience units must focus on using research to improve processes.

To move toward balancing responsiveness and independence, we think units could adopt (and Congress could authorize) the following transparency measures:

• Publishing annual work plans, with major studies preregistered before they begin.

• Releasing findings without agency leadership review.

• Transmitting published analyses to congressional authorizing committees and sharing them with the public.

• Building a strong norm that the agency head will ensure that the metascience unit can access vital information without difficulty or delay.

In addition to Congress authorizing institutional independence, the White House can also encourage publication of metascience unit findings. Indeed, in the Science: A New Golden Age report, OSTP suggested that metascience results should be public. It’s possible (fingers crossed) this kind of transparency could be a point of executive and legislative branch agreement.4

Subscribe now

Question 4: How do you build a metascience unit that lasts, and does it need to?

Metascience units should last long enough to (1) produce a body of evidence about interventions that meaningfully change outcomes, (2) build evaluation and data infrastructure, and (3) diffuse a culture of experimentation within science funding programs. The minimum time to do this is probably around 10 years given how long it takes to connect science funding choices to outcomes. To reach those milestones, metascience units will have to overcome the four failure modes of similar projects that Jenn saw in the federal government: cycles of responsibility, change of administration, unsustainable funding models, and rejection by the agency.

Cycles of responsibility: When you zoom out over decades, where the government places the responsibility for experimentation and process reinvention follows a cycle. At the beginning of the cycle, innovation is everyone’s job. This continues until leadership comes to realize if it’s everyone’s job, it’s no one’s job. That realization activates the next phase of the cycle. Innovation capacity becomes centralized and made available as a service to other offices.5 This continues until leadership comes to see this capacity as unnecessary overhead, especially if innovation units don’t have strong champions within other offices. Then the cycle starts all over again. This may be inevitable in the very long term, but better institutional design can give the unit a longer lifespan and make it more effective during that time.

Change of administration: With every new administration comes new or changed priorities. If one initiative becomes associated too strongly with a prior administration, it will be more vulnerable in a new administration. This can be counterweighted by avoiding politically driven topics, authorizing and appropriating funds through Congress in a bipartisan fashion, and building buy-in from senior career staff at the outset, since they’re the ones who carry good work from one administration to the next.

For metascience units, political appointee involvement would give agency leadership more ownership but would leave the unit more exposed when administrations change and would make institutional independence (discussed in question 3) impossible. For these reasons, we think metascience units should not be helmed by political appointees but instead by a strong career director paired with metascience experts who set agendas on both near-term and decadal time horizons.

Unsustainable funding models: Reimbursable shared service models — where a unit provides a service within the agency or other agencies in exchange for financial reimbursement — can exacerbate the temptation to end centralized capabilities, especially when the units aren’t fully self-sustaining. For example, 18F (of the General Services Administration) and the Lab (at the Office of Personnel Management (OPM)) provided government-wide shared services, the former for building, buying, and sharing technology products, and the latter for providing federal workers with 21st-century skills in design-led innovation. While they both had numerous high-profile successes, when the General Services Administration and OPM cut non-essential functions in 2025, these units weren’t perceived to be central to their agencies’ mission and weren’t a profit center, so they were shut down.

Metascience units should not run on a reimbursable model but should have their own appropriation that funds the metascience expertise, control groups for evidence building, and the evaluation and data infrastructure. To support sustainability and independence, budget requests for metascience units should reach Congress intact as required under statute for OIGs (5 U.S.C. § 406(g)), to avoid a situation where political leadership at the agency or the Office of Management and Budget deprioritizes their functions.

Rejection by the agency: Functions that are perceived as compliance requirements or activities that aren’t central to an agency’s core purpose and that “tax” other organizations’ budgets invite pushback from the agency. Without internal champions outside of the unit, no one will be advocating for it whenever pushback arrives. So unit designers must be thoughtful about structuring units to cultivate internal champions. They can do so by explicitly serving three customers: Director/Administrator priorities, directorate/center priorities, and a bottom-up channel from program directors.6 By creating room in the portfolio (ideally the majority of it) for priorities that come from directorates/centers and program directors, units can achieve longer-term sustainability.

Question 5: Who should work in metascience units? What skill sets are needed?

Metascience units will benefit from a mix of permanent civil service staff and academic researchers who might serve in government via the Intergovernment Personnel Act Mobility Program. To meet research needs, agencies should recruit members of the six camps of metascience, depending on the agencies’ specific focus areas. We are partial to innovation economists, but these might not be the most needed profile everywhere. For example, if NIH is focused on replications, people from the Research Integrity camp might make more sense. As discussed in question 7, units shouldn’t just focus on quantitative research and so will need experts who are comfortable with different types of social science research.

Finally, metascience units also need people with expertise in agency operations. It can’t be all detailees and rotators from academia or nothing will get done. You want nimble bureaucratic ninjas.

Our suggestion for metascience unit leadership: have a metascience unit director who is a career staff member, with a deputy who is an external research leader. Having a career staff member leading the unit provides the continuity and institutional knowledge needed to ensure the unit can effectively work with agency leadership and across agency offices.

Subscribe now

Question 6: How should metascience units engage with evaluation offices and other agency structures for collecting data? How can metascience units get other parts of their agency to participate in experiments and analysis?

Multiple friends mentioned that metascience units had commonalities with evaluation offices and asked how they were different and, more importantly, how they might work together. This question makes sense: while program evaluation was already integrated into many government programs, the Foundations for Evidence-Based Policymaking Act of 2018 (widely known as the Evidence Act) required agencies to develop evaluation plans and agency learning agendas. In response to this act, many agencies staffed evaluation offices that conducted analysis on program implementation, structured and disseminated agency learning agendas, and reported to Congress on their activities.

We agree that there are similarities between metascience units and evaluation offices, but there are important differences. For example, evaluation offices tend to spend most of their time evaluating whatever their agency is spending the most money on. This makes sense, but the consequence for a place like the Department of Health and Human Services is that most of the emphasis is on evaluating things like Medicare and Medicaid and not research funding. By comparison, metascience units should be relentlessly focused on analyzing the effectiveness of science funding approaches and identifying potential improvements.

There are many lessons from evaluation offices and Evidence Act activities of the early 2020s as well as lessons from evidence translation to policy that should be applied to making metascience units successful. Many of those have been incorporated into our recommendations.

Further, some agencies likely have evaluation office capacity that can help metascience units get started. Evaluation offices have different levels of capacity; some primarily serve as compliance offices while others have inculcated a culture of learning in their agencies. Those that have been effective in conducting internal analysis could provide helpful support to fledgling metascience units.

One thing we want to reiterate from our original metascience units article: where the unit sits within an agency is important. As we wrote in response to an OSTP Request for Information, metascience units should be placed in leadership offices “to ensure cross-agency visibility and protect against capture by any particular program.” That’s the plan at NSF. The issue with a unit placed several levels down is that it will pick up incentives and culture from wherever it sits. For example, one grouped with offices conducting compliance or performance-reporting work will tend to think in terms of compliance and performance reporting. Which leads us to the second part of the question.

Getting other parts of agencies to participate in metascience analysis is crucial and can be one of the weak knees of an evaluation effort. Agency leadership buy-in might be necessary but not sufficient. So how can the units get other parts of agencies to cooperate? As was noted in the Science: A New Golden Age report, metascience units can have authority to require other parts of the agency to collaborate on research and the funding to encourage them to expand projects that might cost extra money.

To be useful to the agency — and not be perceived as solely extracting information from agency staff — metascience units must be responsive to bottom-up questions and project ideas. We’re not recommending a figurative suggestion box that nobody uses. Rather, metascience unit staff should conduct outreach across their agency, provide readily accessible avenues for offices to request help, and ensure that they respond to all requests. For example, an agency office might request help in figuring out the right structure for “golden tickets” — which allow individual peer reviewers to advance a project for funding even when other reviewers disagree — in its external review processes and analyzing the effectiveness of the golden-ticket approach in selecting novel projects. In short, while metascience units analyze the effectiveness of science funding across an agency, they should be obviously useful to the offices that are funding the science.

Finally, having centralized access to data will be hugely important for making coordination with offices across their agencies effective (and less painful than it otherwise might be). Metascience units should be able to access grant and contract data without having to constantly harass different offices. Agencies therefore need centralized repositories of agency-wide grant and contract data.

Question 7: What sorts of evidence should metascience units focus on? Should metascience units conduct qualitative analysis?

Much of the impetus for launching metascience units is that they can conduct quantitative analysis of agency-level data that is useful for decision-making. That is often done via simulations based on real data, historical analysis of data, and randomized controlled trials (RCTs). RCTs are a gold-standard approach to demonstrating causation, but they are also difficult to pull off in government programs due to challenges with (or staff resistance to) randomization and getting sufficiently large sample sizes. These experiments should be designed primarily for questions that are “big if true.” They must have enough statistical power to identify meaningful results — that is, detect realistic changes that leadership will care about.7

Metascience units are not RCT units, though. Small studies can help in making the case for bigger studies, and simulations can often provide significant insights while being much easier than RCTs. Finally, we think there is an important role for case studies. Especially with questions of institutional design or reform, quantitative data will miss crucial details, and the sample size might be too small to reach statistical significance.

As an illustrative example, NSF recently announced the X-Labs program, which intends to fund a small number of large labs to conduct different types of frontier research. Due to limited sample sizes and differences between grantees, it would be unrealistic to study X-Labs via an RCT. But we do need high-quality information about what is working well about the project and what can be improved, both so that NSF can make changes and for future government programs that might be modeled on X-Labs. Studying programs like X-Labs well will require qualitative or mixed-methods approaches.

Subscribe now

Question 8: What should metascience units produce, and how should their findings change what the agency does?

Metascience units are designed to conduct research that helps agency leaders make decisions about how they structure, fund, and conduct science. In pursuing their mission, metascience units should produce analytic products — research papers, policy briefs, etc. — that provide evidence about how effectively an agency is funding and conducting scientific research. But a metascience unit can still fail if the evidence it produces does not influence decision-making.

As a recent piece by Leya Mohsin says:

Rather than assuming the founding design is correct and static, new institutions have a rare opportunity to build in real feedback loops and the flexibility to course-correct, treating the initial design as a hypothesis rather than a final answer. In the R&D context, proposed metascience units are a promising way to tackle this — but if, and only if, mechanisms are constructed to ensure that the evidence they generate has a meaningful impact on decision making.

What does this require? While we should have some humility about how even the most theoretically sound designs will work when they make first contact with a federal agency, we have some ideas about how to increase the likelihood of evidence informing decision-making.

Two approaches that we have already described could help. First, units should conduct research that is useful to agency leadership and take on projects requested by agency offices. Second, units should make concrete policy recommendations whenever they produce research findings.8

In addition to those, units can build capacity for effective communication between the metascience unit and program designers. This doesn’t mean having people whose job title is “communications.” Rather, metascience units should recruit staff with strong science communication and policy entrepreneurship skills — who can understand research findings and seize opportunities. Cultivating a learning mindset and prioritizing evidence-based iteration can also help substantially; if results are not informing decisions to the extent desired, what can the unit change to increase the likelihood of utilization?

We would like to thank friends who provided thoughtful feedback on the initial Science Agencies Need Metascience Units piece, including Kelly Bidwell, Amira Boland, Albert Bravo-Biosca, Erica Goldman, Kei Koizumi, Lauren Supplee, Christina Yancey, and others.

When I have heard people’s grand visions of how AI will reshape scientific progress, I often find myself thinking: a few more people should talk to an economist. I have no doubt that AI systems will be revolutionary for science. Earlier this week we got a glimpse of it in mathematics. But economists inject the right kind of realism into the conversation.

Thankfully, this week’s Economics of Ideas, Science, and Innovation lecture brings us one of my favorite economists on AI.

Kevin Bryan has spent time writing benchmarks for AI and is a professor of strategy at the University of Toronto.

Visions of a million Einsteins in a data center will take time to transform many economically relevant undertakings, especially in the sciences. Even for our own Einstein, it took about 100 years to detect gravitational waves directly. Science runs on decades of institutions built around systems, many physical, all of them working together: lab instruments, journals, grants, to name a few. For advances in AI to have the largest positive effect, current institutions will need to be rearranged around it, and new ones built around AI from scratch will likely form too.

Subscribe now

Kevin provides a compelling example in the lecture. When factories switched away from steam engines to electric motors, productivity barely moved. Factories were built around a single shaft overhead with belts running down to each machine. To paraphrase Kevin:

Adding an electric motor to a factory that’s built around belts and shafts barely matters. Reorganizing your factory around electricity matters a lot.

Subscribe now

Reorganizing firms isn’t the only model to consider for how advances in AI plug into our economy and our scientific enterprise. Kevin presents us with a handful of models of growth and productivity. None are exactly right, but in certain applications, some are more right than others. As we’re reminded in the lecture, “all theory is wrong; it’s for a specific use.” Yet thinking through each of them is an enriching exercise, and for that, I’ll leave it to Kevin.

Here are the readings that go with this lecture:

• David, Paul A. “The Dynamo and the Computer: An Historical Perspective on the Modern Productivity Paradox.“ American Economic Review 80, no. 2 (1990): 355-361.

• Agrawal, Ajay, Joshua S. Gans, and Avi Goldfarb. “Exploring the Impact of Artificial Intelligence: Prediction versus Judgment.” NBER Working Paper 24626 (2018).

• Shahidi, Peyman, Gili Rusak, Benjamin S. Manning, Andrey Fradkin, and John J. Horton. “The Coasean Singularity? Demand, Supply, and Market Design with AI Agents.“ In The Economics of Transformative AI. University of Chicago Press (2025).

• Patwardhan, Tejal, et al. “GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks.“ arXiv (2025).

• Ide, Enrique, and Eduard Talamàs. “Artificial Intelligence in the Knowledge Economy.” Journal of Political Economy (2025).

Thanks to William Higbie and Harry Fletcher-Wood for their support in producing this video and the transcript.

Subscribe now

Lecture Transcript:

The Economics of AI and Innovation

This is a topic that we’ve had filtering into some similar programs we’ve run — I teach the PhD innovation course up here at Toronto as well, and we’re always incorporating more and more of this econ. We’re at the point now where it just has to be a core component of the course.

For those of you who don’t know me, I’m trained as a game theorist — a weird game theorist who studied under Joel Mokyr. I’ve always been interested in innovation, and I’ve been doing work in the theory, history, and empirics of innovation for many years. I started working in AI quite a while ago. I came to Toronto in 2014. If you know your history of AI, 2012 is the big ImageNet competition, where a team of students and professors from this university used deep learning to win a major contest for the first time. That led to a complete blow-up in Silicon Valley. When OpenAI starts, one of our PhD students here, a guy named Ilya Sutskever — Elon Musk and Google are having a bidding war to hire him, because even by 2015 it was widely understood in Silicon Valley where AI was going.

In 2017 we have the famous Go match — the move 37 match — where a deep learning model beats the world champion. Then in 2022 we have ChatGPT. A lot of people now, when they hear AI, they’re thinking of large language models you interact with in a chatbot. That won’t be all I’m talking about. AI is a much broader problem. I’ll give you a bit of structure on how we think about it. But in order to make sure we’re all on the same page, I need to show you a handful of charts, which are hopefully going to convince you that AI as of March 2026 is relatively uninteresting. The interesting thing is the lines on the graph and their implications.

This is the time-horizon benchmark from METR, a private organization.

They set up a small number of computational tasks, where they wanted to see how long a model could go until it screwed things up. How long does it go, 50% of the time I run the model, before it makes mistakes and destroys the task? A few years ago the answer was a minute or less. You can see on this graph, we get up to late 2024 and we’re talking a half hour. This is from January — it’s four and a half hours. Actually, the benchmark is broken now. They tried to measure a new model and got 14 hours, and realized the problem is it’s just too saturated to really know at this point — the variance has gotten too big. This benchmark’s already destroyed, that quickly.

Here’s another one that I like, called FrontierMath.

They hired top mathematicians to write problems they thought were hard. Tier 3 is: “Someone in my field could solve it, but it would be pretty difficult; for a PhD student this one would be quite a challenge.” As of November 2024, it’s 2% solved. By December 2025 we’re talking just over 30%. Now we’re up over 50%.

They had to make a new benchmark called FrontierMath Tier 4.

These problems are really, really hard — legitimately difficult problems written by people like Terence Tao out of their file drawers. This was meant to be one that was going to last a few years. They release it in late 2024. No model solves any of the problems until — I want to say February 2025 — o3-mini solves one. Then we start cooking, and we’re going up and up. Now we’re almost to 40% on FrontierMath Tier 4.

They had to create another new one they’re going to release, called FrontierMath Open Problems, which are literally unsolved problems in the file drawers of some of the world’s best mathematicians. On the math side, that’s where we’ve gotten to with these models. Forget about the absolute number we’re at now — just follow the line on the graph up. That’s the interesting thing.

We’ve gotten to the point now in 2026 where — I write benchmarks for one of the big labs. I’m very close to not being able to do it. I can’t come up with problems that AI consistently cannot answer. There are some classes where we know there are issues: putting together a thousand answers correctly in a row is still difficult; consistency is difficult. But these problems are all quite solvable.

Even the type of things that large language models are going to have trouble with — large language models know how to use tools through agentic frameworks, and large language models are at the point where, inside the big labs, they’re contributing to research on the next model. So even if the breakthroughs we need aren’t large language models — they’re other forms of a deep learning model — it doesn’t matter, because large language models now are sufficient to speed up the research on developing that next model.

If you want to see where this is heading — a guy named Leopold Aschenbrenner wrote ‘Situational Awareness’ in 2024. You may have known him if you’re an economist. Six or seven years ago he wrote this growth theory paper — it’s a pretty good paper. When it was put online, he was at Columbia, and people were like, “Who is this kid, some new grad student?” No, he was a 19-year-old undergraduate. So he decides not to do his PhD in econ. He goes to OpenAI, because apparently he made good decisions the whole time, writes this book, gets fired from OpenAI for — I don’t know — talking to the press too much, and then he starts a hedge fund for AI investments and now makes a bajillion percent return per year.

He wrote this long essay called ‘Situational Awareness’, where he tries to explain what people are thinking inside the labs. He’s going through some of the benchmarks they had been trying to make.

For example, we tried to make this MATH benchmark in 2022 — doesn’t matter, it’s already solved by 2024, there’s no point in looking at it more. Then we try to create GPQA. These are like PhD-level qualifying-exam questions across fields. You wouldn’t do well — an in-domain PhD would get 80% on average. And he goes, “I expect this to fall within a generation or two.” We’re 21 months later, and there are models well into the 90s on GPQA. It’s more than fallen.

The reason the lines on the graph matter is: if you look at some of the other graphs that Leo talks about, we’re going to project those types of increases in capability forward another two or three years, and then things start getting interesting/very weird.

Particularly for economics of innovation, they get interesting/really weird, because we start to get to the place where, at least on technical grounds, it seems like AI should be able to contribute to speeding up the rate of science in a really meaningful way.

When Dario Amodei at Anthropic talks about a million Einsteins inside a data center, he’s imagining that you’re going to get a huge increase in growth because of these technical capabilities. We’re going to go in a slightly different direction. We’re going to take the technical projections from AI experts seriously, but we’re going to look at the economic reasons why that outcome might or might not happen, and where there might be remaining unknowns or open research questions that matter for whether in 2030 we have a million Einsteins in a data center doing a million Einsteins’ worth of work and causing an appreciable rise in growth.

We don’t traditionally do big-lab or big-money research in economics, with a few exceptions like Raj Chetty’s group. We work on whatever we want to work on; we have control. There are some kinds of papers, like GDPval, that are not really possible to write in that format.

It does make me think whether there are different ways to set up the internal production of science that we might want to pursue in a world with a lot of AI.

Let’s put all this together. This is what we’re going to take for granted — you can disagree; I’m just going to say you’re wrong if you disagree.

The vast majority of tasks that you can do on paper or on a computer, AI will be able to do in the near future. Shane Legg, the co-founder of DeepMind, would call this minimal AGI. Past that, I don’t know — a lot of uncertainty. It might depend on robotics advances and on policy. But at the very least, a lot of important tasks are going to be doable by AI, or are doable already.

Despite that, the technical improvements in AI will not map very quickly into productivity or growth. To give you one example, that I got from Ben Jones, if it is 1960 and you need to run a regression, you’ve got to invert a matrix. Inverting a 10x10 matrix — I don’t know if you were ever forced to do this in your undergraduate math courses, but it takes forever. It’s a very simple matrix to invert. At this point the cost of inverting a matrix is whatever the smallest epsilon you want it to be.

Despite the fact that we are so productive on inverting matrices that I can invert as many as I want and not even think about the cost, it barely affects growth. The reason is that any additional improvements — if we made the cost of inverting a matrix 90% cheaper at this point — it’s the Baumol effect; so unimportant in the production function compared to other bottlenecks that further improvements basically don’t matter. At least for the kind of work we do. It might matter for computer chip efficiency, but it barely matters. The mapping between “a technology gets 10 times better” and “growth gets 10 times better” is not going to work. We’re going to see why.

The dynamo and the computer

Let’s start from our friend Paul David. I love this paper, ‘The Dynamo and the Computer.’ Everyone reads this paper. I think it’s come up when I’ve talked to private-sector folks half a dozen times — somehow it’s made the jump from academia; they’re all aware of this one.

This is a factory prior to electrification.

You can see on the top I’ve got a shaft. It’s going to turn these belts here; they’re going to hit the individual machine and provide power to the individual machine. Somewhere in the basement I’ve got a water wheel or a steam engine that’s creating power. I’ve got one source of power for the whole factory. So it’s either turned on or turned off — meaning I want all the machines to be used at the same time.

It’s somewhat dangerous. You probably notice those old garment-district-type buildings in New York City — tall brick buildings with heavy floors. That’s because of all the shaft apparatus we need up here in the sky and the heavy machines that we have to bring these belts down to. We’re really restricted in where we place the machines. We often are going to do them stacked vertically in order to move the power more efficiently. So the entire design of the factory is based on these belts.

It’s 1882. Our friend Thomas Edison founds General Electric and sets up the first power plant in New York City. We can turn on the lights now. We can just plug in the machines. How much efficiency are we going to get from this invention? Initially the answer is hardly any. If it cost us five cents of electricity to turn this machine all day, now it’s going to cost us four cents because we plug it in instead of using the steam engine. Who really cares? It barely matters for the overall productivity of the firm. If our costs were 5% power and 95% other things, we just saved 1% of costs. You’re barely going to notice this in productivity figures. But you probably have a suspicion that electricity matters a lot for productivity — and you’re right.

What Paul David points out is that this factory barely gains from electricity. This factory, in River Rouge in Detroit, gains a ton from electricity.

You notice something different about this factory, other than the lights being a little nicer: these fellas are standing here at the end of an assembly line. To have an assembly line, I need to turn the factory sideways. We all have to be on the same factory floor. To do that, I need to locate the factory where land is really cheap. When I locate the factory where land’s really cheap, can I do that? If I can plug in every individual machine and I don’t need to be near a source of power like a water wheel — I just need electricity — then I can move the factory to a place that makes sense for this type of production.

Once I have an assembly line, I’m going to change the type of workers that I hire. Once I change the type of workers that I hire, I’m going to change the incentive system that I use inside the firm. Now we’re really cooking — now we’re getting productivity improvements. The plug-in-the-machine-that-had-the-belt-coming-down-from-the-shaft barely matters. The reorganize-the-factory-in-response-to-this-technology matters a lot.

I think this is true for essentially every new technology. So if I’m looking at AI and where it’s going to start seeing impacts on science, or productivity more generally, my thought is not, “What is the existing task content of jobs, and how much does AI affect that task content?” That’s nonsense — despite the fact that this is half the papers you’ll read on this topic. The interesting thing is: when are we going to start doing new tasks that we can only do with organizational changes in response to what the technology makes possible? How long is that going to take? What are the barriers? That’s really what we want to know, because that’s what affects growth.

Pick your favorite technology. A well-known example is ATMs.

This is the diffusion curve for ATMs. We had them in the mid-’70s. They slowly diffuse over 30 years. What happens to employment inside the firm? It goes up initially, as we move the teller slowly into higher-productivity jobs. By this point we’re totally reorganizing the type of task done inside a bank. The person who takes your money and puts it in the vault — that person’s gone by this point, the ATMs are doing that, and the people inside the bank are doing completely different tasks. But you’ll notice labor doesn’t fall, productivity still goes up. And importantly, we’re talking 30 years here.

Conditional on technical improvements, what will happen to income and growth?

This is the key question behind everything we’re going to see today. I’m going to take the technical improvements as given: AI will be able to do the things that we said before. Given that it will be able to do those things, and given that AI might also improve the method of invention — it may give me new ways to do research on robotics, for example — what’s going to happen to income, what’s going to happen to growth, and what’s going to happen to TFP?

I think this is the right model here.

This is a bad model:

This is the model like in your growth theory papers. The theory is right; it’s just — all theory is wrong, it’s for a specific use. In these endogenous and semi-endogenous models that Chad would have taught you about already, generally what happens is I’ve got some function that depends on how many scientists are working on new things, and the more scientists work on new things, the more growth I get. This might have decreasing returns to scale, but basically that’s every endogenous growth model.

I think that’s a bad way to think about AI. It’s true that we’re going to have a million Einsteins in a data center in terms of intelligence. I don’t think that’s the important factor for the outcomes we care about. The reason is that I’m going to expand the function of scientists to a more interesting model, and we’re just going to look at each of these components today.

All the existing organizations in the world have some type of structure. The management folks would call this architecture. This includes everything from fixed components that are hard to change — your brand — but also things like relational contracts inside the firm. For instance, if your firm currently finds it optimal to promote internally and use career concerns rather than bonus pay to incentivize workers, and you see AI coming and think, “We’re not going to promote internally anymore,” you’re going to find out that you can’t actually do that. Everyone’s going to quit, because they’re going to think you reneged on a relational contract. So there are many aspects of organizations that are sticky, that go beyond physical things.

Those organizations exist, however. They are not going to disappear tomorrow. They’re the ones that are going to take some existing stock of ideas and their current structure, combine them, and try to produce new ideas. Hopefully they’re going to try to modify the structure of their organization a little bit to more effectively use AI in doing this task.

Once I have new ideas, I need to diffuse them across society, or I need to extend them by creating complements. That’s what’s going to lead to growth.

You can probably already imagine quite a few pieces of this little syllogism are not necessarily affected positively by AI. Other things need to happen, or things that take time or investment need to happen, before we get improvements. That’s essentially how we’re going to structure today’s lecture.

The sticky structure of organizations

Here we’re going to build on top of Paul David. Paul David is an economic historian by trade; he does have some light models in some other papers. If I were to try to model this — how does organizational architecture limit the ability to use new tools? — I think this is the right way to model it. I haven’t seen anything that formal on this, to be honest. I’m building on a paper by Henderson and Clark in the early ‘90s that’s also not super formal. If you’re a good theorist, this is worth working on.

Because of these organizational frictions, it seems to be that with new technologies — especially general-purpose technologies — we see adoption follow essentially the same pattern in temporal order.

First, a new technology is used by one person inside an organization who can use it within their current workflow without changing anything else. You’d see something like this on AI: “I write code for a company, now I write some of my code using an AI agent.” Everyone is doing this. It requires very little change in the organization. It’s very easy to do.

If we step to the next fastest, it’s that a team within an organization adopts without changing their existing workflows. This is tougher. The key parts of how teams perform work are team incentives and costly knowledge flow. A person can adopt something without needing to communicate with anyone else, or without worrying about team incentive problems. A team doesn’t have that same luxury. If I’m writing computer code as a team, I probably want to also change how I do peer review, how I do unit tests — checking whether the code worked — and how I do evaluation and verification. In order to adopt an AI-first workflow as a team in writing code, I also need to adopt changes to those other practices. Some of those changes in practices might mean the kind of people I hire on my team will be different.

This is where you hear — with a lot of caveats — that a few research papers have found a decline in the hiring of young people inside firms. That’s being driven by this team adoption. We’re starting to change the nature of how we set up our team in response to AI. I don’t think those papers are totally right, but I think that’s the right intuition.

Third thing — this is when it’s really hard. This is the point where, with AI, basically no one’s here yet, but with old GPTs we eventually get there. The organization needs to change some architectural feature in order to adopt. Usually this means they’re doing some new task, not an old one. If you think about the kind of things an organization does, they take the existing technology, preferences, and constraints, and in equilibrium they maximize given their environment. We drop this new change in a factor price — this new change in the price of predicting things with AI — and the type of tasks I should do should be different. The problem is, when the factor price changes, as we go from the short to the long run, I also need to change the mix of capital and labor. I need to change the way I incentivize workers internally. I need to change the type of talent that I hire, and so on. This is fairly difficult. Now we’re getting to the point where I’m moving the factory with electricity out of New York to the suburbs of Detroit and turning it sideways. Now we’re getting pretty big improvements.

The option value bites — it’s costly to make these architectural changes, so you don’t want to do them 10 times. There’s some benefit, if there’s uncertainty in how the technology is going to develop, from waiting until you’re more certain about what the capabilities of the technology will be before paying this cost of reorganizing your firm.

The fourth one is really hard. Sometimes I would like to do stuff that requires regulation to change, requires partners in my supply chain to do things differently, or clients to be able to adopt a technology before I can sell them the thing. These are really difficult. My favorite example of this one is this particular fellow.

If you’re an economic historian, this is a legend from rural North Carolina. This is Malcom McLean. He invented containerized shipping. Containerized shipping is a little bit similar to AI when you think of the frictions. Imagine what it takes to get a productivity benefit from containerized shipping. It’s not just the idea that I can move the shipping container right onto the truck. I’m only getting the benefit once:

• The port is set up for this,

• The unions are allowing it,

• I have the cranes that pick up the box and move the box onto the truck

• The trucks are built to drive more efficiently given exactly the size of container; and,

• The ships are built to stack containers exactly.

Once all that happens, then we get a massive productivity gain from containerized shipping. Until that happens, putting your stuff into a box of this particular size barely matters for productivity.

We’re in the era of AI, just like any new technology, where we’re a long way from the containerized-shipping world. Even worse, the capabilities of AI are changing and improving over time — the lines are going up on those graphs I showed you, which was not true of containerized shipping. What it would be able to do was identical in 1970, 1980, and 1990. That’s step one, I think, on why we’re going to see slow diffusion of AI, and slow productivity growth in science.

The productivity J curve

There’s one great paper here by Erik Brynjolfsson, Daniel Rock and Chad Syverson.

They wrote this less about AI than about information technology. They call this the “J curve.”

Measured productivity looks really bad with all new technologies. Why is that? In order to make these architectural changes, I need to invest some money today. I’ve still got a bunch of workers. Imagine the worker that was using the old technology now needs to learn how to use AI, so for the next six months they don’t produce any output, they’re just learning how to use the AI. Or I’ve got duplicated effort as I set up teams to experiment with using AI inside my firm. Now the inputs stay really high, the outputs fall — what happens to productivity? It goes down.

But it’s only because I’m measuring the intangible capital I’m developing incorrectly. What that worker is learning should count as capital owned by the firm. If I properly measure intangible capital, I’m getting the productivity shock from the start. What they do in their paper is run Tobin’s Q-type finance regressions and try to measure intangible capital. What they see is that the firms that get big benefits from digitization later on turned out to have very high levels of intangible capital very early on, which made them look unproductive because we didn’t include the intangible capital when we were doing our productivity regressions.

You might imagine something like this is going to happen also for AI, where we actually get a short-run decline in productivity because of the need to retool. How true is this? The interesting thing, for a PhD student especially, is where the open questions are.

• Do other big general-purpose technologies have J curves or not? We don’t know — no one’s measured it. I suspect it’s probably a general trend, but I don’t know.

• This little model I just gave you suggests that new tasks matter more for the productivity benefits of a general-purpose technology than substituting for labor and capital in existing tasks. From my read of history, if I look at the chemical industry, the industrial revolution, or digitization, I would be shocked if this wasn’t true. That said, I don’t know of any good evidence on AI, or outside of AI, that this pattern exists. Some stuff by Steve Klepper comes closest.

I think there’s a lot of nice empirical work that could be done here, looking at the extent to which new tasks really matter versus old tasks. Maybe the closest we have to this is that David Autor and friends have a really nice paper where they look at the number of new jobs that appear over time. It turns out, decade by decade, a quite large fraction of job titles you’ve never heard of before — they didn’t exist at the start of the decade. In some sense this is some evidence on what the turnover of new tasks is. But it doesn’t tell me specifically, for a given technology, how often we get new tasks.

AI in the decision problem

We’ve seen the sticky structure. What can I do, or how should I change that structure if I can make it non-sticky? Before I get there, I want to talk about what AI does in terms of creating ideas in the first place — literally, what can it do?

Another way to put that: let’s throw AI into a decision problem. We’re going way down to microeconomic theory here. There is a very simple paper by Ajay Agrawal, Josh Gans and Avi Goldfarb, related to a book they wrote called Prediction Machines. It’s one of those books written by economists that literally everyone in Silicon Valley reads at some point.

In their model, there are a couple of states of the world, and I don’t know which state is true.

I can take a safe action, but in State 2 of the world I should actually take a risky action that has a very high payoff. In State 1 of the world I shouldn’t take the risky action — it has a very low payoff. The expectation, if I don’t know what state of the world we’re in, is that I just play the safe action.

So what’s AI going to do? AI is going to tell me what the state is, with some probability. It can predict. Fundamentally, that’s what all machine learning models are doing: they’re taking in data and making a prediction. It’s a non-parametric prediction on a high-dimensional surface, but they’re making a prediction. When they make that prediction, the question is: do I want to act on that? Is it a valuable prediction? What should I do in response to it?

Ajay, Avi and Josh call this judgment, which is basically: I know my utility function. There’s knowing the state, and knowing the utility function. I take an action based on the combination of the state and the utility function. This means that across a wide part of the parameter space, AI predictions and human judgment are actually complements. As AI gets better, we’re lowering the cost of prediction. If it’s a complement to judgment, this raises the value of judgment. Knowing what to do conditional on a prediction becomes very valuable. Being able to make the prediction becomes much less valuable.

The canonical example they give of this is: take a cell phone that’s got a map on it. Up until very recently, if you wanted to be a cab driver in London, you had to study for a couple of years, know every single street, and pass this exam on something called the Knowledge. It was a very challenging exam, because London’s got all these little curvy streets — you’ll never find anything. Until you have the Knowledge, you can’t be a taxi driver. But once we have a smartphone, the Knowledge — the prediction of where to go — is: I just press a button on my phone and it tells me, even controlling for the traffic. That cost of prediction has gotten very cheap. What’s gotten very valuable is anything that’s a complement to that prediction.

We can do things within our existing system, like hit less traffic as a London cab driver — that’s mildly interesting. Or, once I get to the point where I can redesign the organization and get around organizational frictions, I realize I can just have people coming home from work pick you up in an Uber and drive a taxi for two hours with no training at all. I can redesign the entire taxi system. That’s the mapping from the decision problem into the organizational structure.

So AI is a drop in the cost of prediction. You might wonder about judgment: are humans that good at this?

We held a conference in 2017 — it was the first NBER AI conference — and Dan Kahneman was there. It was a crazy conference. I think we had 10 Nobel prize winners that attended this conference, out of about 50 people, and half of them won it since the conference. We invited the computer scientists and we invited the economists. Kahneman was giving a comment on a paper about decision-making, and he was trying to think, how should I think of human errors? Normally you think of human errors in behavioral economics as bias. But he’s like, “You should think of it like noise in the prediction function — random noise.” It’s not that the humans are necessarily in an agency problem or face some kind of behavioral constraint; it’s just that prediction is hard, and we don’t do a good job of making predictions often, as humans. He tries to take seriously this idea that it’s not that AI will have more or less bias — it’s that AI will have less noise. That’s the interesting thing about it, and he tries to draw the implications. It’s very consistent with the idea in Agrawal, Gans and Goldfarb.

If I want to map from “I have AI, it’s used in a decision problem, the decision problem is used in an organization, the organization has some frictions, given those frictions it translates those decisions into some outcome like a new idea, that outcome then diffuses, then we get growth” — I think that’s the right way to think about AI for science. We have to understand that first problem, the microeconomics of how AI feeds into a problem.

Since this paper, there have been quite a few other papers. With Susan Athey and Josh Gans, we solved a 1980s-style delegation problem: when should the AI make a decision, when should the human, and who should have the final decision rights? The interesting result there is that you may want the AI to not give its most accurate prediction to the humans all the time, because humans face agency problems.

Imagine I’ve got a truck driver down the road. 99% of the time the AI makes a correct prediction on how to drive, but I want the truck driver to pay attention in case unusual events happen. If I just give you that prediction, that’s going to dull the incentive for the truck driver to pay attention, because the probability they crash goes way down. What I want is to choose the prediction by the AI that maximizes the — conditional on the human not getting in a car crash — paying-enough-attention probability. That’s essentially what we’re going to solve for. It turns out that if I’m not just delegating but I’m designing the mechanism, I can actually do better: I don’t have to make the AI be wrong. I can just use full information, probably not surprisingly for those of you who know your micro theory.

This paper by a different Agarwal — Nikhil Agarwal, Moehring and Wolitzky — gives a really nice sufficient statistic for AI-human collaboration and shows how it applies across behavioral agency settings.

I have a paper with Josh Gans on: given that humans and AI work together on a problem, do these benchmarks that we’ve set up make any sense anyway? Surely what we want is the benchmark that maximizes the value created by the human plus the AI together, in whatever organizational setting they’re in. We show how you might be able to measure that and design better AIs conditional on their use inside organizations.

There are a lot of papers trying to get at this idea of what exactly AI is doing. If you’ve never seen these papers, your starting point should be: AI lowers the cost of prediction — and then follow from there.

AI and organizational structure

We know what AI does in the decision function. We know that organizations have sticky structure. So what should happen over time to organizational structure?

There are a couple of papers on this that I like. This paper by Ide and Talamàs is building on a Garicano-style model. That model is: there’s some continuum of possible problems. If I’m 0.8 smart, I can solve any problem between 0 and 0.8. If you’re 0.9 smart, you can solve anything between 0 and 0.9. It takes some amount of time to solve a problem, and it takes some amount of time for me to ask someone what their answer to a problem was — though less time than it takes to solve it.

With just that simple structure, I’ve generated the entire corporate form that everyone uses, with reporting structures. No agency problems at all need to be assumed here. I’m going to put the problems that everybody can solve, that are kind of trivial, at the bottom of the structure. I have all the people who can only solve problems up to 0.3 work on those. Then, as those people are working on problems and they get something that’s harder — say a 0.6 — I have them kick that up to me, their boss, and I solve that. I hire just enough people that the number of problems that get kicked up to me, and the number of explanations they tell me of the stuff they solved, exactly fill up my day.

Imagine we’ve got more and more problems and I still can’t solve them. I’m going to have someone above me who can solve from 0 to 0.8. My kid will kick up the 0.7 to me. I realize I can’t solve it, so I’m going to kick that one up to my boss and they’re going to solve that one. Now we just get this nice structure. That Garicano model tells me exactly how many employees a given manager can manage, what happens if the cost of communications falls or rises, and so on. The model explains the world with humans.

In this Ide-Talamàs paper, we’re going to drop AI into this structure.

What can the AI do? Let’s say it can predict the answer to any problem in a given set — maybe between 0 and 0.4. We’re also going to assume that all AIs are the same, so you can ask a million AIs — it doesn’t matter; they can either solve it or not. We’re going to assume the AIs are relatively cheap — you’re just paying for the API. The final thing is that any person, in principle, could query an AI, so it doesn’t have to be just the boss, it could be anyone. Sometimes we’re going to let the AI make autonomous decisions, and sometimes — for legal or other reasons — we’re going to have to have a human spend some time implementing the AI’s decision.

Let’s combine all that and see what happens. We start with this baseline Garicano model.

I know how many workers I’m going to hire — the better workers I have, the less they escalate, so the more I can manage, because it takes less time to manage them.

Now I drop the AI in.

What’s going to happen? It turns out to depend on how good the AI is, and on whether the AI can be autonomous. Let S be the set of managers. If the AI is as good as my baseline workers but not as good as the managers — it can do some of the tasks my workers can do, but none of the tasks that I can do that my workers can’t do, this is really bad for the low-level workers. I’m going to fire some of the junior workers and query the AI directly for the answer for these easy problems. My wages are going to go up; I’m super productive.

On the other hand, if the AI gets a little bit better, so now it’s middle-management quality in the type of problems it can solve, it actually starts doing the opposite. These middle managers were being paid a lot for their unique ability to do this work; the low-level people, I just needed time to throw questions down to them. So we’re going to get rid of these expensive workers now and rely on the lower-level workers to clean things up, or do the complements only humans can do if AI can’t act autonomously.

I think that’s a good example — good enough, it’s in the JPE — of someone taking organizational structure seriously, seeing what AI is going to do to who does work and who earns wages in general equilibrium, and just solving out the model.

This paper doesn’t have any innovation in it — it doesn’t have any science. So I actually think it’s pretty open, both to map this paper’s results empirically into the structure of scientific labs, for instance, and it’s quite open even just to think about the theory of what might be different. In this world the type of tasks that the organization does is fixed, but you can imagine a world in the sciences where the type of things I do look wildly different. If I have AI to do a bunch of stuff — we spend zero time inverting matrices now. The structure of our work looks very different, because inverting matrices is free. Adding those complications, I think, gets this a little bit closer to being useful.

A similar one, on how AI might change organizations, is from Shahidi et al. Their question is interesting: if you know your theory of the firm, you might be asking, why do firms exist at all? We’ve got a number of theories — residual control rights theory, resource theory, and transaction cost theory. In a transaction cost theory of the firm, what I’m doing is balancing bureaucratic costs versus the transaction cost of operating in a market. It’s somewhat difficult to gather prices, find customers, haggle, and all these things I need to do in the market, that might be cheaper inside the firm. But inside the firm it’s quite bureaucratic, and I’ve got coordination failures. So we’re balancing those two.

This is an interesting paper that says maybe the big effect of AI, at least in the short to medium run, is reducing transaction costs. If I want to buy something online, I could try to search for the lowest price, but realistically I’m not going to do it — I’m going to go to Amazon. If I could just ask my agent, “I’m buying this shirt, find me the best price that ships quickly and is reliable,” it’ll do something for 15 minutes and then buy the shirt. Now the transaction cost of doing that search is zero. If you imagine all sorts of other things — we’re already seeing AI used in monitoring of workers everywhere; simplifying bargaining, doing search, enforcing contracts — in principle, this should reduce transaction costs, allowing me to do more things outside the boundary of the organization.

This is going to matter a ton for wages — I’ll let the labor folks worry about that. But for science it also matters a ton. Right now transaction costs seem to be quite dire in limiting the ability to do large-scale science across organizational boundaries, either within the academic or the private sector. To the extent that we can verify you’re doing something useful for my firm earlier, it might open up all sorts of new scientific institutions that can contribute to innovation.

I wouldn’t say AI as a reduction in Coasean transaction cost is necessarily the most interesting thing about AI, but I do think this is a good example of taking the organization seriously, rather than just assuming I have some fixed number of scientists and that’s going to translate into growth in some way.

I would also say — combining these papers — I think this is what people are worried about in some sense, but I think it’s a good thing for science. These two results combined mean that people who have good taste as scientists — in research questions, on what to do — and are able to have a greater span of control — because they use AI for some tasks, or AI can manage other AIs — have very high marginal product.

You always hear one million Einsteins in a data center — not one million scientists. Part of it is that we think there’s a big difference between Einstein’s ability to identify interesting scientific questions, and to know it’s worth continuing down this road or not, compared to the average scientist. The average scientist does what Einstein says, and not vice versa. So you might wind up with some really interesting new organizational structures.

If you want to see some empirical papers on this that are already floating around: AlphaFold, DeepMind’s protein-folding results — AI for protein folding dominates prior approaches to understanding protein structure. We have a few nice papers:

• Carolyn Stein and Ryan Hill have a nice paper that just came out.

• My student Gabriel Cavalli has a nice paper.

• I have a few papers on AlphaFold in particular, because the shock hit a few years ago.

I think you’ll start seeing a lot of “organization of science” papers in the near future, because the optimal organization of science in response to this AI shock — there’s no way it looks like the status quo. You might think especially sectors like academia are very sticky. There’s a paper in NBER that shows a top-percentile-by-citation AI researcher — the median salary in the US is $1.8 million, in the private sector. In the public sector it’s $360,000. Why? Because the productivity of these researchers has gone way up, but academic salaries at a department change very slowly. Now we’ve opened up a 5x gap, and we can’t retain these people. Apply that stickiness to all other parts of the academic organization of science, and you’ll start to see where there are really interesting inefficiencies beginning to arise because of organizational structure.

New ideas

What new ideas are AI going to help with? There are quite a few papers on this. One I like quite a bit is by Ajay Agrawal, John McHale and Alex Oettl. What they’re thinking about is: I have a bunch of experiments I can run. Those experiments generate information about what the next experiment I should run is. I have a limited amount of time. As a human, of these million things I can do, I have some ability to rank them, but I rank them imperfectly.

What the AI can do is tell me, given where I am in this decision problem, which are the highest-value experiments to run in the short run. AI is not going to run the experiment. We’re not having the robots carry the pipette across the lab — although, if you haven’t seen an automated lab, I would recommend visiting one, because the robots do in fact carry the pipette across the lab. But we’re just going to use the AI to rank designs.

What’s going to happen is: I’m going to propose, or the AI is going to propose, what I should work on.

There’s some exponential hazard rate that I find the thing that I’m looking for. Once I find it, or if I decide it’s not worth working on anymore, I can start working on some alternative. Just this reordering of the things I work on, without changing anything about what I could have worked on, is actually substantial, for reasonable parameters, in its effect on the number of useful new ideas I get.

I think this theoretical idea is actually all over the place, including with skepticism. If you’re an empiricist, look at something like AI for drug design or drug targeting. There’s a ton of well-funded companies working on this problem. There are a lot of people in the biology and pharmaceutical world who don’t think this really matters that much. They think they have way too many targets already, and the problem is it just takes too long to run things through trials. I guess if I could find something that was more likely to get through trials, that’d be useful. Can AI help with that? A little bit. It’s not totally clear we’re at the right stage. But at least theoretically, if we were able to not start a billion-dollar trial on something that’s going to fail — it’s a huge bang for our buck in drug development, and we can move quite quickly.

This is a nice paper, because the AI doesn’t just raise the productivity of biology researchers. The AI helps solve a specific problem that I can map into parameters of existing models, and I can try to put some empirics onto this question. That’s what we want.

Bottlenecks and the Baumol effect

I just told you that the pharma people think we have a bunch of targets. The bottleneck is getting them through trials. Running the actual trials is very expensive. If AI can reduce the cost of running trials, that’s actually more important.

Whenever you have AI improve on some part of a production function that involves complements — something that’s in a nice Cobb-Douglas form — that actually just makes the bottlenecks more important. You probably know this as a Baumol effect, after William Baumol. He said it doesn’t really matter how rich we get: to play a barbershop quartet still requires four people playing instruments. You can be as rich as you want, you still need that. Even worse, I need to get those four people to not work on the high-productivity things they could work on, because our economy’s grown, so I had to pay them more than I would have paid them back in the day, even though they’re doing the exact same work.

When I’ve got these complements, the relative importance of the remaining bottlenecks becomes increasingly important over time. It’s much better, for growth, to move through all tasks equally, rather than to move through one task to the extreme and another barely moves at all. I think this is a primary driver of economists’ skepticism toward Dario Amodei saying we’re going to have 10% growth per year. After a few years of 10% growth, I’ve now bottlenecked something else. Unless I solve that bottleneck — what are you going to do?

There’s a formal model of this that I quite like, from a paper by Ben Jones. Ben wrote a really nice paper that gives me some empirics to look at. R&D — there is a continuum of things I can do. Some the machines can do; some the humans can do. We’ll make the machines as good as you want at doing their tasks, and we’re going to see what happens to growth.

• There’s the productivity of the machines, m.

• The fraction of that continuous task that AI can work on is γ.

• The bottleneck strength — meaning how complementary production is — is going to be θ, just like in a CES production function.

What we’re going to do is: you tell me, this is my research objective, these are the things that are necessary in order to do the task. I can estimate from current data what θ is — how complementary they are. I can estimate from your projections what AI can do — what m is going to be and what γ is going to be. Now I can just tell you what growth is going to be, at least in this sector.

In this model, the task-level output — per task, on things I’m going to use the AI for — is going to be the quality of the AI times the current state of that task. On things I’m going to use the human for, I have to use some amount of human labor in order to do that task. Let’s use a nice CES-style aggregator.

What I’m going to do is look at the total contribution of the machines to growth. It’s going to combine the machines’ quality across these tasks that it knows how to work on, and then adjust for the fact that it can’t work on some tasks. As long as the machine rent is not too high, then we’ll use the machine — but it doesn’t really matter, because I’m going to let the machine quality go to infinity.

Now I can just put numbers on this, and it’s quite surprising.

The rate of growth depends on the machines’ expenditure share in R&D, and the expenditure share turns out to be really constrained by this complementarity. I get really good at inverting matrices, but I still — when the mouse escapes that I inject a cancer into — I still have to run across the room and catch the mouse as a human. A little bit more improvement in the speed of inverting the matrix barely matters, because the whole constraint is: every time a mouse runs away, I need to catch it.

To give you some numbers: let’s imagine we have a nice harmonic average across tasks — that would be θ equal to −1 in a CES. The overall productivity is a harmonic average of my quality on each individual task. Then, with reasonable assumptions, letting infinite quality for the machines, we wind up getting growth of 2.25 times today’s productivity. So, 10 times intelligence, 100 times machine intelligence — it’s not 10 times growth, it’s not 5 times growth, it’s 2.25. The only real way out of this is that we kill off all the bottlenecks. Even a small number of bottlenecks on machine intelligence in the science production function starts to bite really hard.

Conclusions

I hope you can see there’s just a ton of open questions.

• I’ve got these existing organizations. How quickly are they going to be put out of business by more productive firms? Normal competition policy, IO.

• When the new firms do things, how are they going to use AI? That’s for you micro theorists, in the decision function.

• How am I going to incentivize people, and how is that going to map up into the structure of the new organization? That’s for you organizational-econ, IO folks.

• Given that new organization, and its mixture of humans plus AI, and taking the projection seriously about what the future looks like from the computer scientists — how is that going to map into new ideas in each sector? How is that going to map into growth?

• What can we say at each stage of those things empirically right now that’s informative, given those theoretical models of where we’re going to wind up in the future?

If I were a student right now interested in innovation — hopefully I’ve convinced you that a lot of the interesting innovation questions are going to touch AI. Those lines on the graph we started with, they’re still going up. The million Einsteins in the data center will happen. It’s just that the implication of that might not be what a lot of folks who aren’t economists think, and we have a ton of scope for helping people understand things — even understand that in the short run we might see declining measured productivity because of that J curve I showed you. It’s quite an interesting time for this type of research.

Ben Reinhardt and Dan Recht responded to Jordi’s article; they’re less sanguine than he is about the likelihood of National Labs transforming. Jordi was kind enough to respond to their response, which means that we have our first Macroscience debate! Whatever your baseline appreciation for existing institutions or desire for new ones, I hope this debate helps you hone your views and sharpen your arguments.

The views expressed in this piece — by Ben, Dan, and Jordi — are their personal opinions and do not represent the views of any of their employers.

Subscribe now

Ben and Dan respond:

Jordi Cabana argues that proponents of focused research organizations (FROs), X-Labs, and other new institutions have missed something big: in our excitement about creating homes for work that is a poor fit for both academia and companies, we’re forgetting that we already have an experienced, well-funded institution whose mandate is exactly that work — the National Labs.

To the contrary, we haven’t forgotten about the National Labs. But instead of asking “what are the National Labs supposed to do?” we should ask “what do the National Labs actually do?” When you ask that question, it becomes clear that the National Labs are poorly suited for the roles we’re creating new institutions for. And, while in theory you could reform them, in practice real reform will only come through competition with other institutions.

The National Labs’ actual activities can be roughly divided into three buckets:

1. Curiosity-driven research that is similar to academic work.

2. Running very large, expensive facilities like the Advanced Photon Source at Argonne, supercomputers, and the National Ignition Facility.

3. National security-relevant research that is too sensitive for any other organization.

(There are, of course, many corner cases that don’t fall into those three categories. Within any billion-dollar budget there will always be outlier activities.)

These functions are valuable, but they’re far from the purpose of an FRO or X-Lab.

Let’s go through Jordi’s points one at a time:

Point: The National Labs’ mission is “both to advance fundamental science and to occupy a missing middle: research that is too complex for a single principal investigator grant, but too risky or oriented on public goods for a startup or a large industrial incumbent to take on.”

Counterpoint: Realism and history argue against this: The National Labs were built to create and maintain the United States’ nuclear arsenal and peaceful uses of nuclear power. Everything else they do has been bolted on over time. Successive statutes added additional mandates like supporting other forms of energy and technology transfer, but the core institutional structures were built for a different era and mission.

Point: The National Labs have a large budget (and presumably that implies they can do a lot of good work).

Counterpoint:

1. Research is a domain with weak correlations between funding amounts and output. Furthermore, there is a lot of evidence that research has been getting less productive over time, so the burden of proof should be on any legacy institution to show that they are deploying money effectively.

2. The vast majority of that large budget is not available for the flexible spending that’s at the core of the new institutions. Beyond the 40% of the budget that goes to weapons maintenance, much of the National Labs’ money is tied up in project-specific funding (which the new institutions are meant to circumvent) and continued funding to large facilities (which are important, but not the point of X-Labs or FROs).

Point: The structure of National Labs allows them to perform many of the functions associated with new institutions for science.

Counterpoint: Jordi asserts this without actually talking about the National Labs’ structure and incentives. First, US law caps National Labs’ spending on self-directed R&D at about 6% of their budgets, so the main way for a National Lab to fund “work of a creative and innovative nature” is to grow ever larger. A subset of Labs achieve this percentage but most do not, with the total spend across all National Labs only reaching about $1B in 2025. For comparison, Alphabet’s “Other Bets” segment, including their X moonshot lab, Waymo, and other long-term R&D efforts, lost $7.5B over the same period.

Second, National Labs serve too many masters. At minimum, each Laboratory Director must answer to (1) an independent management and operations contractor (M&O) that can fire them, (2) the DOE site office that enforces laws and regulations, AND (3) sponsoring offices within the DOE that fund their Lab’s work. This structure results in a range of idiosyncratic behaviors (some due to legislation, some due to perceived legal risk). It’s cute that the Labs tend to serve cake and not beer at parties because alcohol is an unallowable use of government funds.1 It’s darkly funny that Lawrence Livermore National Lab got in serious trouble for trying to promote itself by blowing things up for a National Geographic documentary about the sinking of the Lusitania because those explosions competed with the private sector. However, it’s a real shame that a National Lab researcher hoping to form a spinout around their own invention (a route that is formally encouraged by the DOE) is required to go through the Lab’s (typically lengthy) technology transfer process, the M&O contractor’s conflict of interest process, and multiple layers of DOE approval to do so. To cut this knot, National Labs have been standing up their own non-profit foundations to facilitate technology transfer and sponsor work (and beer) that the government won’t.

Subscribe now

But really, the biggest counterarguments to Jordi’s piece are his caveats at the end:

• Despite asserting earlier that the National Labs are not academia, he admits that their current model “may lead to mimicking the academic approach.”

• The National Labs have structural barriers limiting technology transfer to the private sector, including “intellectual property agreements and security clearances.” They typically see the government itself as their primary customer, especially in their core nuclear mandate.

• The National Labs, like any organization operating at their scale, have significant institutional baggage, which is one of the main reasons to create a new institution.

• He concludes that all of this is baked into the core of the Labs because they are built around maintaining and improving the nuclear arsenal!

The argument that “we already have a perfectly good institution, why do we need a new one?” is an example of a common mistake in metascience (and the world of new organizations more generally): the assumption that because an institution is chartered to perform function X, we should count on that institution to perform function X well and do not need more institutions that are trying to do X.2 The maximally ridiculous (but unfortunately common) version of the fallacy is “well, because there is a Department for Good Stuff we shouldn’t start any new organizations to do Good Stuff and we should count on them for all efforts to do Good Stuff.”

It’s clear that the argument is a mistake in business: “Yahoo already does internet search, why do we need Google?” “Nissan already makes the electric Leaf, why do we need Tesla?”3 The mistake is more subtle in government and nonprofits, where overlapping organizational roles can be more confusing and wasteful. However, many organizations do not actually do what it says on the box well, or at all.

Institutional gaps can be filled either by reforming old institutions or creating new ones. Which approach is best depends on the situation and the domain. In the domain of the military, for example, you generally want the former because new institutions are called “private militias” or “a revolution.” In the business world, you generally want the latter because startups can strengthen competition and engage in creative destruction (though old companies certainly do reinvent themselves).

We believe that federally funded research looks more like business than the military. Institutional diversity in research organizations enables many parallel experiments. Overlapping research efforts are actually good. And frankly, 21st century politics does not seem capable of the reforms that would bring the National Labs in line with the idealized vision of them.

Finally, reform and creation are not as diametrically opposed as it might seem. New institutions can serve as laboratories, testing out new approaches that legacy institutions can copy. Legitimate competition from new institutions can drive reform far more effectively than some vague mandate from above.

Subscribe now

Jordi counters:

With “Metascience Is Ignoring the National Labs,” I set out to correct the fact that every piece I read advocating for new R&D institutions heavily relied on the university model as a reference point while treating the US National Laboratories, at most, as a note in passing.4 I wanted to nudge the authors (and others) into explicitly assessing whether the National Labs already reach or could conceivably reach some of their goals. Thus, I must thank Ben Reinhardt and Dan Recht for taking the time to respond.

Ben, Dan, and I agree that the current moment demands experimenting with new institutional models. We even agree that universities and National Labs do not currently give us everything we need. Where we diverge is that I think the Labs have a lot to offer, both as institutions and as case studies to inform new institutional design. Their response makes two broad arguments: first, implicitly, that we already know what we need to know about National Labs, and second, explicitly, that they are not what we need. This response will focus on the first argument. I will refer readers to my piece to decide on the second point.5

Ben and Dan state that my “assumption is that because an institution is chartered to perform function X, we should count on that institution to perform function X well.” Yes, we should count on an institution to do what we pay it to do, and that includes verifying success! In the case of the National Labs, we need to know how to get all our $30B worth. But my conclusion is not to then say we “do not need more institutions that are trying to do X.” Rather, it is that new institutions should learn from why existing institutions are not doing X as well as we want.

Ben and Dan argue it is a “common mistake” to say, “‘we already have a perfectly good institution, why do we need a new one?’” I could not disagree more; I do not think this is a mistake at all. Isn’t asking whether we need a new entrant what any good VC does in the business world? Google not only answered the question, but it succeeded because it answered it so well! Do we remember other human-curated portals like AltaVista, Go.com, or LookSmart? They failed by playing the same game as Yahoo!

Subscribe now

Ben and Dan appear to see the challenges of National Labs as practically insurmountable. I am not convinced the prospects are so bleak.

Yes, National Labs can sometimes hew too close to academia. But there are also many parts that indeed occupy the “missing middle” and do not worry about publications. Look at the applied programs for examples. Yes, a large portion of National Lab funding is tied to specific projects. But did you know that, not so long ago, large parts received de facto block funding with local discretion, just like we want for FROs?

The challenges are not set in stone. We need rigorous metascientific analyses to know what can and cannot be readily improved.

Relatedly, Ben and Dan correctly diagnose that National Labs are hamstrung by their federal masters, and they write that “21st century politics does not seem capable of the [necessary] reforms.”

Any FROs that are funded by the taxpayer are going to have the same masters. What is going to stop them from imposing the same seemingly insurmountable shackles? The National Science Foundation’s recently launched X-Labs also won’t be able to use federal money to buy beer.6

I will echo Ben and Dan’s final argument. A fantastic reason to experiment with new institutions is that they “act as laboratories for trying new approaches that legacy institutions can copy.” One reason I am excited by NSF’s X-Labs is that, because they are not like typical grants, they will be free from some of the rules and processes that hamper many other programs. We need to analyze how much this flexibility plays a role in the outcomes of X-Labs and how to extend it to existing institutions and other agencies like DOE. We should also flip the script and use what we observe in National Laboratories to direct future experiments.

There is much room to experiment with and learn from the National Labs. How much of each will not be settled with a blog post and a two-part debate. Just as metascience can support new institutions, we need to apply the same level of rigor and vigor to understanding the ones we have.

Subscribe now

While it’s easy to connect specific research to innovations, and those innovations to improvements in well being (Gila monster spit → ozempic → weight loss), doing so at the level of an economy is difficult.

As Matt Clancy discusses in his lecture, part of the challenge of measuring the returns to research and development (R&D) is weighing the lags in when research pays off and considering these alongside discount rates. The economic benefits of different types of R&D vary by when they occur, with technology development providing economic benefits almost immediately and basic research taking 20 years to have the greatest impact. I find this chart from Matt’s presentation helpful in visualizing when we might expect payoffs from various types of R&D funding.

Subscribe now

While not the focus of Matt’s presentation, this visual makes clear a political economy problem for governments that want to fund innovation. While an optimal strategy might be to patiently fund basic research that is unlikely to be supported by the private sector, it’s unlikely that political actors will be rewarded for this in a reasonable time frame. So they may be tempted to focus on development activities that can provide quick wins (but which rely on earlier basic research).

This gives us another reason to try to measure the economy-wide benefits of R&D: perhaps the measurements can help convince political actors of the long-run benefits of research investments. According to Jones and Summers, the social returns to R&D are around 67% annually. Fieldhouse and Mertens suggest that returns to federally funded, non-defense R&D range between 140% and 210%. Those are ginormous public benefits.

Matt has been interested in the returns to R&D for quite a while. His blog New Things Under the Sun has covered topics such as public R&D funding and productivity growth and the value of knowledge spillovers. Its Frequently Asked Questions section is the only FAQs section I can think of that I regularly send people to read.

Subscribe now

Here are the readings that go with this lecture:

• Azoulay, Pierre, Joshua S. Graff Zivin, Danielle Li, and Bhaven N. Sampat. 2019. Public R&D Investments and Private-sector Patenting: Evidence from NIH Funding Rules. The Review of Economic Studies 86(1): 117-152.

• Dyevre, Arnaud. 2024. Public R&D Spillovers and Productivity Growth. Working paper.

• Fieldhouse, Andrew, and Karel Mertens. 2023. The Returns to Government R&D: Evidence from U.S. Appropriations Shocks. Federal Reserve Bank of Dallas Working Paper 2305.

• Fieldhouse, Andrew J., and Karel Mertens. 2025. The Social Returns to Public R&D. NBER Working Paper 33780.

• Myers, Kyle, and Lauren Lanahan. 2022. Estimating spillovers from publicly funded R&D: Evidence from the US Department of Energy. American Economic Review. 112(7): 2393-2423.

Subscribe now

Thanks to William Higbie and Harry Fletcher-Wood for their support in producing this video and the transcript.

Lecture Transcript:

The Returns to Science

Iowa State University is where I’m from. I did science policy when I was at the Department of Agriculture. I had a weird career. I created a living literature review on the economics of innovation that I worked on for a long time and got pretty well known. Eventually I joined Open Philanthropy, and then joined Coefficient Giving to run their innovation policy program. About a year ago I took over a fund called the Abundance and Growth Fund, which does innovation but other stuff too. Since I’ve left the university, doing the research hasn’t been as much of a focus, but I stay up with New Things Under the Sun. I did get one paper out last year with some of your teachers here — Pierre was one of them — what if the NIH had been 40% smaller? I’ll talk briefly about that later.

The other thing I wanted to flag is that at Coefficient Giving we’re starting this project called “The Pop Up Journal.” A pop up journal is a bit of an experiment in a new kind of science communication platform or format. It’s a project that lasts for a fixed period of time — five years. It’s going to be focused on trying to answer a specific question, and our first one is going to be, “What is the social return on publicly funded R&D?” We gave a grant to the National Bureau of Economic Research (NBER) to run this alongside the Alfred P. Sloan Foundation, and they’re going to have a series of conferences, commissioned research papers, and surveys. We’re coordinating research funders to provide support to do new research on this. The idea is: normally when there’s a special issue of a journal or a conference, it’s announced about a year out. If you’re working on something in that area, you go for it, and if not, you’re out of luck. This is — what if we do a five-year thing, so you have enough time to think it through, start a project, and see it through?

Why we care about the returns to science

Let’s motivate things. Why do we care about the returns to science? Let’s put on our economist hats first and abstract away from politics. We’re a benevolent social planner. If we’re going to choose the optimal level of R&D spending, we’ve got to know the answers to some questions:

• What’s the return on investment from public R&D?

• How does that compare to alternative uses of tax revenue? You could imagine spending it on lots of things that are public-good in nature, and growth-promoting.

• What’s the deadweight loss from raising this through taxation?

• The marginal return on a dollar of R&D — so we can tell if we need to spend more or less.

This is a reason to understand the ROI of R&D. In practice, people’s answer is that it’s high — but you can’t get much more specific than that, or that it’s greater than one.

Returning to the real world: government is not well modeled necessarily as a benevolent social planner, and was not persuaded to begin supporting R&D with public resources purely on the basis of academic research that showed the ROI was high. Instead, science proved its merit as important for national security and so forth in World War II. It directly contributed to atom bombs, radar, and wartime medicine. Vannevar Bush took the insights from that period and applied them to a vision for what science could look like in peacetime, saying: “We’ve seen how powerful it can be during war; the same kind of benefits can flow to us in peacetime if we continue government support for research.”

On the back of that — not just Vannevar Bush but other people too — the National Science Foundation (NSF) was created. The National Institutes of Health (NIH) had already existed, but it was dramatically ramped up and its role became this central hub for funding biomedical research. The role of the modern research university, and how it interacts with the government in terms of getting research support, kicked off here. For the last 75 years or so, there’s been this bipartisan consensus — although not always unambiguous. That’s my little potted history.

In the last couple of years, there’s been some stress on this consensus. This comes from the Trump administration proposing very large budget cuts to government support for some of the agencies — the National Institutes of Health, the National Science Foundation — proposing these 40% cuts; a 60% cut to NSF; overall one-third of non-defense R&D, which would have taken R&D spending down to the lowest levels it’s been since the ’90s.

That was just the president’s budget. The president does not actually dictate policy; he proposes it. Congress decides what it’s going to appropriate, and they largely rebuffed these proposals. NIH actually saw an increase. National Science Foundation did see a cut, and a 3.4% cut is unusually large. We’ll look at a paper later that defines large cuts as anything greater than 2.5%.

Measuring the returns to science

Before this had all shaken out, and we were unsure where things might go, I wrote this paper with Pierre Azoulay, Danielle Li, and Bhaven Sampat: “What if the NIH had been 40% smaller?” The idea was to do a very simple exercise that somebody could understand without a lot of econometric training. Pierre, Danielle, and Bhaven had access to the actual grant applications from the NIH going back to the 1980s. We could see the peer-review scores for those and how they were prioritized, and then we could chop off the bottom 40% of these grants — assume they don’t happen going back to the 1980s — and ask: what would have happened? What kind of research acknowledged those grants? What kind of drugs cited papers that acknowledge those grants, and so on. It was a descriptive paper. Our conclusion was that there’s no sign this research was just ignored and didn’t contribute anything. Over 50% of drugs had some kind of connection to these counterfactually cut papers.

That’s contemporary history. Now let’s return to our economic framework and think through how we might take into account some of these real-world things. Rather than thinking of the government as this benevolent social planner unconstrained by politics, questions about the return to R&D can matter in a model where you’re thinking of the policymaker as constrained in various ways.

• How long does it take for the benefits of R&D to become manifest? Things that mostly occur after a politician’s tenure are going to be harder for them to justify as an electoral strategy to stay in office.

• Geographic spillovers. If we think most of the benefits of R&D accrue to places outside the local government that’s making the spending, that might make it harder for politicians to support R&D in their area. They have to be thinking more about constituencies that are not directly voting for them.

• Legibility. If you support R&D and there are benefits, is it easy to draw a line from the benefits to the R&D that you supported? If not, it’s going to be harder to get credit for the investments you made.

• Steerability. If there are specific national priorities, how well can you use research to target those — versus how much is research just completely unpredictable, with the benefits mostly arising from spillovers to unanticipated uses?

Let’s talk about how science is maybe different from other parts of returns, or R&D in general. I’m going to focus not on science specifically but on publicly funded R&D, which is to a greater extent what we would call science. This is some data from Arnaud Dyèvre’s 2024 paper.

He’s got the share of public R&D that is spent on basic research, applied research, or development, and then the same thing for the private sector. You can see that publicly funded non-defense R&D is much more concentrated in basic and applied research than the private sector. You can also see evidence of a greater reliance on science in publicly funded R&D: the patents that belong to government agencies cite scientific papers at a much higher rate.

It’s useful to clarify that even though government spending is much more concentrated in basic and applied research, because the government is maybe 20% of R&D funding relative to the private sector in the US, in terms of the actual total amount of funding for basic research it’s more neck and neck than would be implied by this figure.

Why do we think R&D, or science particularly, is interesting? We can look at a couple of facts that Arnaud highlights about publicly funded R&D. He focuses on the patents of government agencies — government agencies occasionally do get patents — and he compares them to private-sector patents. Public-sector patents he calls more ahead of their time. The patent office is always updating its classification of technologies, and when a new technological category comes along they might create a new classification to put patents into. If you get reclassified into one of these, that’s a signal that you were a cutting-edge patent — a patent that later people were like, “I see now: we didn’t know it at the time, but that was one of the first quantum computing patents, and now we know to properly classify it as a quantum computing patent.” This happens more often for patents with a government acknowledgment.

They’re also cited by a wider range of technology classes than private patents, and they’re disproportionately useful to small firms rather than large firms. This indicates there’s this value of spillovers that you wouldn’t normally get — and large firms can do the research they need themselves, is one interpretation.

A second paper that tries to highlight why we think science in particular is interesting to study — it’s different from regular R&D in general — comes from Krieger, Schnitzer, and Watzinger. They’re also going to look at patents. They’re going to look for patents that are more and less directly connected to science, looking at all patents — not a comparison of government and non-government patents. They’ve got data on these patents from the ’80s to the 2010s. They’ve got estimates of the value of these patents through this paper by Kogan et al., where they look at how the stock market moves when a patent grant is announced.

Then they’re trying to define how closely related a patent is to science based on citations. If a patent directly cites a scientific article, they’re going to say this is a “Distance 1 patent” — they’re adjacent to each other. But a lot of patents don’t cite any scientific articles. Some of those cite a patent that itself cites a scientific article; they’re going to call this a “Distance 2 patent.” Some of them go one step further: they don’t cite any science, they don’t cite any articles that cite science, but they cite a patent that in turn cites a patent that cites a scientific article.

Here’s the takeaway.

We’ve got the distance from science on the horizontal axis, and the premium as measured by these stock-market returns of the patents in these different distance categories. The closer a patent is to science, the higher its valuation.

Dissecting the challenge

We’ve motivated why we might be interested in studying the returns to science. Now let’s talk through why this is a challenging thing to do. To start with, let’s go back to the Jones and Summers paper.

They have a simple framework where you’re imagining that GDP per capita grows at rate g over time, conditional on devoting a share s of resources to R&D, and you do that in perpetuity.

They say: “What if we paused R&D for one year? What would be the benefits and the costs?” The costs would be: we lose growth for one year, then in the future we’re always a little bit poorer than we would have been otherwise — because they assume all growth is coming from technological progress — but we don’t spend as much money for that year, and in future years we also spend a little bit less money, because we’re devoting a constant share of a lower GDP.

They do this and they cash out. There’s the benefit-cost ratio; you do some algebra and it works out to g/sr. You can plug in values for these variables based on long-run averages: 1.8% per-capita growth if you go back to the 1950s; about 2.5% of GDP spent on R&D in the US. If you use a 5% discount rate, you get this benefit-cost ratio of 14.5 roughly. So every dollar you spend on R&D generates $14.5 of present value in benefits.

There’s a bunch of assumptions baked into this. One is: which benefits are we going to actually count? What I just went through is the benefits to the United States.

But you could imagine the benefits being on a spectrum — from the benefits that flow directly to a firm that’s doing R&D, to an industry, to a country, to the entire world. If we’re thinking about an individual firm that’s doing R&D, and we’re thinking that they’re in equilibrium — where they’re going to keep investing until the returns from investing equal their outside option, then we would expect the benefit-cost ratio to equal one — because they’re using the same interest rate to discount as they get. So that’s what we should expect the private benefits to be.

The industry could be something in the middle, or could be something different, because this equilibrium argument about firm behavior won’t apply at the level of an industry: you’ll invest until your private benefits equalize the costs, but there might be benefits that spill over to other firms. At the same time, there might be costs because you’re stealing their business.

On the global side, one of the exercises that Jones and Summers do is use similar numbers for the growth rate for the OECD — it’s not that different — but the share of global resources devoted to R&D, and they get roughly 31.8. We’re going to look at some empirical papers that you can think of as sitting on different points on this line.

Another assumption you have to think about: what kinds of benefits are you going to count? The Jones and Summers thought experiment is basically just income benefits from growth — it’s premised on this GDP-per-capita growth rate. All the empirical papers we’re going to read are also only going to focus on that kind of benefit from R&D. But I wanted to take at least one slide to pause and note there are other ways to think about this.

The environment — carbon emissions going down, extra leisure, the value of having access to new goods — these seem positive — a little bit hard to quantify. Although there was a really interesting paper by Phil Trammell that was trying to quantify the value of access to new goods and some other things, that showed that the traditional benefits of growth, when we’re basing it on income, really understate how much value we’re getting. He looks at the value of a statistical life and shows that if you fit a model to that, it implies the growth of utility has gone up much more than the growth of income would imply, which is super interesting.

Another thing that Jones and Summers do is look at health and longevity. One of the benefits of R&D is that we get years of longer life, and we could put a dollar value on that too if we wanted to. Jones and Summers do a couple of different exercises around that; in one of them they get up to 23.2 if you start to incorporate the benefits of R&D.

Lastly, you could think about population growth being valuable itself. This is more philosophically treacherous terrain to dabble in. If you think each person’s subjective well-being is additive to the total utility function, then being able to support a larger population is itself valuable. You could imagine using a benefit-cost ratio where, instead of GDP per capita, you’re just using GDP — so you’re multiplying by the number of people, not just income per person — and you can calculate other things that tell you how you would value that.

Another angle about benefits you have to think about more carefully is when do they arrive? R&D that’s done today, especially science, does not raise productivity today. The benefits tend to arrive over a long time. We’ve got a couple of different lag structures illustrated here.

The blue line is what is often assumed in empirical R&D papers, where the benefits are strongest today, then there’s this rapid depreciation of the rate of knowledge — often assumed to be the same as the depreciation rate of capital. Ideas that were developed 10 years ago are generating a lot less value than ideas generated today. That’s probably a truer assumption if you’re looking at the private sector, where their R&D is more development and closer to impact. But for science, it’s probably a worse assumption.

Some of the agricultural economics literature — because I used to be at the Department of Agriculture — has thought a lot about this in the context of public investments in agricultural R&D. They use these black and dotted-red-line structures, where the benefits are small at first — you can think of a new kind of seed variety diffusing out, being used by more farmers and having more impact over time. Then over time the value of that research declines. In agriculture it could be because pests evolve and are now reducing the yield of that new seed variety, or because new, better seed varieties have come along that make use of the one you developed earlier.

What if we want to incorporate timing and lags into this benefit-cost ratio? Jones and Summers talk about how you could do that.

You just add a discount rate to the growth, which is like the benefits you’re getting from R&D. You could go through an exercise where you assume development takes five years, applied research has a 10-year lag, and basic research has a 20-year lag. If we apply a discount rate of 5%, we get these different benefit-cost ratios for these different kinds of research. Assuming the growth benefits are equal across them, stuff that occurs farther in the future is going to be worth less. If you take a weighted average based on how much R&D is spent on development, applied, and basic research, you get this updated benefit-cost ratio of 10.

We had this slide showing that the benefits to R&D could decline over time past a certain point.

Why is that? It turns out the knowledge-depreciation-rate question is important. Whether you’re thinking about the private or the social return of R&D matters, because these can move in different directions.

If you’re a private firm and somebody copies your idea, your knowledge is now worth less — to you, the value of that knowledge has declined. But it might be that the social value has begun to go up, because now access to the idea is wider, consumer surplus will rise, and so on. Similarly, when your patent expires, you lose your exclusivity — you could imagine that the value of that research you funded has completely dissipated to a private firm. But on the other hand, that is when it enters the market for generics and lots more people might get access to the R&D that was done.

On the other hand, you can imagine the technology becomes obsolete — a similar problem if you’re a private firm, but in this case the social return might go in the same direction. Society at large is also not benefiting anymore from the R&D that was done. Even here, though, you have to think: did the idea go obsolete because the technology that supplanted it came from a different angle and did not build on the ideas itself? Or was it an improvement that built on those ideas — they were fundamental and necessary to get to that next step? If you’re in the second case, it’s not clear that the ideas really ever fully dissipate, because you couldn’t have gotten the new, better technologies without first developing these old ones.

I also wanted to highlight that some papers take a different approach to this whole question. I’ve been framing this around — we do the R&D now, and the benefits come in the future. But we could also think about the reverse, and some papers do this — here are the benefits, they’re coming in today, we’re measuring something like the number of clean-energy patents that were around in 2008 or 2009. Rather than looking forward, they look backwards and construct an R&D stock, saying the R&D stock today is the useful knowledge available to the firm today to invent things, and that is a weighted average of previous R&D investments.

This is a pretty common approach.

They usually do a standard depreciation rate for R&D investments that happened in the past. It’s often assumed that you use 15% or 20% as the depreciation rate, which is similar to the depreciation rate on capital. In a lot of these applications, the exact choice doesn’t actually matter that much — if you difference the data, for example, then it doesn’t matter that much. But in practice, that’s often what people do.

Another issue: what’s the denominator of this benefit-cost ratio? What kinds of costs should we be thinking about as relevant?

Here, I’ve divided the cost to realize the benefits of R&D into four different pieces.

1. There’s the R&D that was done by perhaps a specific scientist of interest — maybe we gave a grant to a scientist and we want to know the return on that grant. What happened that wouldn’t have happened when we made that grant?

2. Should we also think about the cost of the enabling R&D stock that was done prior? In practice this isn’t usually counted. As a specific example, you can imagine AlphaFold as having two costs: the cost from Google DeepMind to build it and train the model, which would give you one return on investment depending on what you think the benefits of AlphaFold are. Or you could add in the billions of dollars that were spent on assembling the Protein Data Bank, on which it did all of its statistical machinery.

3. A second way you could go: there could be induced private R&D. AlphaFold is this useful technology, but it isn’t going to generate commercially useful results without further fine-tuning; maybe it needs some scaffolding and some things to be used, and maybe other firms build on top of it, or maybe Google invests more before it finally gets there. It’s often the case that science funded by the government, or private-sector science — you can’t just pick it up and turn it into a product; there’s a lot of applied development and research that comes after the fact, and you might want to count that. But you could also think of ignoring it, in the sense that the scientist’s own R&D is counterfactually impactful: if you hadn’t made that grant, you wouldn’t have gotten the benefits. It depends on the application you’re doing.

4. The last piece is the cost of not even R&D, but just the costs of investment to realize the benefits. The COVID vaccines are a good example: it wasn’t that hard to figure out the code of the mRNA that was going to be used for the vaccine, but to actually get the drug in everybody’s arms, you had to invest in these giant manufacturing and distribution services. Jones and Summers think a little bit about this and say: what if we think of capital deepening as part of the cost of R&D? Capital deepening, they argue, you can use as a proxy for this investment. That’s maybe 4% of GDP in addition to the 2.5% that’s spent on just R&D, and that cuts your benefit-cost ratio down to 5.5.

Lastly, we can think about adapting Jones and Summers to estimate the benefit-cost ratio of science specifically.

Their default is about all R&D. But let’s think about if we cared about, “What is the return to science?” A simple way to do it is to modify their equation two ways. On the numerator, we’re going to add this μ term, which is essentially: what share of the benefits from R&D are attributable to science? We’re going to say, “If you just stopped science rather than all R&D, how much would that affect the growth rate?” Maybe it wouldn’t; maybe we would get most of it; maybe we would get a little bit. For illustration I picked 25% or 75%, just as illustrative bounds.

There’s also the question of how long the lags are until we get the benefits of science. Again for illustration: if you assume it’s five years, which is common in these literatures, or 20 years, which is more in line with the historical evidence, we’ll look at how that affects the answer. Lastly, λ is the share of science in all R&D. If we spend 2.5% of GDP on R&D, roughly 15% of that is going to be basic R&D, which is the closest analog for science.

I’m not saying these are canonical — these are just illustrations of how these different assumptions matter. If you assume that science accounts for 25% of growth but it takes 20 years to show up, and you use all the rest of these variables the same, then the return to R&D is about 10. If you’re a real optimist — that science drives everything and it doesn’t take that long in the modern era — then you could get returns of 60-plus.

This is the set of problems we’re going to have to deal with. Jones and Summers is really informative that the returns have to be at some high level. But we’ll feel more comfortable if we have some empirics to back that up. That’s what we’re going to turn to now: a couple of empirical papers that try to estimate something close to these benefit-cost ratios.

What do the empirics say?

We’re going to look at four papers, and they’re going to go from most zoomed in to most zoomed out:

• Myers and Lanahan — the most zoomed in. This looks at the Department of Energy, and a specific grant program within it: the Small Business Innovation Research (SBIR) program.

• Azoulay et al. — zooming out to all the NIH extramural grant funding, looking at biomedical innovation.

• Arnaud Dyèvre — R&D levels across major agencies in the United States, tied to the productivity of individual firms.

• Fieldhouse and Mertens — total non-defense R&D spending by the government and total aggregate national total factor productivity changes.

One problem we haven’t totally left behind — a classic problem that is not unique to R&D — is identification. In the R&D context, the specific challenge is that R&D is not randomly allocated. If we just look at places that got R&D and what the benefits were, and compare that to places that didn’t get as much R&D, that’s going to overstate the value of R&D. We have these entire peer-review processes to try to find the most promising places to stick our R&D dollars: the places that get more R&D dollars we should expect to have higher returns anyway.

We need to find ways that R&D varies that are not correlated with what’s often called technological opportunity. The papers we’re going to look at have a couple of different approaches:

• Program rules and discontinuities — getting into the Byzantine details about how agencies make funding decisions and trying to find situations where it’s quasi-random.

• Policy variation that shifted funding for reasons uncorrelated with science.

• Reading through history — trying to figure out narratively why different changes to R&D happened, and highlighting ones that were uncorrelated with what we think are major confounders.

Myers and Lanahan: Department of Energy SBIR

Let’s start with the Department of Energy’s Small Business Innovation Research program. These are competitive grants given to small firms that are responding to specific technological solicitations. This is a little bit different from the NIH, where you might be able to just propose an idea. This is more like: the Department of Energy is looking to fund solar energy, or looking to fund battery technology, and they’re going to announce a grant program around this — $200 million a year.

What’s interesting is Myers and Lanahan are not looking at what happens to a firm specifically when it gets funding. They’re looking at what happens when a technology area gets more funding. We want to look at both that area and — if we think spillovers are important — what happens to closely related technological areas, especially by firms that didn’t get a grant. We might fund a firm to work on solar technology, but maybe that will have applications in battery technology.

Why are they interested in this? Because these funding opportunity announcements about what kinds of technology we’re looking to fund help you pin down exactly what we’re trying to fund. The Cooperative Patent Classification system lets you map how close different patents are to what the agency was trying to fund. Then they use natural language processing to link these two up without citations. This is a 2022 paper, meaning it was written a couple of years earlier, so when we say natural language processing, it’s the old-school type, not the new large language models.

We still need to solve that identification problem. They don’t want to look at what the Department of Energy funded. Instead, they’re going to look at the states that these firms reside in, because some states have programs that will match or partially match an SBIR award. These matches are not subject to the same selection criteria — they don’t match if it looks particularly promising or not; they just match if you got an award. They argue that whether you get this windfall depends less on how good your technology is, and much more on whether firms that win happen to be located in states with these programs in the years those programs are running.

So there’s a cash drop into solar technology R&D because it just happens that somebody won an SBIR grant who was based in Iowa in the year Iowa had a program like that. That’s their argument. They do some work to try to show this is quasi-random: the states that are matched don’t tend to be secretly more productive anyway; firms aren’t relocating across borders to try to get into match states; and some major states that we care about don’t have these match programs.

Here’s what it looks like over time.

You can see in 1997 just two states — we’ve got Hawaii there. In 2007, 10 years later, a couple more states. In 2018, still some more. It’s also notable that we have some programs that shut off. Here we’ve got this guy — I’m terrible with geography — he’s in the club, and then in 2018 he’s out of the club. So it’s not just that we keep adding every year.

To measure the spillovers, normally what you might do is say: “This firm won a grant, it got a top-up, we look at the patent classification system for the patents that it took out, and we look at who cites patents in that.” But they’re worried that citations are a bad proxy for how related technologies are to each other. There’s a lot of evidence that firms’ citation of other patents is a really noisy indicator that can be misleading. You don’t cite a patent for the same reason you might cite an academic paper. Academics are used to thinking, “I understand how citations work, I cite stuff all the time in my own work” — but patents follow a different set of logic, where what you choose to cite can affect how your patent is processed.

Is the top-up simply exacerbating the SBIR selection? You could imagine saying, “We’re still disproportionately likely to get funding — it might be that the Department of Energy thinks solar is the best, so they give more awards to a solar program, and then a solar firm is more likely to win an award just because they got a grant, and so they’re more likely to randomly win this thing.” The way to think about it is: imagine there are 10 winners for this program, ranked from the first-best to the 10th-best, and the probability that one of those 10 wins one of these matches is essentially random conditional on winning. It doesn’t matter that you were the best program according to the Department of Energy — that’s uncorrelated with whether you reside in one of these states. That’s how they would probably defend this.

Their approach is, instead of looking at citations: look at natural-language-processing similarities between the text from the Department of Energy that’s issuing this call for proposals and the patent abstracts that belong to a given Cooperative Patent Class. As I said, this is old-school natural language processing — do you use a lot of the same words, giving more weight to words that are uncommon in the corpus? They’re also going to look at geographic distance: how far away are you from the firms that won this award?

Here’s their topline result.

For every million dollars in windfall funding, the grant recipients turn out another 0.75 of one patent. But if you look at all US firms and inventors — people who are close to this funding opportunity area category — and you’re assuming that some categories are randomly getting these cash drops and some aren’t, then this is now 1.75. If you include foreign firms we’re getting up to 1.19. An important takeaway for our returns-to-R&D question is the gap between the private return versus the total return estimate. In terms of number of patents, there are three times as many patents attributable to dropping money into this program that are not performed by the person who actually got the money. A lot of it comes from foreign actors, and from patent outputs not closely targeted by the original solicitation — implying that spillovers are a big deal and hard to predict, and the firm that receives them is not getting a majority.

I’ve presented the raw patent count. They also do a version where they weight patents by how many citations they receive, which is a common way to value patents. When you do that, you find that the recipients capture a larger share of the total value — more like 50% rather than 25%.

For each of these papers, we’re going to grade them on some of the issues we highlighted:

• Scope — what benefits are we counting? This is only looking at patented inventions. It’s nice that we’re looking at national and global spillovers, but still, a lot of invention doesn’t show up in patents.

• Costs — this is only looking at the costs of the Department of Energy’s SBIR grants; we’re not thinking about all the other R&D spent to enable this type of work.

• Lags are not really a focus. They do some suggestive work that a five-to-seven-year production lag is right, and then stick with that.

• Identification — they thought a lot about this, trying to find plausibly exogenous variation. They have a nice scatter plot showing the ROI in terms of patents per million dollars spent: if you just look at the raw number, how much the Department of Energy gave to this program, versus looking only at the state-match funding — the OLS, the raw naive one, significantly overstates the productivity of R&D.

• Measurement — they innovated by going beyond citation trails. They redo their analysis relying on the classic citation method, and that would have missed a large chunk of all the spillovers.

Azoulay et al.: NIH grants

Next, Azoulay et al., about NIH grants.

In this case the unit of analysis is going to be the disease-science-year. You can think of this as funding for cancer research using genomics in 2008. So it’s not all cancer research; there might be cancer research that’s also using exercise science in 2008, and that would be a different unit of observation. Their question: does biomedical R&D spending cause private-sector patenting?

Why are they interested in this sector? The biopharma sector is a really good one in terms of how well patents capture innovation. To get a drug on the market, the FDA has to approve it, so we have an objective count of how many new drugs are developed, and we can see how many of them actually have patents — and most of them do. We’re not missing a large share of at least drug innovation in this sector. Secondly, NIH is an important funder — it’s the biggest funder of biomedical research in the world.

Their approach is like what we did — this work inspired the paper we wrote later. It’s the same approach: a grant is made; you can tie it to a publication by seeing if the publication acknowledges support from that specific grant; and then you can draw a link between a patent and a publication by looking at whether the patent cites the publication.

How do they handle the identification strategy?

Just like the Department of Energy, certain disease-science areas in a given year are going to receive more funding because the research agenda is more promising in those years. They take advantage of the fact that the way the NIH scores applications is complicated and Byzantine. First, a review panel looks at all of the proposals within a given science area — so we’re looking at cancer-genomics proposals, but also kidney-disease-genomics and neuro-disease-genomics proposals. All the genomics stuff is being ranked, peer-review style, putting them in order from top to bottom.

After that process happens, the applications get shuttled off to the disease area. Now I’m the cancer division, and I’ve got my proposals from genomics and from exercise science. I can see — this exercise-science cancer research proposal was the best one, they gave it a 1; this genomics one was the fifth-best one the genomics people saw, so it’s number five. That means this exercise one gets funded before this genomics one — but there’s actually been no comparison between those two specific proposals. It could be that the genomics one is in fact better, but there was no venue for comparing across the different science areas in that way. It’s called the “rank of ranks.”

Their approach is to say: let’s look toward the end of the line, where there’s the cutoff, and we aren’t going to fund any grants below that cutoff. We’re going to argue that down there it’s kind of random whether you got funding or not. They look at the five grants above the payline and the five grants below, and say: “If you happen to lie above it, that’s like windfall funding for your science-disease area in this year; and if you’re below it, that’s your counterfactual — you were very close, but you didn’t quite get it.”

That’s how they measure this windfall funding to different kinds of research. Then they have this approach to figure out the effect on biomedical patenting. As I said, grant acknowledgments link them to publications, and citations link those to patents. But there’s potentially a problem if they only do that. It might be that the main way NIH-funded research affects drugs doesn’t show up through this channel. What if I get a grant, and make a major breakthrough, and my major breakthrough is a foundational paper that spawns an entire new literature of related research? Maybe lots of patents will cite the secondary literature that I inspired, but they won’t necessarily cite me, so my contribution will be invisible. One thing they try to do to take care of this is look at patents related to the patent that acknowledges grant support, using the PubMed keyword similarity — an algorithm at the time to find related science. They do it both ways, so you can see how much it matters, and they’re going to be agnostic on lags.

They find that lags are very long: their paper charts top out after 15 years, but there are still benefits occurring at that time, and they’re variable. They find that spillovers are really large. It’s very commonly the case that you make a grant for one disease area, but the patent that ends up citing that research is for treating a separate disease. One caveat: recall we just saw the Myers and Lanahan paper arguing that citations might miss a bunch of the story, so we may be missing some share of the impact here.

Their core result: $10 million in NIH funding is associated causally with another 2.3 private-sector patents, and a large chunk of this comes from spillovers that cross these disease boundaries.

If we’re only looking at the impact from the same disease, we might miss half the story. That’s directionally consistent with Myers and Lanahan, which also found that a large share of the benefits come from spillovers to different areas than were directly targeted.

Now, in terms of valuing the returns — getting an ROI out of this.

If you read this paper, you’re struck by how nervous they are to do this, and how cautious they want to be, because they recognize this is a hard question. It’s potentially an important question that people might run with, so they’re nervous about saying, “We know what the ROI is.” There’s a lot of uncertainty. But here’s the approach they take. How are we going to value the benefits?

• One thing we could do is look at drug sales. That finds $10 million in grants leads to $14.7 million in drug sales.

• A second approach is to use those firm stock-market valuations: when the drug patent gets announced, what happens to the stock-market price of the firm that owns it? You get $34.7 million.

This gives you a rough ROI: $1 in NIH funding is giving you $1.40-$3.47. But importantly, we think these have to be underestimates for a few reasons.

• We’re only capturing the private value to the pharma company — the drug sales, the stock-market valuation — and only through patented drugs (most drugs are patented, but not necessarily all of them).

• We’re not thinking about the social value of a drug. Some papers argue that firms only often capture a small value of the total consumer surplus.

• There could also be non-patent channels that are still important in medicine — clinical-practice changes, surgical techniques. Also training: NIH might be important because it trains somebody who then goes on to work on science that is not connected via these citation chains to any of the other work they got the grant for — but that grant still was a causal chain in their ability to have a biomedical career.

• Lastly, the citation chains might undercount spillovers.

Let’s look at this framework for evaluating the paper:

• On scope: biomedical patents is in some ways better, because we know a large share of the drugs that come out of biomedicine do get patents — but we’re still only looking at health outcomes, private-sector returns, and we can’t see these non-patent channels.

• On cost: again we’re only going to look at the cost of the NIH grant, not the subsequent cost to run the clinical trials to get FDA approval, and those can be really large.

• Lags: this is a strong suit of the paper — they don’t have to make assumptions; they can be agnostic and let the citations show where the links happen.

• On identification: another thing they think carefully about, with this rank-of-ranks approach. As it happens, there’s not a huge difference between their rank-of-ranks instrumental-variables approach and just classic OLS where you add a lot of controls — so they argue the selection effect is not going to be that strong, and it might be that the rank of ranks is just not that good at selecting the top science.

• On measurement: this cross-disease linkage is an interesting spillover story.

Dyèvre: all federal agencies and firm productivity

Next, let’s turn to Arnaud Dyèvre. He’s going to zoom us out even further.

First we looked at the Department of Energy SBIR program, then all of NIH, and now we’re going to look at all 17 federal agencies. We’re going to look at a long time series, 1950 to 2020, and instead of looking at patents, we’re going to look at firm total factor productivity.

How are we going to approach this? We’re going to use patents — but not as a measure of innovation, more like a measure of where you live in innovation space. Agencies sometimes get patents for their work. We’re not going to count patents to see how much innovation the Department of Agriculture did. Instead, we’re going to say: the Department of Agriculture got patents related to corn in this year — that’s a signal that they were doing R&D related to corn. The Department of Energy got patents on solar technology in the 2000s, but wind power in the 1980s, and that’s telling us something about the character of the research they were doing at different times.

Then we can look at the firms’ patents and say, “This firm is working on wind technology, so it probably cares about what the Department of Energy was doing back in the 1980s.” (These are all examples; I don’t actually know when they were working on wind technology, but when they had patents that were similar.) But they don’t care about what the Department of Agriculture is doing, because they can see those patents have very little overlap with the technology classes this firm is working on. A NASA funding shock is going to matter more for an aerospace firm than a pharma firm, and we’re going to see which firms have aerospace patents to identify them.

The identification after that is going to be this shift-share instrumental variable: historically, how closely related am I to this agency’s R&D based on our overlap; and then if there’s a shock to its R&D spending, what happens to my TFP? He’s going to look at different kinds of shocks.

Here’s the variation in R&D budgets over time.

On the left, the overall; on the right, we’ve taken this purple line at the bottom and blown it up to zoom in on some of the smaller agencies. A couple of major things to see. This green line is space R&D — it exploded during the space race, was really large, but then tapered way down. Defense, as a share of R&D, has — surprisingly for some — tapered down over time. Health has grown from a tiny sliver to the second largest. The Department of Agriculture, where I worked, was very steady, although shrinking a bit in real terms.

Their core result is that a 1% increase in public R&D spillovers gives a 0.023-0.025% firm-level TFP increase after five years, and this is still persisting after 10 years. The effect is larger for small firms than large firms. He has a calibrated model showing that declining public R&D investment can explain various things we observe, like the rising firm-size inequality.

Let’s stick in this framework:

• The scope is nice, in that we worry patents undercount a lot of innovation — I’ve written a series about that. In some fields patents are a pretty good capture of what’s going on, but in others most inventions don’t seem to be patented. So it’s nice to do something different: this is total factor productivity, which is itself very hard to measure well, but if you have a lot of firms over a long time, maybe the signal can come through the noise. Still, we’re only looking at firms that are publicly traded, because they’re in the Compustat dataset, so we’re going to miss private firms — which could be particularly acute for this paper if small firms are disproportionately private and benefit more from this R&D.

• On cost — we’re looking at the federal agency budgets only, not the enabling R&D stock from the past.

• Lags — we’re mostly pre-specifying 5 or 10 years based on a read of the literature.

• Identification — the shift-share approach.

• Breadth — a strength of the paper: all agencies, all technology areas, a long time series of 70 years.

Fieldhouse and Mertens: national TFP

The last empirical paper is Fieldhouse and Mertens. We went from one program looking at patents, to one agency looking at patents, to all agencies but looking at a subset of firms and their TFP. Now we’re going to look at all the major agencies and at national TFP. We’re going to capture all the spillovers, across five major agencies. If you’re getting the five major agencies, you’re getting a big chunk of the R&D that’s happening.

Why focus on national TFP? The downside is that you get one time series, which makes things a lot harder than when you’re Dyèvre and you’ve got all the firms and their trends of TFP, or lots of patents. But national TFP captures spillovers everywhere. If an individual firm goes out of business because it can’t compete, and there’s all this business-stealing effect, we’ll see how that all washes out in the aggregate, and that’s maybe valuable for us.

The approach: Fieldhouse and Mertens are going to do something inspired by this famous Romer and Romer paper, where they look at monetary policy statements to read what justified the monetary policy changes, and try to find shocks that were more random, or not correlated with changing economic circumstances, to understand monetary policy. What they’re going to do here is look through appropriation shocks. A shock in this case is an increase in a budget of more than 5% or a decrease of more than 2.5%. We’re going to read through the appropriation history and ask: “Why did we change the budget that much in that year?”

In particular, Fieldhouse and Mertens are most concerned about economic confounders — budget changes because of a recession, for example, leading to increasing a budget as a stimulus measure. If we’re looking at national TFP, we’re going to confound TFP changes driven by changing economic circumstances with the changes from R&D. So they’re going to read these things and try to find shocks that were not driven by economic circumstances.

Here we’ve got annual changes in these agency budgets.

In blue, what they consider exogenous changes; in orange, endogenous changes. The big one is in 2008 and 2009 — the American Recovery and Reinvestment Act following the 2008 financial crisis. There was a huge stimulus package, and that increased the budget of, among other things, the NIH as part of the stimulus. They would count this as an endogenous shock, and we don’t want to be looking at those. We want to look at the times the budget was increased — like in the 2000s — more because, “We’re going to double the budget of the NIH because we think the value of science is high.” That introduces a different concern that they’re selecting on this endogenous technological opportunity — but we’ll come back to that.

There are 257 large appropriation changes. About a fifth of them are endogenous, and they get dropped. The remainder they’re calling “mission-driven,” and they’re not correlated with economic conditions. They’re then going to look at — when you add some controls — what’s the correlation between changes in R&D capital stock and total factor productivity, or lots of other things you can observe at the national level.

What about this problem of technological opportunity, where we’re sometimes increasing the budget because we think there’s a good opportunity in this technology space? They’re going to try to control for that with control variables — the forward-looking stock returns for high-tech sectors, what the market expects the returns are going to be for the high-tech or the health sector, as ways to get at what expectations were at the time around where these technologies might go. You could believe that or not, but that’s the approach they take.

Here’s the knowledge-production pipeline.

These charts show what happens on average to these different output measures when there’s a 1% increase in the non-defense R&D stock. An R&D stock is the weighted sum of all past R&D investments. We’re going to increase that by 1%. When you do that, labor productivity goes up over time. Potential output goes up. They look at a number of proxies you would expect to see if technology was also growing. Patent innovation — this is an index that looks at the text of patents, and if the patents are introducing surprising new terms that go on to be important, then we consider them more innovative patents — this goes up after a lag. The number of STEM PhD recipients goes up after a lag; the number of researchers goes up quickly; and the number of technology books goes up.

On the one hand, this does show there’s something happening to technology after you see these shocks to R&D stocks. I’m not sure the timing of these lines up quite as well — it’s not natural for me to assume that the technology books would happen before some of these other things — but this is what they find.

This is their core result.

We have government R&D capital — how the capital stock evolves after a 1% positive shock to R&D capital stock. Business-sector TFP goes up over time. And real GDP. They’ve got this blue and this orange line, both capturing a bare-bones approach versus an approach with lots of controls. They find a 1% increase in non-defense R&D capital leads to a 0.2% increase in TFP, significant — this is 15 years out — and you can see the statistical significance in these gray bars.

Grading it:

• Scope — the biggest scope: national total factor productivity. We’re getting all the spillovers wherever they land. This is the closest to what Jones and Summers’ hypothetical experiment was going to be like.

• Costs — it’s not covering investment costs, but it’s including the R&D capital stock, not just the current spending or grants, so this is closer to all costs relative to all benefits of R&D.

• Lags are open-ended; they’re not imposing a structure. They just say: when there’s a shock, let’s see what happens to productivity over time on average.

• Identification is a challenge; we’ve highlighted some of these things, so you can compare that to the ones that have a tighter causal identification strategy.

• Breadth is nice — all five major agencies.

Putting it together: is the evidence consistent?

The last step is: how does everything fit together? Fieldhouse and Mertens also report a long-run benefit-cost ratio.

They use a discount rate of 7% rather than the 5% I talked about earlier. That gets you 6-9 for the benefit-cost ratio, with a preferred estimate of 7. Using the same discount rate — 7% — Jones and Summers gave us a range of numbers. The number Fieldhouse and Mertens come up with is in between these low-science-share, long-lag and short-lag numbers, and that probably makes sense: non-defense public spending is a mix of pure science that takes 20 years, and more applied development stuff that has a shorter timeline. So it’s in the theoretically supposed ballpark.

They also reported the internal rate of return.

The internal rate of return is: instead of assuming a discount rate, use this formula to pick the discount rate that equalizes costs and benefits. If you do this with Jones and Summers, you can rearrange their equation by setting the benefit-cost ratio equal to one, pick these values for the growth rate and the R&D share of GDP, and they get an internal rate of return of about 67%.

This is really different from what Fieldhouse and Mertens got, so maybe initially we should be concerned — they found the return is 140-210%. Let’s think about this more carefully, though. 67% is what Jones and Summers do when they’re looking at the average return on all of R&D together — private-sector and public-sector mushed together. What if we only look at the private sector?

You probably covered Bloom, Schankerman, and Van Reenen, looking at Compustat data on publicly traded firms.

They used this instrument of when a state changed its tax credits, which shifted firm-level R&D spending, and then they could see what happens to TFP in the firms. They got that the private return on R&D was about 21%, and the social return, incorporating spillovers to rivals, was 55%. I should also point out that this paper got replicated in 2019 with a bigger dataset, and they got very similar answers.

Let’s see if this all fits together.

Fieldhouse and Mertens get 140-210% for non-defense public R&D. Instead of just comparing those numbers, let’s think about what the weighted average should be. If we think the private sector is 54% of all R&D spending over the 1950-2020 period, and the average return is roughly 55% — what we got with the Bloom and Schankerman paper — then we have the non-defense sector, which we think maybe on average has a 175% return (in the ballpark of what Fieldhouse and Mertens did), and 20% of R&D spending was non-defense. What about defense? Defense is big — it’s 26%. They show some evidence from what the Congressional Budget Office and others assume and argue that maybe the internal rate of return here is 25% — much lower, because we tend to think there aren’t as many spillovers, the research is classified, and there are fewer applications in the commercial sector. If you make these assumptions, you get that the weighted-average internal rate of return is 71%. That’s pretty close to the 67% that comes out of the Jones and Summers calculation for what the average of all R&D should be.

Important to note: if non-defense R&D is only 20% of R&D, then the fact that it has this outrageously high number doesn’t necessarily move the total average around as much. And it’s useful to point out why non-defense R&D would have such a high rate of return. Remember, it’s not operating in a competitive environment where somebody is going to keep spending until the marginal return is equalized to its marginal cost. Recall that at the beginning we had some stylized facts from Dyèvre that showed public-sector patents tended to have these unusual features; non-defense research is more tilted to this basic stuff that we think has more spillovers.

The last thing we’re going to do is see if these are consistent across these papers.

• Fieldhouse and Mertens directly compare their results to Dyèvre and argue they’re in a similar magnitude, so roughly consistent.

• With Azoulay, though, there starts to be a gap that we’re not totally capturing. Azoulay finds a dollar in NIH funding generates $1.40-$3.47 in private benefits to the firm — but remember, the other papers are only looking at GDP; they’re not capturing consumer surplus themselves either. We know this is a conservative floor, but can we get all the way up to seven, which is the preferred benefit-cost-ratio estimate of Fieldhouse and Mertens for all R&D? What are we missing if that’s the case? Maybe NIH just has lower returns than other kinds of agency funding. After all, NIH is the biggest non-defense share of R&D, so maybe we’ve moved down the marginal cost curve there. But that would imply the other areas have returns even higher than 140-210%, which was the weighted average.

• What about Myers and Lanahan? They don’t estimate the benefit-cost ratio, but if we assume — going way back to the beginning — that the private benefit-cost ratio is about one, which is something you can get out of equilibrium, and if we assume the social returns are two to four times higher, then that would imply the social benefit-cost ratio is 2-4. That’s similar to the Azoulay paper, but again isn’t getting us up to the seven that these other papers are implying. Maybe it’s a lag issue — maybe we need to be looking more than five years out. Maybe we’re missing a lot of the spillovers or the innovation by focusing on patents. But there’s still this discrepancy.

The fact that there are these discrepancies is one reason we’re interested in starting this Pop Up Journal project, which is going to try to make headway on this over the next couple of years — because there seems to be a lot of evidence pointing to the idea that the social returns are greater than one. But what should it be more specifically? Harder to say.

I hosted a panel about these questions recently, and Fieldhouse was on it along with other people. They argue we need better data — especially international data on R&D, TFP, and the people who are doing the R&D. We have to think about how uncertainty affects the return on R&D. And especially, politicians and people in the constrained government making decisions about how much to spend — they care a lot about these non-economic values of R&D that they’re trying to achieve by funding it, and it would be good for the field to be responsive to the needs of these policymakers.

Imagine you want to encourage more or better innovation. You want to create a new cure or understand an important social phenomenon. How do you make that happen? Funders have numerous approaches that they can use.

In July, IFP (with Kyle’s help) released the Atlas of Innovation, a website that helps policymakers and philanthropists navigate decisions about how to fund innovation.1 This lecture provides an in-depth complement to the site: Kyle delves into the different economic structures that underpin grants, contracts, prizes, and other funding approaches.

Subscribe now

In the image below, from Azoulay & Li’s paper Scientific Grant Funding — which Kyle discusses — the authors provide intuition on when we should think about using different types of innovation funding mechanisms. It’s worth digging into this chart on grants, prizes, patents, and contracts.

Grants: As described in the Atlas of Innovation, grants support “researcher-led work within a defined scope.” They are funds generally provided to researchers with limited strings attached and an expectation that the research will produce public benefits. According to Azoulay and Li, grants are useful when the innovator shouldn’t or can’t capture the value created through their research and the output desired is open ended.

Prizes: A prize is “a competition that rewards teams for meeting a defined innovation target.” Funding could be provided for reaching a target like solving a mathematics problem or creating an important new technology. Like grants, they are most appropriate when the innovator shouldn’t or can’t capture the value created and where there is a public benefit.

Patents: Patents provide “firms a period of market exclusivity;” they incentivize innovation by protecting innovators when they produce profitable ideas. In this case the government is not providing innovators money, but rather is protecting them from competition. Patents are best when the government doesn’t have a desired end-point (e.g., they don’t specify priority technologies to be patented) and the innovator can and should capture the value created.

Contracts: “A contract is an agreement in which a funder contracts with a specific performer, often a company, to conduct R&D.” Contracts are like prizes in that there is a known end-point — in this case the thing the performer is being contracted to do — but the outputs of the research will be captured by the funder or innovator.

How science is funded is a critical area of study if you’re interested in making government-funded research more effective. Often, federal agency staff are unaware of the tools they have at their disposal, as well as when they might be more or less appropriate to use. This gives them — and you! — the information needed to effectively fund research and innovation.

Subscribe now

Here are the readings that go with this lecture:

• Carnehl, Christoph, Marco Ottaviani, and Justus Preusser. “Designing Scientific Grants.” Entrepreneurship and Innovation Policy and the Economy 4.1 (2025): 139-178.

• Hegde, Deepak, and Bhaven Sampat. “Can private money buy public science? Disease group lobbying and federal funding for biomedical research.” Management Science 61.10 (2015): 2281-2298.

• Myers, Kyle. “The elasticity of science.” American Economic Journal: Applied Economics 12.4 (2020): 103-134.

• Acemoglu, Daron. “Diversity and technological progress.” The Rate and Direction of Inventive Activity Revisited. University of Chicago Press, 2011. 319-356

• Carnehl, Christoph, and Johannes Schneider. “A quest for knowledge.” Econometrica 93.2 (2025): 623-659.

Subscribe now

Thanks to William Higbie and Harry Fletcher-Wood for their support in producing this video and the transcript.

Subscribe now

Lecture Transcript:

The Direction of Science

My name is Kyle. I am an associate professor at the Harvard Business School, and today we’re going to be talking about the direction of science. Some fun stuff. I like to show graphs with things not labeled and have people guess what’s going on.

We’re going to be talking about the direction of science, and I want to be clear about what direction means. You will already have heard from Ben and Chad Jones, the wonderful Joneses. Soon Matt Clancy will talk with you. A general message of a lot of that work is that on average, when we put money into this thing called R&D, or science, or innovation, a lot of great stuff happens. Today is going to be a lot about — conditional on putting a dollar into this magical growth engine, there are lots of specific things to invest in. Where that money goes is what I mean when I say “the direction of science.” We’re usually interested not in the level of investment across all areas but the ratio of investments in different areas.

Reading the direction of science through publication data

To give you a sense of what variation in the direction of science can look like through one lens, I’ve got a few graphs here of publication shares. Publication metrics over time are always hard to interpret. You should take any trend line you see with publication data with a large dose of medicine. But I’ve got a few here. This first one is five different categories inside some type of science. I’m wondering, does anyone know what these topics might be, looking at the trends?

This is energy science.

You could see a very interesting boom in research on natural gas in the tail end of the 2010s and around ye olde epidemic. There were some interesting technological developments with respect to fracking and what it could do in good ways and also possibly in bad ways. You can see, as it expanded, the different areas that it ate into.

Here’s another one that also has a bit of a surge right around 2020.

You’re probably thinking it’s one thing — this is computer science.

That surge is not COVID. That’s large language models. If you broadly define artificial intelligence, very broadly speaking these are five things going on over the past 50 years. You can see the large language models — there were people talking about them back in the ‘70s and ‘80s, but it wasn’t until very recently... You can see they’ve totally dominated, in particular symbolic AI. Symbolic AI is, as opposed to the stochastic approach of most large language models, symbolic AI was a — “We’re going to hardcode a very mechanical, deterministic system and see if we can use that approach to develop intelligence.” It hasn’t faded completely away, but you can see most people these days work on LLMs.

Now this is the one that you probably thought the other two were.

This is a look at disease science over that same period.

You can see there are two interesting things in this data. One is the HIV/AIDS epidemic occurring in the ‘80s and drawing a lot of resources, resulting in a large growth in the publication share. These are shares within these different categories — this isn’t shares across all of science. Then you can see, unsurprisingly, the research on coronaviruses and COVID-based diseases spiking with the pandemic, and still a lot of research going on in that space today. This data goes up till about 2023, where it’s nice and reliable.

From outputs to inputs

Those were outputs, looking at the direction of different kinds of outcomes: publications. All the money has been invested, the papers have been written. A lot of what I want to talk about today is the input side: where do we put our investments, and what do we mean when we talk about managing the direction of science?

One way to think of that is investment in a very particular thing. For example, the ITER fusion reactor in Europe is a joint venture that started in 2006. This is my favorite example of a large-scale targeted research investment that is not going quite according to plan. Twenty years into this project, we’re now 15 years behind schedule and about €15 billion over budget. This is an emblematic example of the difficulties of knowing how much things are going to cost in the world of frontier science and managing that.

Now I’m going to tell this in a way that gives you a much more optimistic lens on intervening in the direction of science, although in truth I make no causal claims with this graph. But it is a fun graph. In 1960, 1970, the National Cancer Institute (NCI) did exist but was very weakly funded, on the scale of maybe $100 million per year, and cancer mortality was on the rise. In 1971, Richard Nixon starts the war on cancer, and it leads to a large and sustained boost in funding for the National Cancer Institute. Was the war on cancer winning? Unclear. We kept spending a lot of money.

In 2008, David Cutler, whose approach I’m drawing very much from, wrote a very nice paper that says, Are We Finally Winning the War on Cancer? The beautiful graph he has, which I’ve replicated here, shows a dramatic decline in age-adjusted cancer mortality in the United States.

How much of that is purely due to lung cancer survival rates going up because people are not smoking as much, and how much do you want to attribute to the NCI? These are tough questions. Your eyeball wants to give it all to the NCI, but also — I don’t know, it started to pay off after 30 years. Maybe this was a large portion of it. We don’t know. A lot of the frontier questions people are asking right now are: what are the returns on investments in these targeted programs?

Spending at the NIH

We’ll spend a bit of today on the US National Institutes of Health (NIH), both because it is large dollar-wise and because it’s a place that has some empirically useful features for studying.

This is the share of the budget going to each of the major institutes and centers of the NIH as a fraction of the total NIH budget. You can see some interesting trends and changes there in terms of where our tax dollars are going.

There have been a lot of papers where people take that data and plot it against something like disease burden. That comes from a notion that, even if you’ve never taken an econ course, your brain sees that spending and says, “I hope we’re spending money where there’s a lot of need for scientific advances.” I’ve drawn a 45-degree line, which represents what it would look like if, for every 1% increase in burden that these diseases have on the US economy, we would spend 1% more dollars on that disease.

Now I’m going to show you the data.

The real elasticity is not one. It’s 0.41. It’s positive. We do spend more money on diseases with larger burden, per WHO disability-adjusted life years, but it’s not a 1% increase.

What should it be? These are interesting and difficult questions, and the kinds of things we’ll be talking about in our lecture today. We’ll come back to this graph at the very end, thinking about — if we took the World Health Organization estimates as the true measure of disease burden, what is the source of the residuals from this line? Why is musculoskeletal so far below the average trend? Why are genetic disorders, or aging and dementia, a little bit above? Is that due to distortions or frictions in the political budgeting process, or is it due to research on dementia being very productive right now? Another reason you will invest in one topic and not another is because every dollar you invest gets you more. So holding fixed demand, you always have to think about supply: the classic econ challenge of separating supply and demand curves.

Outline for today

We’re going to proceed in three parts:

1. A very high-level, institutional-level focus on how budgets are set, as far as we can understand them.

2. A more micro-level look, thinking about scientists’ response to decisions made by scientific institutions.

3. A discussion of some frontier theories about how this works. As evidenced by those empirical charts I showed you before, doing empirics on some of these questions is quite challenging. It would require a counterfactual not war on cancer. For those of you who are empiricists in the crowd, it’s hard to find the right control group for the war on cancer. This is a place where we do need to try to rely on theories when we can.

Who determines the direction of science?

A shout out to Bhaven Sampat, who introduced me to these two fellows, one of whom is the author of the first paper we’re going to discuss. On the left is Vannevar Bush, and on the right is Harley Kilgore.

Vannevar Bush was the director of the Office of Scientific Research and Development during World War II and wrote a very famous memo called Science: the Endless Frontier that founded the National Science Foundation and laid the groundwork for a lot of science policy in the United States. He was a scientist himself at MIT, and as most scientists will say, his answer was, when in doubt, just give the scientists the money and let them figure out what to do with it. That was his answer to the question, who should determine the direction of science?

Harley Kilgore was a senator from West Virginia, a very sharp man who saw what was happening after World War II as the NIH and the NSF started to scale up and thought, “Not so fast, Mr. Bush. Perhaps the public and our policy institutions should have a bit more sway over where this funding is going, because you scientists like to do things that you like to do, which may not be in line with the public’s best interest.” The Bush–Kilgore debates were rampant in the ‘40s and ‘50s, and in today’s political climate have come back into popularity.

Disease group lobbying and the NIH

The first paper that I want to spend a little bit of time discussing was this nice paper by Deepak Hegde and Bhaven Sampat, Can Private Money Buy Public Science? Disease Group Lobbying and Federal Funding for Biomedical Research, which is still one of the better lenses looking at how much of the scientific funding in the United States gets influenced by political forces — in this particular case, disease group lobbying.

Before this paper, and even still now, a question is: how much does lobbying matter in practice? If you are interested in these political economy questions, there’s a terrific paper by John de Figueiredo and co-authors that has a terrific title, Why Is There So Little Money in Politics? which gets at the question you can see in my numbers here: the government disperses a tremendous amount of money.

The NIH budget these days is approaching $50 billion. And yet — these numbers were based on me working with Claude for a few hours to try to figure out the most up-to-date numbers about the size of disease lobbying — we got to an estimate of about $30 million. That’s many orders of magnitude different. But an interesting question is always, perhaps the returns to those dollars are very steep at first and then become very flat, and so we still want to know: are those dollars influencing where our public investments are going?

Before we jump into it, it’s important to think about how those dollars could influence. We’re a lobbyist. We pay money and spend time with the senator and tell them that the disease we are interested in is very important and more resources should be allocated to it. How could the senator from West Virginia influence that?

One way is by setting the budgets of the institutes and centers, which is an explicit mechanism that Congress has control over. They say the NCI budget is a certain amount of money. Another is they could try to fund specific initiatives. In the case of the NIH, this is actually quite rare. Most of the congressional authority comes through the institute and center (IC) budgets, although there are instances of some congressionally mandated set-asides. I believe Type 1 diabetes is one of these, which for you PhD students out there — I tried to look at that program because it has some interesting budget shocks. I never got too far, but that might be something interesting to look into, because it is one of the few congressionally mandated set-asides.

Then there are also these things called soft earmarks. This is clipped out of the 2024 appropriation language for the National Institute of Neurological Disorders and Stroke.

At the top, you can see the line item for this institute, and that’s the big thumb on the scale that Congress has power over. The challenge is, at a scale of multiple billions of dollars, it’s hard to move that number around.

Underneath these things, if you look in the appropriation language, you’ll see many paragraphs that look like what you see here, which will start with some disorder that Congress claims to be interested in. It’ll say things like, “The committee is concerned that one out of every fifty individuals has a brain aneurysm.” This is the soft earmark. It’s called soft because there’s nothing about this where Congress is saying you have to spend x amount of dollars on something. But it does go into the legislative record, and the NIH for sure is aware of these things, and the staffers on the Hill make sure the NIH is aware of them.

They have a number of regressions that I’ve done my best to set up as a nice little directed acyclic graph that we can just walk through, to see what this paper is finding is happening in the data. This is a three-stage event.

First, for every disease, you look out in the world and there’s some burden that it has on the population, some scientific opportunity, which we’re going to proxy with the number of publications on that topic, and some base rate of funding already going into that disease. The first question is: how do those things affect how much lobbying occurs?

I’ve reported the coefficients here. These are all logs, so you can roughly interpret them as elasticities. They’re small but non-trivial. That 0.05 means that for a 10% increase in disease burden, there’s a 0.5% increase in lobbying spending. So there is more lobbying spending when these things go up, but there’s plenty of variation due to the residual there as well.

The next question is: how do all of our three states of the world and lobbying affect the rate at which we’re seeing these soft earmarks?

You can see there is some direct effect from disease opportunity and prior funding. Prior funding is effectively zero. There’s a clear effect from lobbying. I did my best to convert the estimate into an elasticity. It’s usually on the scale of 0.1 or 0.2, depending on the numbers you use from their data. That means for a 10% increase in lobbying, there’s a 2% increase in the count of these soft earmarks in the NIH.

They’re soft. They don’t legally require the NIH to do anything, but it doesn’t take a savvy political scientist to understand how this might influence the NIH. The third and most interesting stage in the paper is thinking about how soft earmarks then themselves affect grants.

A fun thing I’ve shown with the zero on the bottom is that once you condition on soft earmarks, lobbying has no effect on the amount of money going to a particular disease. I wouldn’t call that a state-of-the-art approach for a mediation analysis given modern techniques, but it’s a good first-blush look at saying, “If lobbying does anything, it certainly looks like it does it via soft earmarks.”

The first interesting number there is the 0.05 on the all-dollars, which says that the soft earmarks have some pretty small bite on all of the funding. But if you zoom in on these requests for applications and program announcements — which is when the NIH itself sets aside some money for research on a particular topic — you see a much larger effect of the soft earmarks on the money going through those stores.

The most interesting question here is how this affects science. Unfortunately, I have to leave that in my little dashed gray box, because we don’t really know. I’ve summarized the questions here.

One of my favorite things this table does is break down their analysis into a bit of a decomposition, so that in normal English we can understand just how much the political lobbying going on for disease advocacy is influencing the NIH budget.

They break out the chain of events: the average amount of lobbying, how many earmarks there are in total, how many we think are due to lobbying, the total amount of money going out the door, and how much of that we think is due to the earmarks. We come up with an important number to have in our heads, which is that about 8% of the NIH allocations are due to these earmarks, with about 40% of the earmarks appearing to be due to lobbying. This is a great number to know, because my prior before reading this is that you could talk yourself into a lot of different numbers. This is a really good first step in understanding what’s going on.

A global public good

Another related paper I really enjoyed by Margaret Kyle, David Ridley, and Su Zhang, Strategic Interaction Among Governments in the Provision of a Global Public Good. It’s related to the findings we just saw with the Hegde–Sampat paper. I’m just going to read you the abstract, because it’s also a wonderful exercise in a well-written abstract that tells you everything you need to know:

“How do governments respond to other governments when providing a global public good? Using data from 2007-2014 on medical research funding, we examine how governments and foundations in 41 countries outside the US respond to funding changes by the US government.”

The idea here is, what does Panama do when the US increases its spending on malaria-related research? Getting to the identification challenge, they have a pretty nice approach where they do something similar to the Hegde–Sampat: they say, because funding is positively correlated with things we can’t see, we’re going to use the representation of research-intensive universities on congressional committees as an instrument. The idea being, if a senator from Wisconsin gets onto the Health and Human Services committee, and Wisconsin has a lot of people that want to study malaria, and we get more money into malaria research in the US, it’s only because that senator rotated on, and not because malaria science became more powerful, or the demand for malaria science grew.

They have a wonderful elasticity in the abstract that’s good to know, which is that for every 10% the US spends on a disease, there’s a 2-3% reduction by the other governments. This is a classic substitution — you see versions of this in many different sectors outside of science, but it’s very important. Science is a global public good. This is a challenge of managing the direction of science: lots of other countries will benefit from their spillovers. “The US government is double-funding something” — that might be a pessimistic take. An optimistic take might be, “In the long run, we’re attracting talent to our country, so the returns to scale are actually increasing.” There are lots of interesting debates around that. But at least in terms of the short run, this is a very important paper.

More generally, because of the challenge of studying how the resources get determined, there are a lot of very interesting open theoretical and empirical questions about how these decisions are made at institutions like the NIH or the NSF, where I think there could be lots more work to be done.

Which scientists do we fund?

For the rest of this, we’re going to be working through the lens of, “Let’s all assume we’ve agreed we’ve got to spend $10 billion on cancer research. Which scientists do we fund? How do we give them the money?” All of those kinds of questions.

Before we get into grants specifically, another terrific review paper is by Pierre Azoulay and Danielle Li, Scientific Grant Funding. It starts with a step backwards in the chain of thought and says, “When and why might one want to use grants?” which is the bread and butter of what you see many public scientific institutions using, “As opposed to the three other mechanisms you might consider when trying to incentivize innovation.” They have this very nice little diagram that helps you think about when these different mechanisms are optimal in theory.

Grants are great when the question in question is very open-ended and hard to describe, in terms of the nature of idea search and in terms of appropriability. By appropriability, we mean, if the scientist is going to create $100 of value in the world with their discovery, how much of that value will they capture in some way? There are certain ideas whereby, just by the nature of that idea, the scientist will never be able to capture much of that $100 at all. That leads you to a world of grants, where you’re going to give the money up front and say, “Please do something nice for the world.” That please will have a lot of restrictions and rules and regulations around it, but that’s the world that gets you into grants.

We’ll come back to grants in a little bit. I thought it would be worthwhile working through the other mechanisms as well and giving some examples of what they look like, because in a full portfolio of managing the direction of innovation, these other things are very important. The wall of cites is me just trying to put some stuff there so you can see a few references to pursue if any of these mechanisms look interesting to you.

Prizes and contests

Prizes or contests are where we’re going to, ex-ante, specify a reward that we’ll give to somebody if they achieve a certain outcome. This obviously requires the output to be verifiable. The challenge here is going to be classic multitasking trade-offs to be balanced, which is, you better be sure you are incentivizing task A to get people to do A, because if you say we’re going to pay for A, but everybody can only actually do B, or you actually want B, you’re going to get A and not B. It turns out, when you think about frontier science and new ideas, the whole idea is you don’t know the solution to the question, so it’s very hard to ex-ante write down a description of something that can be verified ex post.

Where does this work? You see this in things like the Netflix Prize. This is a verifiable thing, where Netflix said, “We want to improve the mean squared error,” or whatever fit metric they had, “of our algorithm. Here’s all of our data. Let’s see if someone can come up with a better algorithm.” That’s verifiable, because you just submit your algorithm, run it on a benchmark set of data, and there’s no debate as to who wins the prize.

One interesting thing, which I’m always a bit cautious to talk too much about as an economist and not a political scientist, is that these ex-post rewards open the room for broadly construed politics, or risks of things that are a little bit harder to handle. In the Netflix case, when they tried to run it again, they wanted to publicly release the data, but it turns out there was some concern that the data they had released would actually allow the public to de-anonymize and identify individuals.

Another great example people often use is the famous longitude rewards, as an example of a successful prize in science. They were both successful and a good example of what can go wrong with prizes. John Harrison was a watchmaker in England, and he was the wacko who thought he could measure longitude just with clocks. The board of the longitude prize was mostly comprised of astronomers, who all assumed that, “Whoever wins this is going to come up with some new fancy way of using the stars to figure out longitude,” which would be a variant of how sailors had been navigating up until that time. So John starts sending in his chronometers that are crushing the competition when it comes to measuring longitude, and the board of astronomers kept moving the goalpost, asking more of him, and delaying his payments. It’s something like 80 years after his first chronometer was submitted that he actually got his final payment. That difficulty in committing to the payment is another political concern of using things like prizes.

Research contracts

Research contracts are kind of like a prize, but it’s a prize on the input side, which is saying, “If you put a certain amount of inputs in a certain direction, regardless of what happens, we will pay you for it.”

You have the same multitasking trade-offs; they’re just realized at the input stage. The challenge here is that you have to be very clear about the inputs. We have to have verifiable inputs, and the person using this mechanism has to know exactly what needs to be done.

A good example of this is ARPANET, the origins of the internet. After ARPA had the idea of how they were going to try to build the first internet, they needed to make these things. They’re called IMPs, but it’s basically a router — the machine that would stand in between two computers to send the signal back and forth. They had enough sense of how it should be done, but not quite the resources. So they developed a research contract with the company Bolt, Beranek and Newman, who’s now part of Raytheon and is a legendary company within the history of the internet. Fun fact I learned developing the slides: the first message they sent was “LO,” because they were trying to send “login,” but three letters crashed the system, so they had to do a few hours of recoding to get it to work again.

An example of leaning into research contracts when a little too early was in the 2000s. The Department of Defense, specifically within the Army, had a very big push which they called Future Combat Systems. As this quote from a RAND report afterwards indicates, they might have embraced this mechanism before they had a clear understanding of what was technologically possible. Writing contracts over things you don’t understand, unsurprisingly, can have some downsides.

Intellectual property

Last up before we get to grants is intellectual property rights writ large.

You’re probably thinking of patents as the default thing to think about here, but when we’re talking about IP in general, it really is just: someone has ownership over the idea and can do with it whatever they want, including selling rights to use that idea. These are really interesting things, in the sense that it’s going to direct people’s output — they’re going to shift their effort towards things that are well defined and have legally defensible boundaries, whether it’s via trade secrets, contractual licensing, or patent licensing.

The big challenge here is balancing static and dynamic trade-offs. The static positive incentive is clear. If I know that when I invent the next magical drug, anyone who wants to use it has to pay me, that’s a very large incentive for me to go after high-market things. The downside is that if I invent something that is very important for a lot of other people to invent other things, then there might be a dynamic friction whereby I would extract so much rents from other people who want to use my new invention for their new inventions that it would actually be suboptimal to give me full property rights over that idea. Balancing the current incentives for innovation versus the dynamic incentives for cumulative innovations is at the core of what a lot of these theories and empirics are getting at.

Grants

Now let’s start to get more into grants, which are the bread and butter for a lot of the frontier basic science where appropriability is very hard and the definition of ideas is very nebulous. The best we can do is going to involve a lot of frontloading of resources before anything verifiable comes out.

In terms of the trade-offs here, there’s good evidence from a lot of different papers that these things do work in practice. One of my favorite more dramatic examples is by my colleague Ina Ganguli. She has a wonderful paper, Saving Soviet Science: The Impact of Grants When Government R&D Funding Disappears, that showed how, during the fall of the Soviet Union, there was a push from George Soros to give a lot of money to Soviet scientists to try to help them continue doing work in science. She has access to the very simple algorithm that defined who got the money. There’s a very nice regression discontinuity for those of you who know what that means — a really good treatment-control setup. She shows that these grants, which were about a year’s worth of salary, were very influential in keeping scientists in the scientific profession and publishing during this time.

The downside here — another political-risk undercurrent — is that when you’re giving people money to do things when you don’t really know what’s going to happen, it’s easy to grab the title of these efforts and make it sound like wasteful spending. I have three examples here of things that, in different avenues, certain people would probably think is a poor use of our taxpayer dollars. All three of them are super important ex-post.

• Horseshoe crab blood is now the key way we test anything we inject in humans for safety before it goes out the door. During the pandemic horseshoe crab blood was one of the things that got short very fast as we were trying to ramp up production of vaccines.

• The GLP-1s that are taking the world by storm: one of the origins was researchers studying Gila monsters. It turns out Gila monsters don’t eat for a very long time, and the question was just, “How do they not eat for so long?” Ex-post you can see, “That’s going to prove useful for managing humans and their diet.” But in the moment, it’s the kind of thing that my four-year-old would just be like, “Let’s talk about Gila monsters and why they don’t eat so much.” It sounds a bit silly.

• Another one is, “We’re going to use a very expensive scanning electron microscope on coral reef just to look at it.” It turns out we learn how to make better bone grafts. These sorts of stories are all over the place.

That’s not to say there’s not wasteful spending, which is something we’ll come back to at the end. If you’re interested in this sort of thing, the Golden Goose Award is a wonderful website where they give out fun awards to scientific projects that have this flavor of seemingly silly-sounding but ultimately very influential research.

The granting enterprise

Let’s get into the design of the granting enterprise. I can’t recommend enough this paper, Designing Scientific Grants, by Christoph Carnehl, Marco Ottaviani, and Justus Preusser. It’s a terrifically written review that gives you a really good overview of the state of the predominantly theoretical art when it comes to how economists are thinking about designing scientific grants. I’m going to move through this and talk about a few things they cover.

One thing they do a good job of emphasizing is that this is a multi-step process, and decisions made further down that process will ripple back up through.

Seemingly trivial-sounding things, like “How long do we let people use the money in their grant?” can cascade backwards and ultimately influence the types of scientists who are applying for different types of grants. Keeping those dynamics in mind is very important.

They also have — partly because Marco is a co-author of both this terrific paper as well as the review — a really nice compressed-lesson version of this paper by Jérôme Adda and Marco Ottaviani, Grantmaking, Grading on a Curve, and the Paradox of Relative Evaluation in Nonmarkets. I’m going to walk through that now, because I think it’s really cool intuition. This and a couple of the other theoretical models I’m going to talk about are going to be slightly oversimplified in ways that are not the most technically correct version that’s in the paper, where you take everything very seriously. But it’s a good starting point, and not everybody needs to know the very difficult high-end details of how this theory works. The intuition delivers a lot of what’s important and can motivate a lot of interesting questions.

The first interesting thing is a fun result where perfect information actually leads a market to unravel. A classic result in economics is the paper by George Akerlof on the market for used cars. The idea being, in something that looks like a used-car market, if everybody is uncertain about the quality of vehicles, fewer people want to engage in that market, which means the only people that want to sell their used cars might have worse used cars, which means the people who might buy think everything is worse, which means everybody who sells only sells worse. That’s the kind of unraveling thought exercise that can happen in worlds where there’s imperfect information. This is going to push you to think about a world where unraveling can happen with perfect information.

The setup is very simple.

Let’s assume that every scientist has the same cost of applying and the same benefit if they get the grant. The quality that the funder is going to see is some noisy signal of the true quality of that application. Let’s say the funder is maybe a bit naive and is just going to say, “We’re going to fund the 20%, or 50%, or 10% best applications.” That’s the x̂. For ease of math, let’s say there are 100 scientists. Let’s say there’s no noise and that you see exactly the truth.

All 100 scientists see the announcement from the funder that only x̂ percent will win. Only x̂% of the 100 think they will win, so only 100 × x̂ will apply. But if 100 × x̂ apply, and the funder is naive and just sticking to their x̂, in this perfect-information world, only 100 × x̂ × x̂ will apply at this round of our iteration. You can see how, if you keep walking through this logic of how people’s expectations would update in this game, after that mathematical expression number of rounds, no one wants to apply.

Imagine that x̂ is 10%. With no noise 100 of us look out, and only the people who know they are the top 10 will apply, because they’re only going to fund the top 10%. Then they think, wait a minute, if only us top 10%, they’re only going to give an award to the top 10% of us, which would be the first person. Then the first person says, “I am but one person, so I guess maybe I will apply, depending on the rules. But if they’re only giving awards to the top 10%, then maybe I will not even apply, because I can’t submit a tenth of a percent of my application.” That’s the very stylized — not realistic at all — intuition of how, in this world of relative evaluation with perfect information, the market can actually unravel.

It turns out that a bit of information asymmetry keeps this kind of market from unraveling, because now people don’t know exactly their chances of winning, and that leads them to apply.

That sounds like, “Noise is in our favor.” It turns out noise is not exactly in our favor. I’m going to walk through this graphically using the really nice graph inside the review piece.

First, some fundamentals. The grant size, again for simplicity, let’s just call it 1. If you win a grant, you get a payoff of 1. Every person, in order to apply, has to pay some common cost, c. It’s a pain in the butt to write the application, and we all face the equal pain in the butt of writing it. But some of us have better ideas than others. Let’s consider a baseline setup where the funder is announcing this payline. Let’s say we can describe this system such that there’s this function W, which describes — for a given quality of an idea and the announced payline — the probability of winning. Then there’s some threshold θ^, which says, given the cost of applying, anybody with a θ above or equal to θ^ is going to send in their application.

That shaded blue area there is just the integral over W from θ^ upwards, and that tells you the number of applications you’re going to get. You can see it’s going upward, because people with higher-quality ideas are more likely to get funded. It gets concave eventually, because there’s some noise in the system, so even the person with the best idea has a chance of getting a bad draw. That’s why it doesn’t go all the way to 1.

Let’s do an intermediate thought exercise that’s not realistic but helps with intuition. Let’s add noise to the review process, but we’ll change the payline such that the same scientist, in terms of the quality of their idea, is on the margin.

That is to say, we’re going to add noise to our review process, but we’re going to change the payline so that θ^ stays the same. That threshold that determines who’s going to apply is still the same in this noisier world.

What’s going to happen here? First, what does noise do to the payoff function? It makes it flatter. You can imagine a world where I have a terrific idea and Zachary has a mediocre idea, but the noise is infinite, such that when you see the signal of our ideas, Zachary could get a gigantic awesome draw. Even if I get a pretty nice awesome draw too, the amount of noise in the system is still such that everybody’s like, “Oh my gosh, Zachary’s idea is great.” In the world where noise was enormous, we wouldn’t be able to tell; everybody would have the same win probability. That’s why that W̃ is getting a little bit flatter. But we’ve artificially kept the θ^ threshold the same, so the same person’s indifferent. Mechanically, what happens is we get fewer applications. That blue area is smaller, and that red area is all the applications we’ve lost. In this probabilistic world, what we’re losing is every person with these high thetas is just less likely to apply.

Then you say, “If fewer people are applying, you have more money. You could expand the budget.” That’s what we’re going to do next. We’ve added uncertainty, so we’re giving out fewer awards, which means we have budget slack. The funder naturally says, “Why don’t we increase the payline? We can fund more stuff.” This is where things get interesting. What happens when you bump that payline out? You also bump out the threshold that determines which scientists are applying. You’ve got to push X back into the low-quality distribution to get more people to apply.

What the funder is trying to do is say, “I always want to fund this many people. This is my budget-clearing thing. I’ve got to get my money out the door to scientists.” If we have noise and don’t change anything, we’re going to fund fewer people. We need to attract people to fill in the lost applications from this noise, and the only place to fill them in from is the inframarginal people, going backwards into lower-quality ideas.

If you increase the noise in a field, that will require lowering the payline to attract more people to apply, and the people you’re going to attract will by definition be marginally worse ideas, because they weren’t applying until you lowered the payline. That’s setting aside a lot of very interesting behavioral biases and all sorts of things that could be going on in the wild world of science. But you can see how, in this nice self-contained simple world, more noise is going to hurt the average quality of the people competing for that money.

The last step of this paper takes it into how this can raise some really interesting dynamic problems. To recap where we were: the proposition this paper makes is that if there’s more noise, it’s going to eat more of the budget of your institution. The logic goes as follows. On our prior slide, we showed how, when there’s more noise in a certain review period, you’re going to get more applications, because you’re going to have to lower the payline, so more people apply, and they’re lower-quality people as well.

This paper emphasizes the important point that a lot of the ways institutions operate in practice is that they tie budgets to applications. That is to say, your budget in period t+1 is some increasing function of how many applications you got in the period before. Now you can start to see where this is going to generate problems. If some area starts to get noisier, it’s going to get more applications, and it’s going to get more money. That’s going to be more money going to lower-quality applications.

Their empirical setup is looking at the European Research Council (ERC), because it’s a place where they have a decent way of measuring noise in the review process across different fields. This is me showing one of their figures, where on the x-axis is a measure of the inverse of the variance of the review scores. This is the inter-rater agreement.

A large number means that when a bunch of people see the same proposal, they tend to agree on the quality of that idea, and low numbers mean they tend to disagree. Already, in line with intuition, what you might call the hard sciences — biology, physics kinds of things — have more inter-rater agreement than the social sciences, where we’re dealing with the messy world of humans and we don’t have molecules that we can be very clear about all the time.

The hypothesis of this paper is: if the fields that have high inter-rater agreement start to have to compete for budgets with these low inter-rater agreement fields, those low inter-rater agreement fields are going to start to eat the budgets of the high inter-rater agreement fields. That’s exactly what they find in practice.

There was this big reform at the ERC where, instead of budgets being defined on very narrow levels, they started to expand it. If you had a noisy field through the lens of their model, that field would start to have more applications and would start to get more budget from other fields that had less noise. Going back and forth, you can see all of our noisier fields of the social sciences in green down here. The theory of this paper would suggest that if they started to have to compete for budgets with these up here, they would start to eat those other fields’ budgets, and that’s exactly what you see in practice.

There are lots of interesting things going on here, and taking noise seriously, and what it does to grant systems, is very important. It also raises all sorts of questions about what’s going on inside of peer review. If you’re interested in this, Kevin Boudreau has a very nice review paper that just came out a few weeks ago, focused on experiments done in this space. I call it the Wild West, because there’s lots of really interesting experimental work. Something that’s very unclear at this point is what the long-term effects of different peer-review arrangements are, because, as we talked about at the beginning with the war-on-cancer example, a lot of the real effects of science take so long to manifest that it’s hard to set up good experiments on what’s the right way to peer review, or what makes a good peer reviewer. These are lots of interesting open questions.

Designing grants

Back to our grant pipeline. I want to talk now about the design of the grants themselves: how much money, the structure of the grants, what can you do with the money, when should you be audited, all sorts of things. There’s lots of very interesting theory here. The one I want to focus on is a notion raised by Gustavo Manso’s paper, Motivating Innovation, which builds on a lot of earlier work on principal-agent models. If you want to get people to do high-risk things, you need to give them incentives that tolerate early failure and reward long-run success.

One of my favorite tests of this theory is a fun lab experiment that Florian Ederer and Gustavo Manso did, Is Pay for Performance Detrimental to Innovation? where they ran an adorable little lab experiment.

Imagine you have to play a 20-period video game and you have to sell lemonade. This is like some little game you would play on your phone, where you have to choose where you set up the lemonade stand, what price you’re going to charge, how much sugar you’re going to put in, etc. The analog is, you’re a scientist and you have to choose where you’re going to do your research, who you’re going to hire — lots of choices, and you don’t know what the payoffs are from all these different combinations of choices.

In the experiment, they have three arms.

1. Every period you will get a flat payment regardless of how much money your lemonade stand gets.

2. The pay-for-performance arm, which is, you’re going to get a fixed share of the profit your lemonade stand has.

3. The exploration contract, which has some analogues to the kinds of things you might think would be optimal for a scientific environment: you’ll get no payment for the first 10 periods, but then you get a higher profit share in the last 10 periods. Implicitly, you can see how the goal there is, you’re not getting paid at the beginning anyway, so try some stuff, see what happens, so that when your payments start to come in, you’ve found something that is high-reward.

They’re going to measure what this does in terms of how it affects people’s strategy, with the idea that — this is another use of variance as a measure — if this really does incentivize exploration, you should see people trying a lot of different things. So they’re going to look at the standard deviation of these different strategic choices. Then they’ll look at the actual amount of money these people have made, and they’ll also do some nice moderation, trying to see how much this might depend on these subjects’ risk aversion.

Here’s the graph plotting the standard deviation of the strategy choices across the three arms in the two chunks of periods, the first 10 and the last 10.

Exactly as the theory would suggest, those in the exploration contract tried a lot of different things compared to the others in the first 10, and then settled down. So there’s a first stage of sorts: they did different things. Was it actually helpful in the context of this game? The answer is yes.

On the left, it turns out the school was the best place to put the lemonade stand, so they’re showing the share of the subjects that found the school, and you can see the exploration contract is where basically everybody found it: 80% of the people found it.

The interesting thing here is the actual profits to the subject in the different arms. Whether you measure this by their maximum profit per period, or the profit in the last period, or their average over all of the periods, they did the best. They explored, they found the right combination of things, and once they found it, they stayed on. One other interesting thing is that, comparing the pay-for-performance contract, people who were risk-tolerant did pretty well in that contract, but people who were risk-averse fared quite poorly in the pay-for-performance contract, where you were getting a share of profits every period. Whereas in the exploration contract, your risk aversion didn’t matter, because you’re shielded from the risk entirely in those first 10 periods when you try different things. This was a proof of concept that this might be an interesting thing to really push on.

Then there’s a nice paper by Pierre Azoulay, Josh Graff Zivin, and Gustavo again, Incentives and Creativity: Evidence from the Academic Life Sciences, that does the best of trying to say, “What’s a real-world analog of this kind of arrangement with scientists?” They focus on the Howard Hughes Medical Institute (HHMI) investigator program, which does a lot of things that, compared to the traditional NIH grant that a lot of these people would be eligible for, look like the exploration contract: it’s longer funding, there are fewer reviews, there are fewer strings tied to what you can do with the money.

They wanted to say, “If we do our best to find two scientists, one who got an HHMI grant and one who looked as similar as possible on observables but didn’t get an HHMI and instead got an NIH grant, how would they fare?”

This is one of the results you can see in the paper.

If you focus on the weighted controls, you can see that our HHMI and NIH scientists are trending together towards this common period where they’re both going to get resources, but one gets HHMI and one gets the NIH, and afterwards they depart. This is by no means an RCT, because we don’t have the kinds of data on the HHMI shortlist to know who was the marginal HHMI that just didn’t get it and who was the marginal NIH person that just didn’t get it. The authors did their very best to find a good treatment and control group, but at the end of the day there’s still a question of how much of that gap is due to the HHMI structures per se versus HHMI being very good at spotting scientists who are talented.

Which is why my colleague Wei Yang Tham and I were then like, how much of this is due to the grant per se? That was our question in Money, Time, and Grant Design. How much is this really because people have more money or more time in their grants? We don’t have $1 billion that we can randomize to run that experiment, so we did what we thought was the next best: we ran a survey with actual scientists across all fields of science, not just the life sciences. We asked them what they would do if they got grants of different amounts and with different durations over which that money was available to them. The implicit argument being, if these contractual arrangements around the grant design really matter in practice, then when we give people fake long grants, they should say they’re going to be more likely to take risks in their work, as the theory would suggest, if the grant design itself was the important constraint on their risk-taking.

This was the setup of the question we asked.

Let’s say you got a grant, and we randomized the amount and the duration of these grants. Then we pretty simply asked them, “What they would do?” and we structured the possible answers to be things that aligned with the different theories about how these different incentives could influence scientists’ strategic choices.

In a certain way, this paper is a bit of a null-effects paper.

We’ve got our good old-fashioned statistical significance stars there. We see the obvious thing, which is that if we give people longer grants or more money, they’re very unlikely to say they’re going to move fast, which makes a lot of sense and is a bit of a sniff test that they’re taking these things seriously. If we give people more money, they say they’re slightly more likely to say they’ll take more risks. But if we give people longer grants with the same amount of money — that version of the exploration contract — they don’t say that that’s something important in terms of their risk-taking.

We do some work in the paper and we find that actually, if you look at well-resourced post-tenure researchers, when you give them more time, they are willing to take more risks. Those are the kind of people that look a bit more like people in the Pierre, Josh, and Gustavo HHMI paper, where the scientists are awesome scientists already. Whereas we have a very large population. When you zoom in on people, you can probably find some scientists for whom the grant design is a constraint, but on average, one way of thinking about this is that they’re not a large constraint.

We spent some work in the paper on actually estimating people’s preferences, getting back to the idea that if people have different preferences over the grant designs, then maybe grant design is less about changing what a given scientist does and might be more about changing which scientists show up to your grant door in the first place.

We’re able, with our data, to come up with good old-fashioned indifference curves, where you can say how willing scientists are to trade off dollars versus time.

One of my personal favorite things we do in the paper — you have to buy a lot of assumptions, but you can get an estimate of how willing funders are to make this same trade-off. It looks like funders are much more patient.

That is to say, if you look at the observed grant design, the trade-off that rationalizes what funders put out in the world makes it look like they’re very willing to trade off time and dollars, whereas scientists basically want the money now. So the constraint on their work is less time and more money.

All the earlier work there was very much focused on dollars and years of grants, and less focused on what happens if we put a funding opportunity on a particular topic. Can we get people to focus on that? That is what I focused on in this paper, The Elasticity of Science. I’m going to jump right to the conclusion, which is this graph.

What I’m able to do is say, if I want to move a scientist from point A to point B, and I want to make them make that move only by giving them grant funding, how much money do I have to give them to make that move? That is the elasticity of science I’m focused on in this paper.

This is a graph that says, for every percent change you want to get somebody to make in the direction that they’re pointed with their research, the costs are going to go up pretty quickly. If you look in the data, each researcher, every period, naturally is working on a new topic, and you can think of them as taking a step in a new direction. The standard deviation of steps you see naturally is about 25% of the mean. On average, everybody takes about a 25% difference to their next project. The question is, if I didn’t want to let endogenous natural forces lead people into certain directions, and I just wanted to take money and push them in a particular direction, that would take something like $3-4 million to get them to take a step of the size they naturally take anyway. I would get them to take it in the direction I want, but I would have to pay a pretty hefty price for them to do that.

For reference, the grants at the NIH are on this scale of $3-4 million, and that’s for doing the work of the project that they’ve agreed to already take the step towards. So one very loose way of thinking about this is that if you wanted somebody to do that same amount of work in a different direction, you would have to double the size of the grant. That’s a very casual way of interpreting this, but I think it conveys the magnitudes that I’m finding.

I think this paper was important because a very understandable default assumption for a lot of economic modeling is that adjustment costs are trivial and people can move around, which for certain types of work, especially in most of the 20th century, wasn’t the craziest assumption. Part of what I was trying to convey in this paper is that that is not a good assumption to make in science. It’s very difficult. What drives these costs? It could be that I just don’t want to do it. I’m a scientist and I’m willing to give up a lot of resources to not do that thing you want me to do. Part of it could be that I have a lot of expensive equipment that can only do certain things. There are lots of different reasons there that I haven’t totally unpacked, but the net magnitude is something to take seriously.

Now I want to spend the last bit of time walking through this wonderful paper, A Quest for Knowledge, by Christoph Carnehl and Johannes Schneider, because it’s one of the most interesting theoretical papers in the economics of science that takes direction very seriously as of late. I’m going to try to walk you through these slides in a way that can give you a start of some intuition as to what’s a different way of thinking about what the direction of science even means. Part of the beauty of this paper is it gives you a framework for how to think about what it means, in mathematical terms, to think about the direction of science.

What they do is, first, say there are truths about the world, and those truths are answers to questions. Let’s assume that we can line up every question on some arbitrary scale, and the answers to all of those questions are some number, and the path of that number is this jaggedy — the Brownian motion, that sounds very intimidating, but it’s this messy gray line that is the truth of the universe.

Now, what is knowledge? Knowledge is a point on that line where we know the answer to that question. That’s my red dot here. We know that the answer to Question 0 is 42. It’s nice to put mathematics on this; it sounds silly saying it, but this could be, “How hot do you have to get a PCR machine to adequately produce the right number of copies of the genome?” It’s 42 degrees. That’s the kind of thing we’re talking about here.

Then they bring in the very realistic thing — knowing this point exactly gives us more information than just that point. Since we know there’s some structure to the knowledge of the universe, we can make a conjecture about the answers to the questions around the question that we know the answer to.

You could think of these as probabilistic guesses. We don’t know exactly what the answer to Question -1 is, but we are pretty confident that if the answer to Question 0 is 42, the answer to Question -1 is somewhere between 46 and 38. In this case we’re wrong, as you can see, because we can see the truth, but that is our best guess given what we know. What happens if we learn the answer to Question -1.5?

That buys us our nice red dot up there, and it really shrinks the uncertainty around the truth in between the two questions we know the answer to.

Thinking of the world this way opens up a nice way of thinking about what the optimal direction of science should look like. It formulates it as a very specific thing in terms of where on this question number line we should try to answer the next question.

The way that Christoph is doing this graphically is: let’s go back and say, our x-axis is still our question line, and we know the answer to 0. Now I’ve scaled this by Q. Q is the thing that determines the planner’s uncertainty tolerance. If Q is very low, that means that as the planner — when we’re making decisions about things based on conjectures — we really want to know that what we’re going to do in the world, based on our guess of the answer to the question, we are going to be sure. If we really need to be sure, then we’re really going to be talking about projects taking very small steps in question space. Since we’re very risk-averse, we want to only think about taking very small steps, because the value of information is very high. If we’re very risk-tolerant, then we can jump around on our question number line a lot.

The y-axis — just think of that as the value of knowing the answer to that question. That’s what that triangle is. Don’t worry about the fancy math. That’s the value to society from knowing the answer to Question 0. The question is, where do we go next? Let’s go to -2Q.

What does -2Q buy us? It buys us that green space, the marginal value of answering that question. Why does it look like this? This little curvature is based on that curvature you’re seeing in between these two known questions.

We get some value because we now have more certainty; our conjectures are better about questions that are in between the two questions we know the answer to. This peak is because right in the middle is where we have the least amount of certainty. Our conjectures are the worst, which is why we cut out that value the most in the middle. The only thing we have full value for is these two things we know the actual answers to. That’s why it goes all the way down to zero there.

We could go to -2Q. That sounds like a good idea. That gets us some value. We could go to -3Q.

What’s the value of going to -3Q compared to going just to -2Q? Calculus takes you all the way. This is a really nice graphical way of showing that the marginal value of jumping all the way out to -3 is worth more than jumping out to -2. You just take the area of these two things and see how much value is created. It’s a bigger jump. You can see we have more uncertainty; we’ve carved out more of our value by taking that bigger jump, but we’ve bought ourselves more knowledge closer towards that new question.

Let’s keep going.

Well, no, because eventually you’re going to take a leap that’s so big that the uncertainty resolved in between your two answers does not actually buy you much. You can see in this area, going to -4 actually is not as helpful, because this marginal value of just going to -3 is a bigger chunk of value than going all the way out. The idea is, if we know the answer to this question and we shoot for answering a question that’s super far out, that’ll help us super far out there, but that does nothing to help us learn about what’s going on in between those two points.

Hopefully this is starting to give you intuition that the value of picking new directions is what it buys you in between things we already know, which also means that pushing the frontier can mean pursuing questions in between questions we already know.

In this case, let’s say we actually know 0 and we also know -6. Where do we go from here? You can see value can be created by shooting for Question -3 in the middle, because that’s resolving a lot of uncertainty about what’s going on in between zero and -6. In this setup, it’s optimal to shoot for the dead middle spot, because that’s where the most value is created. But this is where math is tricky and complicated: it’s not always the case.

If we know 0 and we know -8, it turns out that going to -4 is actually not optimal, and stretching a bit out to like -4.5 is actually optimal. This is where math is helpful but also shows you that intuitions can go awry sometimes.

Moonshots

One thing this paper walks through is this idea of moonshots, which is a super interesting concept that’s really hard to think about formally, but this model gives you a nice way of thinking about what moonshots do.

First, on the left, we’re going to walk through what happens in a world where we don’t do a moonshot. In the first period, t equals 1, let’s say we take a medium shot out to Question 3. In the second period, we let the market proceed in equilibrium. Where do people go? They say, okay, I can create the most value by going out to 5.1 in this case. The third period, where do we go? I push the frontier again and go out to 7.2.

What does a moonshot do?

A moonshot says, in Period 1, I don’t want to just settle for 3. I want to go all the way out to 6. That, in and of itself, in that first period, is worse — less value is created. But, in a dynamic sense, under certain conditions, it can actually create more value. If you let the market move forward in equilibrium from there, it says, “We know 6 now, where do we go optimally?” It turns out optimally is to fill in and deepen back at 3, because we can create a lot of value in that middle space that just opened up. Then, in the third period, where do we go from here? If we know 3 and 6, it turns out we can get further than we would have gotten without the moonshot. We can get all the way to 8.1. These magnitudes are irrelevant and very hard to think about in practice. The logic of what a moonshot does is, it makes the value of these connecting ideas, like the three over here, valuable.

To summarize it, I’ll take a phrase of Pierre Azoulay’s: this paper very much is about trying to get you to respect the geometry of knowledge, insofar as the optimal science policy is not a time-invariant thing.

It depends on where all our current knowledge points are. Does that mean we should take some deepening steps to fill in gaps, or does that mean we should take some expansion steps? It introduces this idea of the optimal flow of science looking like a cycle of deepening and expansion, where we throw out a fishing rod very far, we catch a fish, and then now we know where the fish are in between us and that fish, and we fish in between for a while, but then eventually we fill all that out. Then we throw out further again and expand, and then we fill in. That cycle nature is a very interesting concept to engage with.

Conclusions

Moving forward, and where I think things are going in these modern times. First, I think all of these things are still very relevant.

One of them that hopefully has shown through a bit in today’s discussion is that we’re getting a better sense of what matters. We can reject a lot of null hypotheses, but there’s a lot of work to be done in terms of how much a lot of these things matter. To the point earlier about framing research questions and noise, what are the important sources of noise? I don’t think we have a very good sense of those kinds of questions. Two big questions that I’m interested in right now.

“Allocative efficiency”

One is thinking more about allocative efficiency, which gets back to our graph from before.

A fun thing you can do with this graph is say, “Why are there residuals?” Let’s assume even for a moment that the WHO knows how to measure these things and this is a correct measure of demand. In the theoretical ideal, the only reason there would be residuals conditional on demand would be that productivity is different across these places. We don’t invest as much in musculoskeletal because it’s a really hard thing to study. Even though the demand is the same, the supply curve is just flatter, so we’re going to invest less, and that would be economically rational from a very classic utilitarian objective.

Just to give you a sense of how big these gaps are as something to investigate: I’m plotting here those residuals in percentage and dollar amounts, so you can see the scale of these residuals.

The yellow lines are just based on regressing the NIH budget on the US burden. The blue lines add in the global burden, which is important. You can see, for example, in the infection immunity one at the top, just looking at the US it looks like we invest a lot relative to demand, which could mean that infectious disease research is very productive. Or it could mean that we put a weight on infectious diseases that affect not the United States. In fact, that’s exactly what you see: once you include the global burden for infectious diseases, that residual cuts in half. So a lot of what might look like investment not due to demand in the US is actually investment because the US, by revealed preference, cares about the rest of the world. To the right extent is a very interesting question, but you can see that happening in that portion of the data.

In practice, we have no idea. There is so little research that uses the tools of what you would call modern industrial organization and productivity analysis of standard firm-based markets. We don’t have a very good sense of what this allocative efficiency looks like in science. I have a paper with my good friends Fabio and Wei Yang, Productivity Beliefs and Efficiency in Science, where we’ve done our best to take a stab in that direction, but I think that’s a very interesting place to pursue. It’s very hard, because a lot of the things you need to do these kinds of studies are not traditionally recorded, but I think it’s very important.

AI and scientific infrastructure

Last, I can’t not talk about AI, even though I don’t want to talk about AI. Something interesting with respect to the direction of science and AI is not who’s going to come up with the tool that makes science go super fast because it can discover drugs — that is a very important question. I, as an economist interested in social welfare, am interested in the question of how AI will change scientific infrastructure, in terms of how we allocate credit, resources, and the structure of science.

There are fun history examples of even seemingly small things.

AI is clearly a big technology, and if you look back at the world, it took one famous statistician saying, “I think 0.05 is a decent rule of thumb,” to create a tremendous amount of distortion in the publication market. There are lots of very interesting papers going out. P-hacking all traces back to Fisher and a sort of off-handed comment. Another one is citations. Citations initially were just meant to be bibliographic management and acknowledgements to people reading articles. Now they are at the core of many promotion decisions and all sorts of resource-allocation decisions and have real impact. This Hager, Schwarz, and Waldinger paper is a really good paper showing how that played out in practice in the early days of impact factors and citations. The reference to Goodhart’s law is: “Once a metric becomes gameable, it ceases to be a metric.” Once an input becomes an output of measure, you’ve lost the game, because people will just go after that thing and not the ultimate goal that you actually had in mind.

Within the context of AI, the way I’m currently thinking about it, in terms of where we’re allocating resources in the direction of science, is that a lot of how this used to play out worked via a currency called words. The cancer researcher would use big fancy words that convinced other researchers that they deserved money to pursue cancer research and not another infectious disease project. In a world where currency holds value, through the lens of traditional price theory, that can be an effective way of allocating resources, because the cancer researcher had a better idea, so it was easier for them to use words to get resources for their idea. Now, if Zachary truly has a better idea than me, but I can put my crappy idea into Claude and polish it up so that it sounds equally good, that starts to present some problems for how we allocate resources. This is not unique to science per se. People are already looking at this in labor markets — in recruiting, this is a very common problem, anywhere that words were a currency. I think clearly words were a currency in science, and so we’re going to have to think about how things are going to change in this new era, which is very exciting.

I think it’s a tremendous opportunity — if all new ideas are consumed via models. One challenge with all of the research on the direction of science historically has been that we couldn’t observe an idea, or define it, or see how my idea made its way to William’s work. But in a world where my idea is actually digitizable and goes through a model where we can measure its Shapley value, and William has a willingness to pay for OpenAI, who know the Shapley value, who can pay me, and I have data — I think really interesting technological advances might facilitate a more efficient market. That gives me hope amidst the wave of AI slop that appears to be oncoming.

I know Chad makes a very convincing argument that in the long run population is what matters, and when you’re thinking about growth rates. Ideas getting harder to find is a tremendously important idea.

But as a pragmatist who cares about growth levels and has a non-zero discount rate, I think there’s a lot of work to be done to increase productivity, even if it’s just a level effect. I think a lot of value can be created by taking grant design more seriously and taking AI more seriously as a part of the resource allocation.

• Metascience team lead Jenn Gustetic (formerly of NASA) and Senior Metascience Fellow Dan Turner-Evans (most recently of the US Senate) sharing pieces on the impact of AI on the talent pipeline (possibly bad) and on scientific research (probably good),

• Metascience Fellow Matt Esche, who has the longest tenure of anyone on the metascience team at just over three years, discussing the potential for new cancer vaccines, and

• Brand new Associate Metascience Fellow Hunter Wieman (coming from graduate school at Princeton University) reviewing an article on the challenges of conducting randomized controlled trials on AI.

Subscribe now

1. Social Science at the NSF (Matt Clancy, Abundance and Growth Blog).

In early 2026, when the National Science Foundation (NSF) published a budget request to Congress that would eliminate funding for the Social, Behavioral, and Economic Sciences (SBE) Directorate, I was surprised. In addition to funding important social sciences research, SBE manages important national surveys such as the General Social Survey (GSS).

While the motivation for eliminating SBE isn’t totally clear (the budget request said that NSF would continue grants that align with Administration priorities — perhaps the implication is that other grants do not), Clancy’s piece is helpful in showing that this isn’t the first time a president has targeted SBE. The Reagan Administration cut social science funding substantially, and Congress approved a rescission package of a third of NSF’s social science budget.1 However, skepticism about whether NSF should fund the social sciences goes back to its beginnings. As Clancy notes, some drafts of the founding documents for NSF included a ban on funding social sciences, though that language did not ultimately end up in the legislation.

Something I wish Clancy’s piece had delved further into is: why did the Reagan Administration and Congress cut social science funding in the 1980s? Based on what I’ve been able to find, even at the time it wasn’t totally clear.

A Science piece from April 1981 sounds like it could have been written in 2026:

Rumors have been rife, and many people believe that the Office of Management and Budget went over the entire federal budget with magnifying glasses looking for social research to stomp on. The proposed cuts for social and behavioral sciences at the National Science Foundation have been detailed in unprecedented fashion, and have done more than anything else to arouse paranoia among social researchers.

OMB’s 1981 explanation for proposed cuts was that “support for the behavioral social and economic sciences is being reduced significantly because much of the support of these sciences is considered of relatively lesser importance to the economy than support of the natural science.” As with now, it is perhaps a mix of a perceived liberal slant in social science and a belief that, if cuts to science are needed, social science is less important to the economy.

What is the status of SBE now? Contra reporting released in April 2026, SBE has not been shut down preemptively, and its grantmaking picked up in June. However, as of August 16, SBE has only spent about half of what it would have in a typical year.23

Subscribe now

2. The Mystery of Short Indian Children (Wilson King, In Development)

Here’s a mystery: why, despite rapid economic growth in India, have children remained short? This has not been the case in many other countries as they’ve developed economically (see the chart on economic growth and height in Africa). This is important because height, while not itself essential, correlates with health generally as well as cognitive development. As Wilson’s article states, the low height of Indians in India is not genetic; if their families move to wealthy countries, they will grow substantially taller.

Per Wilson, there are multiple potential reasons for this:

First, in India, there is a preference for sons, which might mean that daughters are malnourished. Second, there might be a preference for first children over subsequent children. Finally, India has lagged behind other countries in sanitation and hygiene despite its growth. That could cause children to remain sick, stunting their growth.

It’s likely a combination of these, and — while tackling something like a son preference is particularly difficult — improving sanitation is within the government’s power. Knowing this allows India to make policy decisions and financial investments that could improve the lives of millions of people and, through making them more productive, further enhance economic growth.

Why am I talking about height disparities in a science policy blog? It illustrates how social science, including all the papers discussed in Wilson’s article and linked above, can answer societally important questions.

I recently attended a workshop on the future of NSF’s SBE program, hosted by Arizona State University’s Consortium for Science, Policy & Outcomes. One of the discussions focused on whether there should be a dedicated unit at NSF that funds social science, or whether it should be spread out across the agency.4

I’m skeptical of breaking up a social science unit and using it to improve multidisciplinary projects because the primary benefit of social science research is not through interdisciplinary work with the physical and natural sciences. It’s through answering society’s most pressing questions. This piece on children’s height is a great example of this. Many of the questions that currently vex Americans can be answered, or at least better understood, through social science research:

• What is the impact of AI on the labor market?

• How did the COVID-19 pandemic affect children’s development?

• Why are people getting married at lower rates, and why are they having fewer children?

• Why do young people seem to be struggling with mental health issues? Is it all the screens?

• What are the impacts of immigration on economic growth and labor markets?

Not to mention questions about how to make science more efficient and increase the likelihood of breakthroughs (important metascience papers like this and this came out of SBE funding). You’re not going to answer these questions by sprinkling social scientists around geology and quantum computing programs.

Now whether social science actually does tackle problems important to most Americans is worth considering. Parts of the social sciences have gotten bogged down in poststructuralist approaches rather than focusing on empirical, socially important questions. But SBE doesn’t tend to fund those things anyway.5

Let’s keep funding social science at NSF.

Subscribe now

3. The Tragedy of the Cognitive Commons: How AI Could Disrupt the Regeneration of Professional Expertise (Nolan Lovett, preprint for Human Resource Development Review)

From Jenn Gustetic: AI is eroding the need for humans to conduct entry-level research tasks, and this paper suggests that might soon cause problems. I found the paper interesting because of its ties to the R&D management and implementation camp of metascience. It includes testable hypotheses about how AI is reshaping professional expertise. This may be particularly relevant to the R&D enterprise because declining STEM PhD admissions could compound the effects of AI.

Lovett proposes that by increasingly using AI for tasks that early-career employees used to do, our human capital across professions is experiencing a “tragedy of the cognitive commons.” While organizations make near term efficiency gains replacing entry-level research tasks with AI, the expertise depletes over time for the profession overall. The paper suggests two mechanisms that could weaken the future pipeline of employees with domain expertise (who can verify and refine AI outputs). The first mechanism is position elimination: fewer entry level hires. The second is reduced internalization of domain knowledge, through less cognitive struggle by entry-level employees in creating research products.

The paper posits that human capital systems need both “internalized mastery” and “distributed mastery” in the era of AI. Internalized mastery is the deep domain knowledge that is developed through sustained cognitive struggle and increasingly complex professional experience; it provides the foundation to do substantive validation. Distributed mastery is the ability to “orchestrate” AI systems to produce high quality outputs within a specific domain. Without internalized mastery you only get surface level validation, which decreases the quality of outputs over time.

Training and credentialing should create opportunities for both “tracks” within professions. If the stock of internalized mastery falls, substantive validation degrades, and validation error rates rise, then the quality of those professions decline. The paper proposes some policy ideas but suggests empirically testing the approaches. For example, governments might provide training subsidies in AI-exposed occupations, tax credits for firms that maintain developmental pipelines, or competitive grants to professional associations to experiment with governance approaches. We could also consider expanding federal training grants to the research community.

Perhaps these could be good metascience experiments.

Subscribe now

4. AI for science needs reasoning, not just data (Eric Schmidt and Suhas Mahesh, MIT Technology Review) and Conjecture Machines: AI agents and the new validation bottleneck in science (Don Wallace, Conor Griffin, Sean O’Neill, Thang Luong, Owen Larter; Google DeepMind website)

From Dan Turner-Evans: AI agents are becoming increasingly good at doing science, and current and former Googlers are taking notice. Eric Schmidt, the former CEO of Google, and Suhas Mahesh at Schmidt Sciences argue that agents, rather than custom models like AlphaFold, will ultimately be the form of AI that leads to the greatest acceleration of science. While most existing scientific datasets have already been used to train new AI models, agents can be continuously applied to any field of science. They can automatically log every condition and decision — thereby increasing the likelihood of reproducing any finding while also increasing institutional memory — as well as rapidly testing out new ideas in simulation or against any paper ever published.

Meanwhile, the policy team at Google DeepMind explains why agents have gotten so good at doing science, how they are being incorporated into the research process, and the policy implications of these changes. They argue that better models, stronger scaffolding around these models, and improved customizability have made agents immediately useful for scientists for mundane tasks, coding, and idea generation and validation. But they caution that (1) graduate training needs to be preserved in an age of AI, (2) not everyone has access to agents, (3) not all data is ready for agents, (4) AI-generated ideas must be validated, and (5) peer review is under increasing strain.

AI agents are already proving to be incredible tools for math and computer science, but I am not yet convinced that they can drive breakthroughs in lab-based disciplines that are limited by data and experiments rather than logic. However, I do find the idea of agents helping with reproducibility compelling, and I know many scientists across disciplines who are experiencing noticeable productivity gains from AI. Still, the DeepMind team is right that more work is needed to validate their outputs. I’m still trying to figure out what graduate training should look like in this new era, but I plan to share more thoughts on this soon (and welcome your ideas!).

Subscribe now

5. How to build a cancer vaccine, and whether they will work this time (Abishaike Mahajan, Owl Posting)

From Matt Esche: It’s been an outstanding summer for breakthrough cancer treatments. Pancreatic cancer, with its RAS mutation long thought to be ‘undruggable,’ saw the new drug daraxonrasib approved by the FDA. And with the positive clinical results from Merck and Moderna’s cancer vaccine for melanoma this August, abundant personalized vaccines for cancer treatments look a bit closer on the horizon. These results make me excited to think of all the therapies being developed now for previously untreatable diseases.

Mahajan’s blog post, which was written before the Merck/Moderna vaccine trial results, tempers my expectations a bit. Walking through decades of cancer vaccine approaches, the post shows how long it’s taken scientists to tease out the right approach, with a surplus of failures along the way. Even when an approach does work, the same design doesn’t simply translate to the next cancer, and we still don’t know why certain approaches in cancer immunology have failed.

To get there, we’ll need much more evidence and clinical results. There will be many more Phase 1 trials before we see widely-available personalized cancer vaccines. That’s why I’m glad to see movement on improving clinical trial policy, such as new pilots to improve the speed and quality of Phase 1 trials. Here’s to continued biomedical breakthroughs alongside clinical trial abundance bringing new therapies to market soon.

Subscribe now

6. AI RCTs are booming - to be useful they must evolve (Tim Ohlenburg, Oliver Hanney, Sharif Kazemi, Joseph Levine, and Shahrukh Wani; VoxDev)

From Hunter Wieman: As AI is poised to transform the economy, economists and funders are rushing to study its effects. Last month, Anthropic’s Economic Futures Research Fund announced a $200 million commitment to “support ambitious external research on interventions to prepare society for the economic impacts of AI.” Much of this will go to randomized controlled trials (RCTs) that study the effects of AI initiatives and AI adoption. The authors of this piece warn that the default outcome could be a glut of economic studies that fail to provide meaningful insight.

Designing economic RCTs that generalize is already difficult, and the authors believe that AI will exacerbate this issue. Two mechanisms stood out to me in particular:

• Testing Obsolete Models: Already, many studies of AI’s societal impacts use models that are practically obsolete by the time the study is completed; this could increase as AI progress accelerates.

• Contaminated Control Groups: Due to rapid AI adoption, control group participants may start using AI tools as the study progresses.6 And even if the control group avoids using AI at all, the post-RCT environment will be one where a large fraction of the population is adopting AI.

As a result, researchers conducting RCTs on the economic impacts of AI should ensure they design and test programs whose effects are not specific to a particular AI model, and whose results will remain relevant as AI improves and diffuses through the economy.

This is very hard, and I don’t have any silver bullets. But if I had to design an RCT on the social or economic effects of AI, I would focus on two questions:

Can this RCT help us identify mechanisms that are likely to persist as AI progresses? In a rapidly changing world, theory-agnostic estimates of treatment effects become less durable. Researchers should instead design RCTs around interrogating theories about the mechanisms linking interventions to outcomes (e.g., what broad characteristics of AI models allow them to augment human workers instead of replacing them). Understanding how and why something works can generalize to new environments in ways that isolated estimates of treatment effects do not. With AI, this understanding might help us anticipate which newer models will produce similar outcomes to current ones and which will have qualitatively different effects.

For each plausible trajectory of AI progress and diffusion, what questions do we want answers to? A study need not apply to every possible future to be a good bet. In a world with more uncertainty around the future, we should be focusing on high value bets that give important information in some specific futures instead of ones that will robustly give some information in all futures. For example, we could design some studies under the assumption that AI will replace significant swaths of human labor and others under the assumption that it will instead augment and empower workers. The ambition should be less to future-proof every RCT than to build portfolios of studies that collectively ask high leverage questions across different paths of AI capabilities and adoption.

Economists interested in metascience (those in the “innovation economics” camp) often talk about applying experimentation and econometric methods to science. This piece underscores just how difficult and important it is to design programs and experiments well so that these methods actually answer important questions.

Merton identified five norms that he argues govern scientific communities…

Communalism means scientific discoveries belong to everyone. No one owns E = mc² the way you own a patent. Scientists give up property rights in exchange for recognition and esteem.

Universalism: claims are evaluated on evidence, not the scientist’s identity. Talented people from anywhere can contribute — that’s the theory, at least. Maybe you’re smiling when I say that. But in theory, it doesn’t matter who speaks; it only matters what they have to say, and the substance of it.

Disinterestedness: that doesn’t mean scientists aren’t self-interested, but rather that institutional arrangements reward behavior that appears selfless….

Originality is completely central. The currency of science is credit for being first.

Skepticism enables error correction and filters unreliable claims through organized community scrutiny.

This concept was both new and immediately intuitive. This was my experience in graduate school, though no one explained it. The ethical code of science is as strict as it is unspoken.

Subscribe now

But as Pierre noted, this ethical code is imperfect, and there are challenges to some of these values. The Matthew Effect, in which famous scientists are rewarded for being successful, challenges the universalism norm. The presentation features this quote, which I enjoy.

“If I have not seen as far as others, it is because giants were standing on my shoulders.”

- Computer scientist Hal Abselson.1

This turns around the famous Isaac Newton quote about standing on the shoulders of giants, and gets at the phenomenon of the ideas of obscure or early career scientists being attributed to better known scientists (e.g. their advisors).

Learning about science as an economic institution is valuable because that’s one of the parts of economics that you can interact with regularly and one which you can (maybe, on the margins, especially if you work in policy) influence.

Subscribe now

Here are the readings that go with this lecture (readings with asterisks were required for the class):

• Dasgupta, Partha, and Paul David. 1994. “Toward a New Economics of Science.” Research Policy 23(5): 487-521.

• Merton, Robert K. 1957. “Priorities in Scientific Discovery: A Chapter in the Sociology of Science.” American Sociological Review 22(6): 635-659.

• (*)Azoulay, Pierre, Toby Stuart, and Yanbo Wang. 2014. “Matthew: Effect or Fable?” Management Science 60(1): 92-109.

• Murray, Fiona, Philippe Aghion, Mathias Dewatripont, Julian Kolev, and Scott Stern. 2016. “Of Mice and Academics: Examining the Effect of Openness on Innovation.” American Economic Journal: Economic Policy 8(1): 212-252.

• (*)Stern, Scott. 2004. “Do Scientists Pay to Be Scientists?” Management Science 50(6): 835-853.

• (*)Aghion, Philippe, Mathias Dewatripont, and Jeremy C. Stein. 2008. “Academic Freedom, Private Sector Focus, and the Process of Innovation.” RAND Journal of Economics 39(3): 617-635.

• (*)Furman, Jeffrey, and Scott Stern. 2011. “Climbing Atop the Shoulders of Giants: The Impact of Institutions on Cumulative Research.” American Economic Review 101(5): 1933-1963.

• (*)Hill, Ryan, and Carolyn Stein. 2025. “Race to the Bottom: Competition and Quality in Science.” Quarterly Journal of Economics 140(2): 1111-1185.

Thanks to William Higbie and Harry Fletcher-Wood for their support in producing this video and the transcript.

Subscribe now

Lecture Transcript:

Science as an Economic Institution

We’re going to talk about science as an economic institution. What is science, and how does it actually work? So far, we’ve treated scientific knowledge as an input to innovation — something that exists, gets produced, and contributes to technological progress. But we’ve mostly treated it as a black box. Today we’re opening the black box. We’re going to examine science from two distinct but complementary perspectives.

First, we’ll think about science as a type of knowledge. What makes scientific knowledge different from other forms of knowledge? Second, we’ll examine science as a social institution with its own incentive structures: what motivates scientists, and how does the reward system shape what research actually gets done?

These aren’t just academic questions. Understanding how science works has direct implications for science policy, for how we:

• Organize research universities,

• Structure funding agencies; and,

• Think about the relationship between privately funded applied R&D and basic research.

The policies we adopt — how much we spend on science, which fields to prioritize, whether to allow academic patenting, how to design peer review — all rest on assumptions about what science is and how it operates. It lays the foundation for all those other debates. That’s what we’re doing today.

We’ll start by asking what science is: what makes science different from another type of knowledge, like craft knowledge or engineering knowledge? Once we understand what science is, we’ll examine the relationship between science and technology. Does basic research flow naturally into applied research and then into commercial products? That’s the famous linear model that has historically justified massive public investments in basic science. We’ll look at evidence for this model, but also complicate the picture by examining institutional frictions that impede knowledge flow.

We’ll dig into how specific institutional features — access to frontier knowledge, funding, IP rules, access to materials — actually shape scientific productivity. The key insight, which probably won’t be that surprising, is that institutional details matter enormously for how much science gets produced and how effectively it contributes to innovation. Time permitting, we’ll talk about scientific competition as well.

What is science?

“What is science?” seems like a super simple question, but it’s actually hard to answer precisely. If you ask a philosopher of science, you’ll get debates about falsifiability, paradigms, research programs, and epistemic communities. Ask a scientist and they’re going to say something like, “I know it when I see it.” But they’ll struggle to articulate exactly what distinguishes their work from engineering, from craft, from trial-and-error problem solving.

But Harvey Brooks provides a useful framework for understanding the relationship between science and technology.

Science is about understanding why: testing hypotheses, refining theories, building systematic knowledge. Technology is about capturing and orchestrating regularities in nature to do something we believe might be useful. Brooks argues that science contributes to technology in six ways:

• New knowledge provides ideas for technological possibilities.

• Tools and techniques enable efficient engineering design.

• Research instrumentation migrates into industrial practice.

• Research develops human skills that transfer to applied work.

• Scientific knowledge enables assessment of social and environmental impacts.

• Knowledge bases make applied work more efficient.

But the relationship is actually bidirectional: technology contributes enormously to science, by raising novel questions and practical problems that reveal gaps in theory, and by providing instrumentation that enables more difficult research. You can just ponder the effect of Claude Code on academic research, happening right now, as an exemplar of how technology can impact the conduct of science. But the electron microscope, synchrotron radiation, superconducting magnets — all emerge from applied work and revolutionize basic science.

But Dasgupta and David ask a deeper question.

This is a classic article — if you’re into this topic you should really read it. It’s a manifesto for the so-called new economics of science. They say: given that science and technology are interconnected, why are they organized so differently? Why have universities doing open science alongside firms doing proprietary R&D? Their answer is that science and technology serve different goals, requiring different institutional structures.

Science adds to the stock of public knowledge — discoveries anyone can build on. Technology generates rents from private knowledge — innovations that provide competitive advantage to some firm. So this requires a different incentive system. Science operates under priority-based rewards and openness norms. Scientists compete to be first to publish, earning credit and reputation through disclosure. We’re going to talk about the Mertonian norms that support this system.

Technology operates under a completely different set of rules. Rewards come from secrecy and exclusivity: patents, trade secrets, proprietary know-how. You profit from being first to market and maintaining advantages — not from publishing.

Dasgupta and David argue that this duality is actually functional: both systems have internal logic suited to their purposes. But mixing these incentive systems can create tensions — when you ask scientists to both publish and patent, when universities must commercialize while maintaining academic freedom, that’s when there start to be real strains between those institutional boundaries. We want to understand this distinction to grasp how science contributes to growth, and how policy can enhance or impede that contribution.

Science as a social institution: the Mertonian norms

I want to shift from science as a type of knowledge to science as a social institution with distinctive norms and incentive structures. Robert Merton Sr., the sociologist of science — the father of the Nobel Prize–winning economist and my colleague at MIT, Robert Merton Jr. — identified a set of norms he argued should govern scientific communities.

Merton wasn’t claiming scientists actually follow these norms perfectly. He was describing an institutional ideal against which we can measure reality. The productive research agenda, for both sociology and economics of science, has been to examine who violates these norms, how, and why — and what consequences those violations have for scientific progress.

The violations aren’t just interesting as failures; they’re informative. They reveal the actual incentives that scientists face, the institutional pressures that shape their behavior, and the ways in which the system succeeds or fails at producing reliable knowledge. You can think of those Mertonian norms as a baseline model. Just as we use perfect competition as a benchmark to understand market failures, we can use Merton’s idealized norms to understand where the scientific system deviates from its stated principles, and why those deviations matter. Sometimes the violations are completely dysfunctional — they impede knowledge production. But sometimes they also reveal adaptations to real constraints. Either way, studying that gap between norm and practice tells us how science actually works, and how we might design better scientific institutions.

Merton identified five norms that he argues govern scientific communities, and there’s a cool acronym: CUDOS.

• Communalism means scientific discoveries belong to everyone. No one owns E = mc² the way you own a patent. Scientists give up property rights in exchange for recognition and esteem.

• Universalism: claims are evaluated on evidence, not the scientist’s identity. Talented people from anywhere can contribute — that’s the theory, at least. Maybe you’re smiling when I say that. But in theory, it doesn’t matter who speaks; it only matters what they have to say, and the substance of it.

• Disinterestedness: that doesn’t mean scientists aren’t self-interested, but rather that institutional arrangements reward behavior that appears selfless. We can talk about that one too.

• Originality is completely central. The currency of science is credit for being first.

• Skepticism enables error correction and filters unreliable claims through organized community scrutiny.

You may have ideas about how short we fall of actually honoring them. What Dasgupta and David argue is that these norms aren’t arbitrary — they’re functional adaptations that solve coordination problems in knowledge production.

• The priority system means that disclosure of results is actually a self-interested act.

• Universalism expands the pool of contributors.

• Skepticism filters unreliable claims.

Together, these norms create a reward structure that efficiently increases the stock of reliable public knowledge — even though they also create some problematic incentives.

The Matthew effect: a violation of universalism

I wanted to give you a sense of what it looks like to study a violation of those norms. The one I’m going to pick is the Matthew effect, which is a violation of universalism. Merton coined the Matthew effect; it’s named for a biblical verse — “unto everyone that hath shall be given” — that describes cumulative advantage in science. Scientists with early recognition get more resources, better collaborators, greater visibility, which leads to more recognition, creating a self-reinforcing cycle. The rich get richer.

The Matthew effect is a direct violation of the norm of universalism. According to that norm, claims should be evaluated based on impersonal criteria, not the author’s identity or reputation. But the anecdote you see on the slide shows this norm can be widely violated.

The same paper, the same ideas, different evaluation based solely on authorship.

One version of this is that it’s bias. Another version is that it’s partly rational: eminent scientists’ work gets more attention because they have proven track records, making the allocation of attention more fluid. But it also means that work by unknown scientists gets systematically undervalued. The Hal Abelson quote — “if I have not seen as far as others, it is because giants were standing on my shoulders” — tries to capture how your ideas sometimes get attributed to more famous people.

It’s an empirical fact that scientific output and rewards are extremely skewed. A small fraction of scientists produce most of the highly cited papers, get most of the funding, and train most successful students. But what explains this skew? There are two different interpretations. One is that science is fundamentally meritocratic: the most talented people just produce more and better work, and the reward system accurately reflects real differences in ability. The other is that the Matthew effect operates: early luck and accumulated advantages amplify small initial differences, so the skew reflects status effects rather than pure merit.

Those aren’t mutually exclusive, but their relative importance matters quite a bit for policy. Should we concentrate resources on proven stars, or does doing so amplify the Matthew effect and end up crowding out talent?

I’m going to tell you briefly about a paper of mine. The title is ‘Matthew: Effect or Fable?’ We were trying to make progress, understanding that it’s methodologically challenging to separate the two explanations I just outlined. Both predict the same outcome: skewed distributions of scientific success. So how do you distinguish between “good scientists get cited more because they’re good” versus “famous scientists get cited more because they’re famous”?

The setting here is biomedical scientists, some of whom receive a status shock — and the status shock is receiving a very prestigious appointment, the Howard Hughes Medical Institute investigator appointment. The way we’re going to do this is to look at the pre-appointment output of scientists that are at risk of receiving the status shock. The paper gets published; some of those scientists get appointed, so they get a discrete positive shock to their status. Then we can think about how the rate of citations to those articles treated by the appointment contrasts with what happens for articles associated with scientists that are very similar but somehow didn’t benefit from the status shock.

So it’s a classic difference-in-differences. This is, by the way, a research design we’re going to see multiple times today: we’re comparing citations to articles before and after some shock, for both a treatment group and a control group.

To give you a sense of what we’re doing: here are two papers.

They appear in the same journal in the same month. The last-author scientist, the principal investigator, is the big honcho in the lab. The person on the right gets appointed HHMI in 1990. Those two papers in their first year have roughly equivalent numbers of citations. Then we’re looking at how many more or fewer citations the paper on the right gets relative to the paper on the left. The whole point is to be pretty transparent about the research design.

This is what we find.

The bottom line is that there is such a thing as an HHMI appointment effect, which we interpret as evidence for the Matthew effect, but it’s not super big. It’s a few additional citations a year.

There are lots of claims about the importance of status in science, and scientists coasting on their prior reputation, that seem to imply those effects are absolutely massive. But whenever we look at them carefully — for example, in the paper by Carolyn Stein and Ryan Hill, ‘Scooped,’ they also look at the Matthew effect and find evidence for it — it’s much less quantitatively important than we might have thought. Maybe we should be slightly less cynical, in other words.

Why participate in open science?

There’s a fundamental economic question lurking beneath all this: why would anyone voluntarily participate in this system? The Mertonian framework and the Dasgupta–David model implicitly assume researchers want to be part of open science, but they don’t really explain why.

One way to think about this is from a firm’s perspective. I’m running a biotech firm. Why would I let my scientists publish? Publication spills knowledge to competitors. It takes time away from commercial work. Yet many firms, especially the most innovative ones, do exactly this. Similarly, from a scientist’s perspective, why accept a job that pays less if it means you get to publish?

This is where Stern (2004) becomes crucial. It’s a bridge between understanding science as an institution and understanding science as an economic phenomenon. That paper takes seriously the idea that there are inherent tensions between what researchers want — freedom to choose projects, the right to publish, the ability to collaborate openly — and what firms might prefer, which is to direct research toward commercial goals and keep findings proprietary.

The key insight is that these conflicts create a compensating differential in the labor market. If scientists truly value participation in open science, firms that allow it should be able to pay lower wages. Scientists are essentially paying to be scientists. But this raises a fascinating set of questions:

• How large is this effect?

• Does it apply to all scientists?

• What does it tell us about the returns to organizing research in different ways?

My colleague Scott identifies two different economic mechanisms that could explain why firms allow scientists to publish, and they have opposite implications for wages. First, there’s a preference effect. Scientists may have what economists call a “taste for science”: they derive direct utility from participating in open science. This could be intrinsic — for example, Richard Feynman loved physics for its own sake — or it could be about career concerns: if I work for Google and publish the transformer paper, even if I leave Google tomorrow, I’ve built enormous career capital. The labor-market implication is clear: if scientists value publishing, firms that allow it can pay lower wages. This is the compensating differential.

Second, there could be a productivity effect: firms themselves might benefit from allowing publication. The classic story here is absorptive capacity, a very important concept from Cohen and Levinthal in the late 1980s. To learn from external science, you need to participate in it. If you want to be part of the scientific conversation — to know what’s happening at the frontier and attract the best collaborators — you need your own scientists publishing. This is what people sometimes call science as a ticket of admission to the broader research community. But notice that if publication makes my scientists more productive, I might actually pay them more to compensate them for the extra value they’re creating. Any amount of rent-sharing then predicts that letting my scientists do science will be associated with higher wages.

So the question is, which effect dominates? That’s an empirical question, and a tricky one, because in the cross-section you’d expect firms that allow publication to employ better scientists. That creates a thorny identification problem: are science-oriented firms paying more because of the productivity effect, or because they’re hiring better people? Scott’s methodological innovation is designed to solve that problem.

The fundamental problem is unobserved heterogeneity. Suppose I observe that biotechnology firms allowing publication pay higher average salaries than firms that don’t. I can’t conclude much, because firms that allow publication probably hire better scientists — the ability bias could be enormous. The traditional approach in labor would be to look at job switchers, people who move from restricted to open-science environments. But as studies like Gibbons and Katz have shown, switchers are selected: people switch jobs for reasons correlated with their productivity, so you still have a selection problem.

Scott’s insight is really simple but really productive. Prior to accepting a job, scientists receive multiple job offers simultaneously. These offers vary in their scientific orientation — some allow publication, some don’t — and in compensation. But crucially, they’re all offers to the exact same person. So the market has already assessed this person’s ability, and that assessment is baked into all the offers. This lets you estimate a classic hedonic wage regression at the offer level with individual fixed effects. You’re asking: holding constant everything the labor market knows about your ability, do offers that allow publication pay more or less? It isn’t just a technical fix. It’s reconceptualizing how to think about compensating differentials when ability is unobserved. The key is exploiting variation in job characteristics for a given individual, which purges all the ability confounds.

This is the key regression table.

It’s a small sample — this was done at a postdoc conference. I was a grad student when that paper was being written, but it hasn’t aged. Here we’re regressing wages on whether I permit my scientists to publish. The key thing is that the sign of the coefficient flips if we control for individual fixed effects. What might have looked like a productivity effect in the cross-section turns out to be a compensating differential. Roughly, firms that allow publication pay their scientists 25% less.

Where should research happen?

That empirical result has fueled efforts to formalize Dasgupta and David and science as an economic institution. That’s the paper by Aghion, Dewatripont, and Stein (ADS). If scientists pay for freedom, who’s going to give them freedom? Because this subsidy from scientists to society is what makes open science economically viable. But that only works if scientists are actually getting what they’re paying for.

That leads to very standard questions: given that scientists value freedom, when should we organize science in academia versus the private sector? That’s what Aghion, Dewatripont, and Stein look at. They take the Stern compensating differential seriously and ask what it implies for when research should happen in academia versus the private sector. The traditional answer — if you read Arrow (1962) — to the question, “Why does academia exist?” relies on appropriability: if you can’t patent basic research, firms won’t do it, so we need universities.

That explanation is a bit problematic given how much intellectual property protection has expanded, especially under things like Bayh–Dole. So ADS 2008 offer a completely different rationale, based on control rights. Think about what fundamentally distinguishes academic from private-sector research. It’s not about funding sources or subject matter. It’s about who decides what to work on and how to work on it. In academia, scientists choose their own projects. That’s not just a nice feature — it’s definitional. That’s what tenure protects. In firms, managers can and do tell scientists what to work on. That’s also definitional. That’s what ownership means.

Now we can connect this with Stern. We know that scientists value creative control and will accept lower wages to get it. That creates a fundamental trade-off. Academia’s advantage is that you can hire scientists cheaply, because you’re giving them what they want: freedom. The disadvantage is that they might work on things that they’re intellectually interested in but that are economically useless. Private-sector firms can direct scientists toward high-payoff projects, which is valuable, but they have to pay scientists more to compensate for taking away their autonomy.

The question becomes: when does the benefit of cheap labor outweigh the cost of misdirected effort? ADS formalize this trade-off in a very elegant model that generates sharp predictions about the optimal organization of research.

Imagine you’re trying to develop a new drug or technology. That requires going through a number of stages — let’s say k stages: maybe basic science, then proof of concept, then clinical trials, then manufacturing. You only get paid when you complete all the stages. Each stage is risky, even if you’re working on the right thing.

Here’s the key choice. You can organize research in two ways. In the academic mode, you cede control to scientists. They choose what to work on and how. The advantage is you pay academic wages — let’s call that Wᴀ. The disadvantage is that scientists sometimes choose directions that serve their own interests rather than commercial goals. With probability α, they happen to choose what you’d want anyway — their interests align with the commercial value. But with probability 1 − α, they go off and do their own thing. They still get utility Z from pursuing their own research agenda.

In private-sector mode, you retain control. You tell scientists exactly what to work on. They always work in a commercially valuable direction, but you have to pay higher wages — let’s call this Wᴘ — to compensate them for giving up autonomy. The cool thing about this model is that it captures the Stern compensating differential, but embeds it in a sequential, cumulative innovation process.

The question becomes: at what stage of the research process should you switch from academic organization to private-sector organization? Should early-stage research happen in the university and late-stage in firms, or vice versa? The model gives us a clean answer, and I think it has a really cool intuition.

You start with the very last stage. You’re one step away from commercial payoff V. Should you organize this as academic research or private-sector research? If you cede control to scientists, with probability α they do what you want and you get pV; but with probability 1 − α they pursue their own interest and you get nothing. On the other hand, you save money on wages. If you retain control, you always get V, but you pay more. The firm retains control when expected benefits exceed the wage savings, which is basically Z, the utility scientists get from autonomy.

Now step back one stage. You’re two stages away from commercialization. Should you retain control or cede it? Here’s what’s changed. The benefits from control are now further away and more uncertain. You’re not getting V next period; you’re getting the option value of maybe reaching the next stage, which then gives you another option value. These option values are smaller than direct payoffs because they’re risky and discounted. But the wage savings are immediate and certain — you’re paying academic wages right now.

So at earlier stages, the wage advantage of academic organization becomes relatively more important, and the focus advantage of private-sector organization becomes relatively less important. The result is an endogenous cutoff point. Early stages should happen in academia — that’s where wage savings dominate and where you want exploration anyway. Later stages should happen in firms — that’s where focus becomes valuable enough to justify the wage premium. The model tells you exactly where this transition should occur, based on the parameters P, V, Z, α and the wage differential. There are really cool extensions as well — for example, when a research line can branch into multiple projects.

Does science matter for technology?

Understanding how science works as a social institution is important, but there’s an elephant in the room we haven’t addressed: does this matter at all for technological progress? Could we just fund applied R&D directly and skip the whole scientific enterprise?

Vannevar Bush, writing his famous 1945 report ‘Science, the Endless Frontier,’ articulated what we now call the linear model: the idea that basic science flows naturally into applied science, which generates technology, which produces economic growth. It’s an elegant story, and it provided the intellectual foundation for massive public investments in basic research. In the US, the NSF, the NIH, the growth of research universities — all of this was justified by appeal to the linear model.

You might ask: is Bush right? That’s what we’re going to examine in this section. We’ll look at evidence for the linear model, but we’ll also complicate the picture considerably by examining how science and technology interact. Spoiler alert: it’s messier than what Bush thought, but it’s also more interesting. The key insight we’re going to develop is that science doesn’t just generate specific findings that get applied; it provides something more fundamental — a map of the technological landscape that helps innovators search more effectively.

Let me tell you my favorite example of the linear model in action: Thomas Brock and the discovery of Thermus aquaticus.

It’s 1967, and Brock was literally hunting around Yellowstone National Park studying extremophiles — organisms that survive in extreme environments. In the hot springs of Yellowstone, where water temperature fluctuates widely, he discovered the bacterium Thermus aquaticus, which could survive extreme temperature variation.

This was pure curiosity-driven research. Nobody was thinking about practical applications. Brock published his findings, characterized the organism, and — crucially — deposited it in the American Type Culture Collection (ATCC), a research-materials repository. He got tenure. Great. That’s how science works.

Now fast-forward 16 years. Kary Mullis, working for Cetus Corporation, was trying to develop a technique for amplifying DNA. The problem is that you need a DNA polymerase enzyme that can withstand repeated temperature cycles. This was hard — most enzymes degrade at high temperature. After considerable searching, someone (accounts differ on whether it was a graduate student or someone at a seminar) pointed Mullis to Thermus aquaticus, sitting in the ATCC repository. Its DNA polymerase, Taq, was exactly what was needed.

This led to the Polymerase Chain Reaction, which became absolutely foundational for all modern molecular biology and biotechnology. When you see the show CSI, every time someone gets genetic sequencing, it all depends on PCR. Mullis won the Nobel Prize, Thermus aquaticus was named molecule of the year, and the patent on PCR was worth about $500 million.

That’s the linear model in its classical form: curiosity-driven basic research generates knowledge that, 16 years later, becomes the foundation for a transformative technology. The usefulness of extremophiles was completely impossible to anticipate ex ante.

Science as a map

Here’s a puzzle. More science leads to more technological progress. Yet only a minority of patents directly cite scientific papers. So how is science helping?

An important framing point for the papers I’m about to talk about: they all examine science as a type of knowledge, as an input into technological production. They’re not about science as an incentive system — the priority system, the Matthew effect, compensating differentials. None of that is front and center here. Instead, these papers ask: what does scientific knowledge do for inventors? What is its functional role in technological search?

We’re going to begin with Fleming and Sorenson. Their answer is: science provides a map. Scientific knowledge tells us how the world works — underlying principles and mechanisms. Technology captures and orchestrates these regularities to do something useful. But new technology steps into the unknown, either using novel processes or combining existing ones in novel ways. Science can help inventors navigate the uncertain, unknown terrain. It provides an imperfect map of the space. It tells you where to look, what combinations might work, which are likely to fail. Science reduces search costs by providing cognitive guidance.

Here’s the key insight: not all technologies benefit equally. Some technologies are forgiving — you can combine components many different ways and things still work. Think modular systems: for these, trial and error is fine; you don’t need deep scientific understanding. But other technologies are what you might call fussy: components must combine precisely, or everything falls apart. Small changes can cascade into big problems. For these tightly coupled systems, trial and error is prohibitively expensive. So having a scientific map becomes valuable, because it lets you avoid the vast space of failed combinations and directs you toward the narrow space where success is at least possible.

That’s what Fleming and Sorenson test: does science matter more for fussy technologies? That’s the key result. It’s an interaction effect in a cross-sectional regression.

Patents that cite science always tend to receive more citations than patents that don’t — that’s the baseline science premium we’ve known about for a long time. But what they show is that the gap between patents that refer to science and patents that do not is actually much wider for those so-called “fussy” technologies.

Let’s try to unpack what that means. If you’re inventing something where components are loosely coupled — modular, flexible, forgiving — you can afford to do a lot of trial and error. Having scientific understanding helps a bit, but the gains are modest; you’ll probably figure it out eventually through experimentation. But if you’re inventing something where components are tightly coupled, where small changes in one component cascade through the system and break everything, trial and error is very expensive. The search space is huge and most of it leads to failure. So having a scientific map, which tells you which regions of that search space are worth exploring, becomes really valuable.

There are follow-on papers that do this much better. One is the paper by Ahmadpoor and Jones called ‘The Dual Frontier.’ But Fleming and Sorenson was the first paper of that type, and it’s been very influential.

Pasteur’s Quadrant: basic and applied at once

We’ve seen evidence that science matters for technology through multiple mechanisms. That raises a natural follow-up question: how should we organize the boundary between the activities of science and the activities of technology? What kinds of institutional arrangements best support research that connects scientific understanding with technological application?

The linear model gave us a clean answer: keep them separate. Universities do basic research; firms do applied research and development; knowledge flows from universities to firms through publications. Simple. But is that how the world works?

To tackle this question, we need to rethink some categories. The linear model treats basic and applied as sequential stages: first you do curiosity-driven basic research, then someone else does goal-oriented applied research. But as it turns out, that’s too simple, because research can live in both worlds simultaneously. It can be both basic and applied. So you get an inherent tension between the scientific norms and the commercial norms. Let’s explore that.

What you’re getting in this lecture is a bit of a greatest-hits of examples. We keep coming back to those examples because they can really help tell the story. Thermus aquaticus was one. Another is the OncoMouse.

In 1984, Philip Leder and Timothy Stewart at Harvard Medical School developed the first mouse with genes inserted that predisposed it to cancer. They used an oncogene — a gene that, when activated, promotes tumor development. This is simultaneously a scientific breakthrough and a very practical tool.

On the scientific side, it demonstrates that you can insert specific genes and produce predictable effects on cancer susceptibility. They’re testing a fundamental hypothesis about the genetic basis of cancer. It’s advancing our understanding of why: why do organisms develop cancer? What’s the role of specific genes?

At the same time, it’s a how. How do you test potential cancer therapies? If you want to develop new drugs, having mice that reliably develop particular forms of cancer is enormously valuable. You can test whether your drug prevents tumors, shrinks existing tumors, extends survival. So this OncoMouse is simultaneously answering a scientific question and providing a very practical research tool.

Leder did what good scientists do: he published in Cell, one of the top journals. The scientific priority is established. But Harvard also did what universities increasingly do post Bayh–Dole: they filed for a patent, which was granted in 1988, and licensed the patent exclusively to the firm DuPont.

This is where the friction starts. DuPont didn’t just want to sell mice. They wanted reach-through rights: the idea is that if you use the OncoMouse to develop a therapy, DuPont gets a piece of the action. They wanted the right to review publications using their mice. This created enormous tensions with the research community. Scientists felt these restrictions violated the norms of open science. But from DuPont’s perspective, they’d made a major investment and wanted to earn a return on it.

This framework, developed by Donald Stokes, reconceptualizes the basic/applied distinction in a way that’s much more useful than the linear model.

Instead of treating basic and applied as sequential stages, Stokes says they’re independent dimensions. On one axis you have quest for fundamental understanding: are you trying to advance scientific knowledge about how the world works? On the other axis, considerations of use: are you motivated by a practical problem? Are you trying to develop something useful?

This gives you four quadrants:

• Pure basic research — high on fundamental understanding, low on considerations of use. We could call it Bohr’s quadrant. Niels Bohr wanted to understand the structure of the atom; he didn’t care about applications. Thousands of papers, zero patents.

• Pure applied research — low on fundamental understanding, high on use. We might call that the Edison quadrant. Thomas Edison ran thousands of experiments at Menlo Park trying to make better light bulbs, better phonographs, better everything. He explicitly didn’t care about publishing scientific results; he refused to let anyone publish about the Edison effect, even though it was named after him. Thousands of patents, zero papers.

• Use-inspired basic research — high on both dimensions. This is the Pasteur quadrant, named after Louis Pasteur. Pasteur was working as a consultant for the French wine and dairy industries — very applied problems: why does wine ferment? How do we preserve milk? But his investigation led to the germ theory of disease, one of the fundamental scientific breakthroughs of the 19th century. Pasteurization is simultaneously an industrial process and a profound scientific insight.

The most important research often lives in Pasteur’s quadrant. It’s simultaneously advancing fundamental understanding and solving a practical problem. The OncoMouse and most modern biotech live in Pasteur’s quadrant. This creates inherent tensions, because you’re trying to satisfy both scientific norms and commercial incentives at the same time.

Openness as a natural experiment: Of Mice and Academics

The next study is Murray et al., poetically named ‘Of Mice and Academics.’ These authors — you can notice the author overlap with the ADS 2008 paper — examine a natural experiment in openness. They’re testing the ADS framework empirically using the OncoMouse story.

The theoretical framework from Aghion, Dewatripont, and Stein suggests two distinct effects of openness. Vertical exploitation means more follow-on research: if access costs are lower, more people can build directly on existing work. But there’s also horizontal exploration: openness might enable entirely new researchers to enter the field and pursue novel research directions that wouldn’t have been explored under restricted access.

Here’s the natural experiment. By the mid-1990s, DuPont held patents on two crucial mouse-engineering technologies, Cre-lox and OncoMouse. These patents covered hundreds of different engineered mice, and access was highly restricted — complex licensing agreements with reach-through rights on anything you discovered using the mice, which created significant delays and costs. The research community lobbied for years for more open access.

In 1998 and 1999, the NIH director Harold Varmus — who had won the Nobel Prize and understood the scientific stakes — negotiated memoranda of understanding between the NIH, DuPont, and the Jackson Laboratory, the mouse repository. These agreements dramatically reduced access costs for academic researchers: simple one-page material transfer agreements replaced complex licensing, and the Jackson Lab was committed to distributing mice openly.

The point is that this memorandum was largely unanticipated in its timing and its scope. There had been lobbying for decades, so researchers couldn’t time their work around it. The agreements covered the entire population of DuPont-controlled mice, so there’s no selection bias. Crucially, there’s a natural control group: knockout mice and spontaneous-mutation mice were created using different methods and weren’t subject to DuPont’s restrictions, so their access costs didn’t change. This lets Murray and co-authors implement a very clean difference-in-differences design.

This is the key result for vertical exploitation.

Here again, I’m not doing full justice to the paper — but it’s a great paper, you should absolutely read it. The figure shows annual citations to mouse articles over time. What’s a mouse article? Each time you create an engineered mouse like this, there’s a paper associated with the mouse. You can then think about the extent to which follow-on researchers cite this mouse in their own research. You can do it for treated mouse papers and for control mouse papers — hence the diff-in-diff.

The figure shows the annual citations to mouse articles over time, comparing treatment — OncoMouse and Cre-lox — to control: knockout and spontaneous. The vertical line marks the NIH agreements in ’98-’99. Before the agreements, the two groups track each other relatively closely. Citations to both treatment and control mice are declining over time, which is natural: all papers get cited less as the field moves on. But the rates of decline are parallel. This validates the parallel-trends assumption.

After the agreements, you see a clear divergence. Citations to treatment mice increase substantially relative to the control group. The boost is 20-30%, depending on the specification. So that’s vertical exploitation: more follow-on research using these now-open research tools. This makes intuitive sense. Before the agreements, many researchers who might have wanted to use OncoMouse or Cre-lox mice were deterred by access costs and licensing restrictions. They either used different tools, chose different research questions, or left the field entirely. After the agreements removed the barriers, more researchers could access the tools they needed, and follow-on research accelerated.

Notice one thing that didn’t happen: citations to the control group don’t decrease. This isn’t researchers substituting from knockout mice to OncoMouse. If that were happening, you’d see the control group decline when the treatment group rises. Instead, both increase, with treatment increasing more. The openness shock expanded the total amount of research; it didn’t just redistribute research across tools. This is important, because it suggests the pre-agreement restrictions were genuinely blocking research that had positive social value, not just creating a queue that would have happened anyway. There are additional findings that are very interesting in this paper. There’s also the horizontal exploration effect: they provide evidence of much more diverse experimentation.

Funding design and tolerance for failure

We focused on one particular institutional friction with this paper: what happens when scientific and commercial incentives collide at the frontier, especially in fields like biotech that live in Pasteur’s quadrant. But there are many other ways that institutional design affects productivity within science, independent of commercialization. How do scientists access frontier knowledge produced elsewhere? How do funding agencies decide which research to support? How do scientists access physical research materials they need to build on each other’s work?

This is where I’ve done most of my research. I’ll tell you about just one paper. It’s about science funding, tolerance for failure, and scientific exploration. It has been and will continue to be a very fertile area of research. It’s called ‘Incentives and Creativity: Evidence from the Academic Life Sciences.’ We want to think about whether what explains scientific productivity at the individual level is not just the amount of funding people receive, but how they’re funded — the incentives inherent to the funding contract.

What we want are scientists that are at risk of receiving grants with different incentive structures. We want to see if particular types of grants are associated with greater scientific creativity. To do that, we study the Howard Hughes Medical Institute (HHMI) investigator program. I talked about it earlier with the Matthew effect study. HHMI is the most important private funder of academic biomedical research in the US, and probably the world.

Roughly every three years they select 50 youngish biologists from elite institutions. It’s very prestigious — that’s why we studied it as a status shock. If you get appointed to HHMI, it becomes part of your institutional identity: you might be MIT and HHMI, or Sloan Kettering Institute and HHMI. But it’s also a major source of funding for those scientists.

What does HHMI tell people? They say: “Your job is to change your field.” Here is a pot of money; in five years we’re going to ask whether you’ve used that pot to change the nature of the questions being asked in your field. It’s not just a prize. A prize would be: “You won, we give you money, and then our job is done.” This has real bite, meaning an HHMI appointment is something you are at risk of losing.

What’s important and interesting is that I said it’s five years — that’s not true. It’s really ten years, because at the end of the first five years as an HHMI, what HHMI wants to know is: have you used the freedom you were granted? Have you at least tried? The axe is really going to fall at the end of the second five years. So ten years in, they start asking, “You’ve had ten years to try to change your field, have you?” At that point lots of people are being discontinued — kicked off the program. But that’s a much longer time horizon relative to the NIH R01 grants, which are the standard government grants in biomedical research in the US, where it’s 3-5 years. It’s really not forgiving of failure — people are going to scrutinize all the stuff you’ve published in the last 3-5 years.

Importantly, the NIH funds you for a particular project, whereas HHMI funds the person. That means if, in the middle of your 5-10 years, you decide that the idea you had isn’t going to work, you can just use the funding to do something else instead — which is not possible at the NIH.

What we did in this project is — we’d have loved it if HHMI were ready to randomize after a first cut of selection into their program. They’re not willing to do that. They also don’t really have a regression-discontinuity design — a hard cutoff. What we ended up doing is selection on observables. This is the picture.

It’s a kind of synthetic-control study, if you want to place it in the methodological landscape. We have lots of different outcomes, but relative to the controls, the HHMIs don’t produce more papers — but they produce more important papers. They also fail more often, meaning they produce more papers that are complete citation duds. They swing more for the fences than the controls. The controls revert to their mean, whereas the treated scientists really expand and enlarge an impactful research agenda following selection into the program.

I’m papering over lots of details, and you can read it. This paper was published in 2011. There are tons of things that are wrong with it, and we try to level with the reader on that. It’s totally possible to do much better than what we’ve done. But today it’s my second most highly cited paper, and that’s because it was the first one that asked that type of question.

Synthesis

Let me try to synthesize what we’ve learned.

We asked whether science is more than just a type of knowledge, and the answer is yes. Science is a distinctive economic institution with specific incentive structures that solve market failures in knowledge production. The priority-based reward system, where scientists compete to be first to publish, creates incentive compatibility: it aligns private incentives with the social benefit of rapid knowledge disclosure.

This solves what we might call the paradox of directly paying for ideas. How do you reward discovery when knowledge is non-rival and hard to exclude? The priority system does this by turning knowledge production into a winner-take-all race. Actually, it turns out it’s not winner-take-all, but maybe winner-take-most, where the reward is reputation rather than control rights over the knowledge itself.

The question you want to ask is: is science really central to endogenous growth and technical progress? We’ve seen some evidence today that disruptions to open science reduce both scientific productivity and downstream technological innovation. Institutional design matters just as much as funding levels. The details — how we structure access to knowledge and materials, how we design funding mechanisms, how we balance openness against IP — those are first-order policy instruments. Strategic behavior can undermine open-science objectives, and the institutions that worked well in 1950 may not be optimal for 21st-century science. The big question is: how should we evolve them? That’s one of the frontier questions for both policy and research.

He (Waldinger) estimates the effect of shocks to human and physical capital during World War II. He can look at random destruction in physical capital — university buildings in Germany — and at departments that lost star scientists. He uses the dismissal of Jewish scientists in Nazi Germany between 1933 and 1944 as the shock to human capital. He attributes about a 33% decline in publications, at the department level [to dismissals]. He contrasts that with the destruction of universities during the Allied bombing campaign...He’s able to use this as an exogenous shock — some departments were affected by these bombings and some weren’t…What’s striking about it is that the decline in publications is much larger relative to the decline due to the physical capital. He has this quote: “The dismissal of scientists in Nazi Germany contributed about nine times more to the decline of German science than physical destruction during World War II.”

On one level this might seem obvious — buildings don’t write academic papers; scientists do. But we might expect that access to working labs would be necessary for scientists to produce research (especially before the advent of personal computers and the internet). And it is important, but not nearly as crucial as having talented colleagues.

Good science (depends on) the friends we make along the way.

Subscribe now

Here are the readings that go with this lecture:

• Agarwal, R. and P. Gaule. Invisible geniuses: Could the knowledge frontier advance faster? American Economic Review: Insights, 2(4) (2020): 409-24.

• Bell, Alexander M., Raj Chetty, Xavier Jaravel, Neviana Petkova, and John Van Reenen. Who Becomes an Inventor in America? The Importance of Exposure to Innovation. Quarterly Journal of Economics, 134(2) (2019): 647-713.

• Ganguli, I., P. Gaulé, and D.V. Čugalj. Chasing the academic dream: Biased beliefs and scientific labor markets. Journal of Economic Behavior & Organization, 202 (2022): 17-33.

• Waldinger, Fabian. Bombs, Brains, and Science: The Role of Human and Physical Capital for the Production of Scientific Knowledge. The Review of Economics and Statistics, vol. 98, no. 5 (2016): 811-831.
  

  Subscribe now

Thanks to William Higbie and Harry Fletcher-Wood for their support in producing this video and the transcript.

Subscribe now

Lecture Transcript:

The Supply of Innovators

My name is Ina Ganguli. I’m a professor in the economics department and the management department at the University of Massachusetts Amherst. I was trained as a labor economist, then I got into the economics of science and innovation — my research is at the intersection of those two fields.

I’m going to talk about what I call “the supply of innovators” and knowledge production, which is at that intersection. I’ll tie it back to things that Ben and Chad talked about — going from the macro to looking very micro at the decisions that people make. Then I’ll take you back to the early days of what labor economists who worked on these topics were thinking about, going through some models and conceptual approaches. I broadly call it “earnings” and “entry into science.” We’ll go through several models and some of the empirical evidence. I’ll talk about:

• Cobweb models,

• Compensating differentials. One of the papers that I’ll cover looks at how much scientists pay to do science. There’s a well-known Scott Stern paper.

• The Roy model, which is a way to think about selection: who self-selects into the innovative sector.

• Then, getting into the broader labor economics literature, what have we started to look at in this area of the economics of innovation: biased beliefs, exposure to science, diversity and the economics of talent.

John van Reenen will talk about this too, but there’s this literature on the lost Einsteins that’s gotten to be quite popular recently — looking at what happens to talent that isn’t nurtured.

We’re going from the macro: thinking about endogenous growth models and how knowledge is produced. Those of us who work in this area on labor and innovation think very micro: what are the choices that individual people make?

Here we think of a simple Romer-type model.

LA is the number of people searching for new ideas. If we’re going to have ideas produced, you’re going to have people who are searching for them. Things are changing now — we can think about AI replacing those people. But back to the basic model: we have LA, and if you increase LA — the number of people — then you will get more ideas produced. We’re going to examine LA. Who’s in it? The number of people.

If we think about how to measure that — who is it that’s searching for new ideas? — there are different ways you can think about it.

• People who work a lot with patent data will look at inventors: people who have invented an idea that’s been patented, conditional on patenting.

• A lot of my research — the research of those of us who work on scientists — will use publication data and then back out scientists from that.

This is data from different sources — an estimated number of researchers across countries.

You can see that in most countries it’s increasing. I’m not going to have time to show a lot of data, but I’ve done some work recently with Megan MacGarvie where we’ve looked at international students studying in the US and trends from different countries. There’s work showing that maybe these researchers haven’t been that productive: the number of researchers is going up, but whether they’re actually producing as many new ideas is something that people have been looking at. Right now in the US — but I think in other countries too — this topic is very policy relevant. There’s a lot of focus on what’s happening to the scientific workforce.

There was a recent Science article talking about different aspects of the Trump administration’s agenda and that it’s likely to reshape the scientific workforce. Why is that? What kind of policies are we talking about?

Immigration: restricting immigration from certain countries, or making it more costly to get an H-1B visa — which is what you get if you’re coming as a high-skilled worker to work in a company. Now that’s expanding to universities. Michael Clemens will talk much more about that.

There’s also government scientists — there have been a lot of reductions in the government scientific workforce. There’s funding that has been reduced, and so fewer students can be hired to be graduate students. There’s a lot going on right now. I will say, as someone who works on these topics, in the last year I’ve had a lot of journalists come and ask me: “What’s the evidence? Will there be a brain drain? What will the impacts be on US science?” Those of you who are interested in this topic, it’s an important time to be working on it, and I hope some of you will think about getting into these issues.

If we’re thinking about the different inputs into the knowledge production function, there’s human capital, but also physical capital — the lab equipment. We can think about computers, AI potentially: how do we think about that — is it human or not? Here I want to think about the people. How important are the people?

I really like this paper by Fabian Waldinger. He’s done a lot of great work on World War II and science, and in particular the emigration of Jewish scientists from Germany. This paper focuses on this question of human versus physical capital. It’s a motivating piece of evidence for me when I do my work on the decisions that scientists make.

He estimates the effect of shocks to human and physical capital during World War II. He can look at random destruction in physical capital — university buildings in Germany — and at departments that lost star scientists. He uses the dismissal of Jewish scientists in Nazi Germany between 1933 and 1944 as the shock to human capital. He attributes about a 33% decline in publications, at the department level [to this]. He contrasts that with the destruction of universities during the Allied bombing campaign of World War II. He’s able to use this as an exogenous shock — some departments were affected by these bombings and some weren’t. That’s the physical capital shock. What’s striking about it is that the decline in publications is much larger relative to the decline due to the physical capital. He has this quote: “The dismissal of scientists in Nazi Germany contributed about nine times more to the decline of German science than physical destruction during World War II.”

I’ve done a little work recently with Ukrainian researchers, looking at the impacts of the full-scale invasion of Russia in Ukraine. They’re really worried about brain drain — scientists leaving — and just measuring that. There’s this idea that yes, there’s a lot of destruction and horrible bombings, but some of the scientists say that if we can build it back, it actually can be built back better — better labs and equipment. You can build back labs pretty quickly if you have the resources, but it’s really hard to bring back the people who have left. That’s one of the takeaways from Fabian’s work.

This is a little bit small here, but this is one of the tables.

You can see that he’s got the data on universities. There’s this variation in the intensity of how many Jewish scientists left during this time. Destruction is the bombing. He has these dismissal shocks and bombing shocks, and he looks at them by field, and that’s where he’s able to back out his estimates.

One thing then to think about, when I was talking about how to measure who is an innovator — who belongs in LA — is what it means to be an innovation worker versus a production worker. That’s really the question for today: what are the factors that shape who is part of the supply of innovators?

Everyone listening either has a PhD or is doing a PhD. As you know, it’s a long process to actually get to the frontier. That’s a specific feature of this group of workers: they need to bring themselves to the frontier before they can be productive. This idea of the gestation period — it’s long, and it’s getting longer. Ben talked about his “burden of knowledge” work [in Lecture 1] — it takes a long time to get the knowledge and resources to be able to be at the frontier.

A few other features: job prospects at the time of graduation are difficult to predict in advance, because when you enter the labor market, that’s many years into the future. If we think about models that economists use where people compare expected salaries in different jobs when choosing what to study, it’s actually hard at the time you make your decision to predict that. I’ll talk about this idea of biased beliefs, and the fact that people often don’t have reliable information about job outcomes of recent grads.

Pierre will probably talk more about this, but if you’ve read much Merton, about norms in science and what motivates scientists — it may be that scientists love the subject. As economists, how do we think about those motives for doing science that are not extrinsic? Merton talks about ribbon, gold, and puzzle as things that motivate scientists. There are things that, when you’re studying this group, may be different than other models that economists have.

This is a great segue into who selects into the idea sector. The idea sector is very heterogeneous: what people are doing is going to differ. But here we’re going to think about the fact that you need to have some skill, to specialize, and to reach the frontier. Some of the questions are:

• Do institutions provide the right incentives for the right people to work on innovations?

• What kind of policies and shocks stimulate entry into STEM careers?

• Who is in the pipeline that produces innovation?

• Are there barriers limiting diversity?

I took this from Pierre — this was his image of someone who I think we all recognize.

If we think about Einstein, we all know he worked in the patent office early on. What if he didn’t choose to go into academia? Today, if you look at some undergrads with quantitative skills, maybe they’re going into private equity or hedge funds. What about that? As a society, what would we want? I’m guessing most of us would probably agree we want Einstein to go into academia. But what if he didn’t choose it, or what if he actually wanted to just earn a lot of money? These get into interesting questions.

There are so many headlines, but in this time right now we have a lot of scientists — innovators — who have now left the government. What is going to happen to that pipeline in the US is a topic that’s really important.

With AI, there’s also the question about the computer science labor market and the role of AI.

In the last year or two, things have changed a lot, and so there’s a lot to think about in this topic.

The role of China: I won’t get to talk about international mobility today, but this is a big topic — where is the talent going to go? The US used to be more or less the top destination for international students and scientists, but that’s changing as where the science is being done is changing.

This is a very interesting graph that I’m hoping someone will work on to try to understand the causal factors.

This is the share of students who plan to major in computer science by gender, from 1970 to 2015. What people always talk about is that there’s pretty much parity around the ‘80s — this is from a survey of freshmen. What you can see is that it then spreads out, and even today there’s been a bit of an uptick, but for a while there were very few women compared to men.

I’ve heard some hypotheses that it was because parents bought their sons personal computers, so when they got to college in the ‘80s they were ready to go anyway. It’s interesting, all the different factors that can play a role in who ends up being in that pipeline.

Cobweb models

I want to get into the first model: cobweb models. This is Richard Freeman, who was my adviser.

He’s a labor economist, but one of the topics that he worked on in the ‘70s is looking at the US engineering labor market. This is the post-war period. He was looking at some data — Richard is really great about understanding the data and looking at the big picture trends. Sometimes we’re so interested in the causal questions that we look very narrowly. He had been looking at freshmen in engineering majors and noticed this interesting trend of the share who were enrolling versus the share graduating. What he noticed with this pattern was an interesting feature, for which he developed the cobweb model. It’s a very simple way of thinking about the labor market, but there are some aspects that can be useful when we think about what’s going on, even today.

What he recognized is that salaries four years earlier determine entry decisions. People in this engineering labor market would look at what the salaries of engineers are and then decide, “Engineers earn a lot of money. I’m going to be an engineering major.” But by the time they graduate, the labor market conditions have changed. This is very relevant now. I don’t know how many of you have talked to people in computer science departments, but even here at UMass there have been some big changes in enrollment of undergrads in computer science, because the labor market is not very good if you’re just graduating with a computer science degree.

This was from The Economist not too long ago, where they have job postings — job openings for software developers — and you can see there’s been a decline very recently.

At some point a few years ago it was, “Computer science majors are earning a lot, this is a hot field,” and then things changed.

That’s the basic idea with the cobweb model: that labor markets don’t adjust quickly to shifts in supply and demand. There are two assumptions that are needed:

1. Time is needed to produce skilled workers. You need some time to be an engineer or be someone who’s working on AI — not working with it, but developing the AI models.

2. People decide to become skilled workers by looking at conditions in the labor market at the time that they enter school.

What happens is that a “cobweb” pattern forms around the equilibrium, which arises when people are misinformed. It’s really this idea of misinformation, which is a thread that runs through most of the concepts we’ll talk about today. If we look at most models, we assume that people have information about the labor market — so what happens when you don’t have that?

You might have seen this in an undergraduate labor course.

The basic idea is that you start at some equilibrium employment and wage. Then you have some demand shock. Let’s say for AI specialists there was increased demand. That’s going to shift your demand curve out. In the short run you’re going to have an increase in the wage. With the demand shock, you go up to W1. But in the short run there’s a limited supply of, in this case, engineers — they aren’t produced immediately — so you have this inelastic short-run supply curve. Students are going to see that you have this higher wage, so you’ll have more students enter. But when they enter the market they’re only going to earn this lower W2 wage. This continues for a while, and you get this cobweb pattern.

The big picture is that in the long run, that initial demand shock would lead to a higher equilibrium wage and employment, but it’s a series of booms and busts that get you eventually to that point. As students thinking about what you’re going to major in — there’s a whole literature on biased beliefs; that students often don’t even know what salaries are in different fields — but even if they did, that’s current. How can you predict what it’s going to be in the future? This is an older model, a pretty simple way of thinking about the labor market, but there are some useful features for those of you who are thinking about these topics.

Compensating differentials

I’m going to talk very briefly about compensating differentials, because Pierre will talk about this paper more. In the labor economics literature there’s been a lot of interest in this — there have been new papers in the last few years trying to understand what workers value in a job. You get your wage, but you may actually be willing to take a lower wage if there are some characteristics of the job that you value.

Going back to Adam Smith, it’s the advantages and disadvantages of the job that have to be equated across jobs. The original way of thinking about it is that if you have a firm with unpleasant working conditions — say very manual work, or very hot on the job — then those jobs should pay workers more. Jobs that offer more pleasant conditions could offer workers less, because workers are willing to give up some of their pay for the pleasantness.

There’s a whole literature on how you can estimate that price for these non-wage characteristics. It’s challenging, because if we just go to a data set — say one that shows what workers earn and the features of their job — you would actually tend to find that the more pleasant jobs are paid more, because there’s a whole bunch of things, like education, that are correlated with the kinds of jobs that people are getting.

There have been experimental approaches to try to measure this. I went from Richard to Scott Stern, who in 2004 wrote this paper, “Do scientists pay to do science?” It was an exciting paper because it was one of the first that came up with this quasi-experimental approach to measure this trade-off. He’s measuring this wage-versus-science trade-off. This goes back to what we’re talking about with Merton: that there are things that scientists value. Going back to the discussion earlier about who innovators are — in this case we’re thinking about scientists, who might have very particular preferences for doing science, or for having autonomy in what they’re going to research.

Stern’s method was a survey of postdocs in biology — within one field — and he was able to ask them about their different job offers, the wages, and the amenities. He had this data set that had a very small sample size — 66 individuals and 164 offers. Today you probably couldn’t get away with such a small sample, but at the time it was really exciting, because he could look at one individual, look at the different offers that they had, and estimate what’s called willingness to pay.

Once you put in individual fixed effects — you account for these unobserved individual differences — he estimates a willingness to pay of 20-25% lower wage to be able to do their own research. Why is this important? One of the questions that people are often interested in is who wants to go into industry versus academia? He was able to show that people really value being able to do research — if the firm permits publication, they’re willing to take a lower salary. It gets into this idea of what motivates scientists.

This was one of the regression tables.

He has self-reported salary. There have been recent studies — not with scientists, but in the labor literature — where they were able to randomize job offers and salaries. Obviously the world is different in terms of what would be evidenced in this space, but this was a very important paper in terms of the approach.

Earnings & Entrance into Science: the Roy model

I don’t know how many of you are familiar with the Roy model. If you’ve taken a labor class, then you will have been exposed. Thinking back to industry versus academia, or finance versus biology, the question we’re thinking about is how workers sort themselves across different sectors. You could also think about the idea sector versus management, or whatever you’re going to choose to work in.

It’s always interesting to go back to the paper. It was all words, no equations. This is the first page of Roy.

The original Roy was actually about workers choosing between hunting or fishing. You might have seen the Roy model used a lot in the immigration literature, thinking about who chooses to immigrate to different places, or not. Borjas 1986 is the reference that people often use. He essentially took Roy and wrote down the equations, so that you could have the mathematical setup to look at selection.

Here’s a very simple version.

The idea is that you’re going to have workers who are making decisions about which sector to work in. It’s a pretty simplistic way of thinking about it, and obviously you can get to more complex models, but essentially workers are comparing earnings in each sector. Their objective is to maximize earnings. There’s positive selection, where the very skilled — the people in the upper part in the distribution of skills on the x-axis — if they’re going to choose a given sector, then they’re going to be positively selected. If it tends to be people from the lower part of the skill distribution, then that’s called negative selection. One of the Roy model’s predictions is whether there’s going to be positive or negative selection, based on the distribution of wages in a country — how compressed they are.

For our world, when we think about innovators, the other question becomes whether there’s correlation between the skills that individuals have in the two sectors that people are thinking about. You want to think about an individual that we’re observing choosing between the idea sector and finance. The question is: are the skills needed in finance and the skills needed in the idea sector correlated?

I wanted to point you to this paper by Shu. We were talking about different sources of data. Patents are bread and butter, but I urge you to think about sources of data that you might have access to that people haven’t used before. Shu got great data on different cohorts of MIT students, with all kinds of things like the courses they took, their GPA, even in high school, and was able to link that to patents as well. One of the papers was looking at this question of the Roy model and the decision about working in finance versus in STEM. This issue of whether the best and the brightest are actually going on to finance, and whether that’s what we want as a society, is an interesting question.

What she finds — using data on what students are doing in college and in high school — is that finance doesn’t attract the best and the brightest STEM students, but it does attract those with more finance-relevant skills. She essentially shows that finance and science and engineering demand different skills, and still finds that the best STEM students have a preference for going into science and engineering.

She finds what is called negative correlation of skills: the skills required are very different in students going into finance versus science. Those going into STEM have better academic records in high school and focus more on developing academic skills during college. This is where it’s interesting if you know the literature by David Deming on the importance of social skills — people right now are also talking about how, with AI, that people who have the social skills might be the ones who do better in the labor market. She found evidence that people entering finance had more leadership experiences in high school and focused more on developing social skills during college. The idea is people selecting into each sector have different skills.

She also has some work looking at the financial crisis in 2008 and shows that that’s a shock to the returns to doing finance.

After the crisis, those who stayed in finance actually have better academic qualifications. She can look at who’s the marginal student who’s going to stay in finance versus not.

That’s just to give you a flavor of the Roy model. There have been some recent papers that also use the Roy setup. You can think about a setting where you have a change in the returns to a degree, or in the salaries, and you can look at changes in who selects. She has this admission index, which is her measure of skill — you need to observe something about the skills or the ability of the individuals, to understand who are the more qualified, choosing, let’s say, the idea sector versus not.

Earnings & Entrance into Science: Biased Beliefs

Here we’re going into the role of information. This has also gotten to be a very popular topic in general labor economics. For our world of the economics of science and innovation, there is a focus on PhD students, because in some fields, to be an innovator, you need to get a PhD, and it’s a long process. There is a lot of focus on this group: people who choose to do a PhD and then go on to do a postdoc.

Several years ago there started to be this discussion that the job market for PhD students was really bad. There were a lot of discussions around whether we have enough jobs, and whether students know. One thing we know in economics is that it’s pretty transparent — you can go and see from different departments what the placements are of PhD students. In the sciences, most people do a postdoc, and so it’s actually hard for students to see what the careers are like of people in their field.

PhD and postdoc training — if we go especially to the STEM fields — is designed to prepare individuals for academic careers. It used to be more so. Now we’ve seen that there’s much more awareness, by faculty PIs, that students are going into industry or entrepreneurship. But in STEM fields, about 10% of PhD graduates go on to get a faculty position. In 2016, there were 2,700 new chemistry PhDs, but only 152 openings in research universities.

You probably all know the PhD Comics, about how ambition changes as you go along the PhD.

At the beginning, you think, “I’m going to win the Nobel Prize,” and then at some point you start to get more information.

How do you get this information? Some of it is through your peers, but should there be more structured effort to have data for PhDs? I’m going to tell you about a study that I did with some co-authors where we tried to look at this. There was this idea that students don’t know what the job prospects are when they’re finishing. The labor literature had grown a lot on this topic — that students are not fully informed when making educational choices. Experimentally, there were several influential studies that showed that if you give accurate information, students update their beliefs and their decisions. There was a paper by Rob Jensen giving students information on the returns. There was work by Wiswall and Zafar showing that NYU students, if you give them information on salaries in different careers, change their majors. It’s been pretty well shown, across a range of settings, that students and parents don’t have accurate information.

I wanted to talk to you about this paper because one of the things in this field of the economics of science and innovation is that there are still not a lot of papers where they’re doing experiments. I know that there are some efforts with J-PAL and the Innovation Growth Lab to try to encourage and provide resources to do experiments. If you’re thinking about doing experiments, that’s exciting. I’m happy to talk more; I’ve been involved in a few of them.

This one is still challenging to do, but it’s an easier way to do an experiment, which is embedding an experiment within a survey. We wanted to see whether our PhD students are informed about the labor market, and whether providing information can change their beliefs. We did a several-year project where we did a survey of about 1,300 PhD students in chemistry in 2017. If I had more time, I could tell you about just getting the email addresses of these PhD students. We did 54 US chemistry departments. There’s no list out there of who all the PhD students are in a field, so we had to scrape those from websites.

Then we did a randomized information intervention. The focus was providing a subset of students — since it was an experiment — actual historical academic placement records by program. We showed them a website that had, by department, what share of graduates got a tenure-track faculty job at a research university. As you can imagine, it was quite low: in most places it was 5-10%. Then, in another treatment, we also linked to scientist profiles on the American Chemical Society web page.

We did a follow-up survey one year later, and we tried to look at placement data four years later. The whole thing took about five or six years to get the paper finished. In our field there’s so much of a focus on causal estimates. This is a paper, and I have another one like this, where there often ends up being a lot of interest in the descriptive results too — just showing what the beliefs are.

There’s some related work by Henry Sauermann and Michael Roach, where they also did a big survey of many PhD students and tracked them over time in different fields. They didn’t do an experiment, but they also have some of the descriptive pieces that are very interesting. The takeaway, if you just look at the beliefs, is that people are — we called it “at times excessively optimistic.” They were probably very good. But in general, people did not seem to be informed about their actual chances. There’s some interesting variation: international students, and students earlier in the program were especially overly optimistic.

This was one of the descriptive pieces — this is the share of PhDs from their own program getting US tenure-track positions.

The mean was 24.5%, the median was 20%, but about 5% was the actual number. Most of the sample was reporting above that.

We did this information intervention. One of the issues with this project was that the results were a bit puzzling: we found that they did update their beliefs about their own chances of becoming faculty, but not about the market overall. It was a puzzle of why they would update. We also didn’t find any gender differences, which people are always interested in. Then we waited and collected — back in the days before what everyone’s using now, getting LinkedIn data — we looked up each person and tried to see whether they actually did a postdoc after graduation. We found no change across the treatment and control in doing a postdoc. Once people do the PhD, they’re down that path. Maybe providing this information earlier — but the issue is that so few people of the general undergraduate population would want to go on to do a PhD, so you’d have to think about what the right sample would be. That could be an interesting thing to do.

Here’s some of the data on initial and final beliefs.

You can see that there’s a bit of a downward adjustment, and in particular the people who were the most misinformed — who had the highest initial beliefs about their own chances.

The takeaways are that there might be other reasons that people do a postdoc, thinking about international students, the fact that a postdoc can still be valuable if you’re going to choose an industry career, family reasons, and maybe their preferences are quite stable, because you’re down this path.

Earnings & Entrance into Science: Exposure to Science

This is a very recent topic I’ve gotten into, which is this idea of exposure to science. Going back to who chooses to go into this innovative workforce — you could think about entrepreneurs, or invention. What I’ve been working on is this idea of who decides to go on to do a PhD. We know that one of the biggest predictors of who goes on to do a PhD is if your parents have one. What is the reason behind it? Is it because you have more information, a role model?

What we wanted to get into is this topic of being exposed to what science is. There are a lot of stories. You probably know Jennifer Doudna, who won the Nobel Prize for her work with CRISPR, and was also involved in the last patent interference, the dispute between Doudna and Zhang on who gets the patent rights to CRISPR. If you look into her background, she talks about how she went to a smaller college, and she said, “My biochemistry professor in college gave me a chance to work in her lab over the summer, which is critical, where I really figured out, ‘Wow! I love lab work, this is really great, this is exactly what I want to be doing.’”

What if Jennifer Doudna had not gone and worked in this lab? Would she have gone into science, and would we have had CRISPR? That’s a counterfactual we don’t know, but in this paper we’re trying to get close to that.

There was a big scandal on econ Twitter when there was a discussion about the $15-an-hour minimum wage, and Jennifer Doleac said, “I’m required to pay undergrad RAs $15 an hour. I will definitely hire fewer of them. The silver lining: less paperwork for me.”

People were shocked that she would say this: “You’re not going to pay your undergrads $15 an hour.” It was interesting because these are economists. If you think about a price shock here — that now you have to pay your undergrads this much — if you have fixed budgets, you could imagine that this could be an issue, that people wouldn’t hire their RAs. She had this other tweet: “I provided research opportunities at my lab for undergrads as a way to expose them to the research process, to facilitate the pipeline into academia. There are other, more efficient, cheaper ways I could get the work done.”

That’s what we were looking at in this project: how much do those early lab experiences matter, and can something like the minimum wage also impact it? There are other policies that can play a role in this idea of who’s in the pipeline. Today it’s interesting because we think about NIH funding being cut, NSF funding — PIs are working with smaller budgets, and who they’re going to be able to train can then impact who’s in that pipeline going forward.

We found that writing a labor economics paper, and then a paper that’s interesting for the economics of science and innovation community — we could have the same results, but how we frame it mattered for who was interested. The labor economists are interested in this question, but there’s a whole minimum wage literature — trying to navigate that has been an interesting process.

We are using variation in minimum wage increases across states and time to look at whether that impacts the employment of undergrads in labs. Then we use that as an instrument for exposure of undergrads to science — their lab experience. We use that to link to career decisions of undergrads.

I also wanted to mention this paper because it uses UMETRICS data. It’s an exciting data set, but it hasn’t been used as much as it could be in our field, because you have to get approval to work on it. It provides very micro-level data — grant-level expenditures at major US universities. Bitsy Perlman has some papers using UMETRICS. They link this to other things like publications, and we also now have longer-run education data. We can observe students who worked in a lab as an undergrad, and then see what they do after they graduate from college. The trade-off with UMETRICS is that it doesn’t have names, so we can’t match it to anything else that you might want to match it to.

Our empirical approach uses the recent minimum wage literature — staggered difference-in-differences, using the timing of when you have minimum wage changes, variation in that timing but also in the magnitude, and then an IV for exposure to lab work.

What we find is that when PIs are exposed to increases in the minimum wage, they do reduce undergrad lab employment. They also increase their graduate student employment a bit, so it looks like there is some substitution to graduate students. We also see that they seem to be budget constrained. That’s a feature of why it’s interesting to look at minimum wages in a university lab setting: people are constrained. You have your grants, and they’re fixed. We do find that labs are more likely to seek supplemental funding when you have a minimum wage increase. But unlike with firms, you can’t pass the increased cost through to consumers — unless you think of the consumer as the NSF or NIH.

Going back to framing for our community, the economics of science and innovation: we show that reduced lab experience does affect careers like your Jennifer Doudnas. Students who are exposed to those minimum wage increases work 8.2% fewer quarters in labs. Then we show that those students are less likely to pursue a PhD or work in the life sciences — even just working in that sector. We see smaller effects for students with federal work-study support, which is a subsidy to pay the wage of the student.

Here you can see one visualization, the event study, where the undergrad is the orange line. The red line is the timing of the minimum wage.

You see a decline in employment. The green is the graduate students; you see a bit of an increase. It’s a bit noisy.

This is not super compelling, but it’s just to show you the relationship — blue is students who didn’t experience any change in the minimum wage by graduation year, then we have the quarters that they worked in the lab.

We do this in a regression framework too, but it’s suggestive that this is our first stage: if there was no change in the minimum wage when you were in undergrad, then you were more likely to work in a lab. Then this is with doing a PhD. This is just supposed to be suggestive, and then we do it in a regression.

Minimum wage increase, you work fewer quarters. Then [in the right-hand column] this is the IV, that you are more likely to go on to do a PhD.

This is Richard Freeman. Richard had this piece that he called “five rules for empirical work in labor economics.” He wrote this a while ago, but I think it’s pretty good, especially for those of you who don’t have a dissertation topic yet — or maybe you do, but you’re not sure which way to take it. Some of these rules are nice to reflect on if you need some ideas. A lot of them, when he was writing this — which was probably 30 years ago — weren’t as obvious, but it’s interesting how much is still relevant today.

1. “Create your own variation through an experiment (field or laboratory) or study markets when they experience sufficiently sharp exogenous shocks to create ‘natural experiments’ or learn the institutional details of markets to find plausible sources of variation (policy shifts and institutional quirks).” This goes back to this focus on causality. But in our field, if you can try to run an experiment, it’s challenging — but if you also have a chance to go and work for a government agency and learn about some of the details, you can come across some interesting data. These are also ways to find some variation.

2. “Focus on fundamental first-order economic and behavioral principles (supply and demand; incentives; altruism and reciprocity).” Economics of science and innovation is not one of the typical fields. To get into a general-interest journal — what would a macroeconomist care about if you’re doing a very micro study on the scientists? That’s something that I often get in my referee reports or editors’ letters: “Why should I care about PhD students in chemistry?” Going back to what the big economic principles are is important.

3. This is now standard, “Probe the robustness of empirical findings with different data sets, different specifications, and across time and space — tension vs. pre-analysis plans, pre-specification and concerns with data mining.” The more ways you can convince readers that you have strong results, the better.

4. This is the one that Richard always pushed, and in the profession people have done this: “Don’t be satisfied with just standard and easily available data sets – be willing to do your own survey research, use the resources of the web to collect data (eBay; on-line newspaper archives; on-line school or arrest record data); or to work with relevant organizations to collect/gain access to administrative data (IRS tax records, matched employer-employee data, Scandinavian matched registry data, Social Security admin data, personnel data, …).” A lot of our data is patent data, so if you can get at some of these questions with new data or survey research that you’re doing, I think it makes it more exciting.

5. People do this, but it’s still not the case that we talk about it often in our papers: the importance of qualitative research. “Discuss issues and analyses with the participants in the markets under study – ‘In a field lacking decisive tests of hypotheses, it is worth listening to what eyewitnesses and participants have to say’ (”quarks can’t speak” but humans can tell you what they think is going on!).” In a field lacking decisive tests of hypotheses, it’s worth listening to what eyewitnesses and participants have to say. If you read the original piece that Richard wrote, he talks about how we’re different from physicists, because we can talk to people. He talks about how Richard Feynman can’t go talk to what he’s studying — quarks can’t speak, but humans can tell you what they think is going on.
   

I have some friends who are scientists, and they always know when I’m testing out a new idea, like with that minimum wage study. I was like, “When the minimum wage increases, what does that do to your undergrads in your labs?” Just as a check, because when we were first presenting that, I remember people were like, “Why are PIs responding to the minimum wage?” I had talked to several people who were like, “We have fixed budgets; if the cost rises, you’ve got to let someone go.” The more that you can do that — if you have some friends who are scientists, that’s always helpful.

This piece has a simple suggestion: Rather than letting those ideas sit in digital file drawers, we offer them to philanthropists who might fund them. Perhaps instead of spending another year modifying a grant and throwing it back into an application portal, a scientist could be funded immediately by a foundation that finds their idea exciting.

I think this idea, sharing unfunded grant applications with other potential funders, has a real chance of being implemented. It’s easier than ever to do technologically and I haven’t heard a compelling reason why it shouldn’t happen.2

About the authors: Sandeep Patel and Amit Gupta are co-founders of Catalyze. Stuart Buck is Executive Director of the Good Science Project (where we are cross-posting this piece). And Ian Banks is Director of Science Policy at the Foundation for American Innovation.

Every year, federal agencies assess over a hundred thousand proposals that they never fund. Treating those peer review as public infrastructure could help unlock much more philanthropic capital for American research.

The National Institutes of Health (NIH) and the National Science Foundation (NSF) together run the most prolific scientific evaluation operations on the planet. Every year, they collectively review 110,000+ proposals, spanning everything from fundamental biology to quantum computing. While there is not wide agreement regarding how reviews should be conducted or how funding decisions should be made, no other process exists that applies rigorous, standardized review on such a comprehensive set of proposals.

In 2025, NIH reviewed 62,592 research grant proposals and only funded 8,161 (13%). That leaves thousands of proposals that were deemed to be in the top quartile, but are going unfunded due to a lack of money rather than a lack of merit. NSF exhibits the same pattern across a far broader set of fields (NIH only funds biomedicine, whereas NSF funds many other areas of basic science, including materials science, engineering, computer science, basic biology, and the social sciences).

The cost to produce reviews at this scale is substantial. Taxpayers support infrastructure at federal agencies, which totals roughly $263 million at NIH alone. Thousands of expert reviewers each volunteer on average 4-8 hours per application. And applicants spend an average of 116 hours writing each application. The time from when a scientist writes a proposal to when they receive grant funding can be up to two years.

Subscribe now

Whether a proposal gets funded is largely driven by peer review scores, but the detailed record (such as the actual scores and reviewer commentary) isn’t transparent at all. Peer review data disappears into an agency database that is almost never made available to anyone else. We think that these data should be far more transparent, so that they can help inform science funding organizations and the public.

Our proposal: make peer review scores and comments on unfunded proposals available to private funders through a shared platform. The scientist should be able to opt in and retain some control over what to share, but then the advantages are two-fold: (1) extra opportunities for the scientists to get funded (few researchers would object to that!), and (2) funders can more easily find vetted work that is matched to their interests.

What is needed, in effect, is a way for the public to reap the benefits of the millions of hours of due diligence that they have already paid for.

The moment is ripe. American philanthropy is arguably about to enter its third great wave, beginning now with Silicon Valley wealth (much from AI) and intergenerational wealth transfer, driven by donors who intend to give while they are living, who think in terms of measurable outcomes, and who are likely to be more comfortable with risk and with novel mechanisms.

Even when put against the NIH’s roughly $48 billion budget, private philanthropy is already quite significant: The Gates Foundation gave away over $8 billion in 2024, while the OpenAI Foundation is capitalized at more than $220 billion and has announced an initial $25 billion commitment to curing diseases and bioresilience.

Subscribe now

Many new donors want to fund science, including relatively new fields like biosecurity. Our federal scientific institutions (even with budgets growing at a typical pace, let alone with budgets under attack as they currently are) cannot possibly fund all of the meritorious proposals they receive, leaving great ideas unpursued. The third wave is arriving with its checkbook open and limited immediately-available options to write the check. Federal peer review could easily be reused as a signal of what private philanthropists should fund.

A skeptic might reasonably ask whether scientists themselves want their unfunded proposals to be circulated more broadly. Over the past several months we have spoken with more than 250 researchers, institutions, and companies across the United States and around the world, including interviewing dozens of principal investigators (PIs) directly about this question. We have heard near unanimous willingness, if not eagerness, to participate in such a system if it existed, assuming that scientists still maintained some level of visibility and control over what was shared and with whom.

Several small-scale examples of a marketplace for federal peer review already exist. The NIH has attempted a narrow pilot of a similar program that received researcher support. NIH’s Online Partnership to Accelerate Research effort gave highly rated but unfunded applicants a second channel to private funders, but ultimately it was too administratively complicated and its funder base too narrow for widespread adoption. The NSF’s Collaborative Research in Computational Neuroscience (CRCNS) program, still in operation today, offers other funders access to NSF reviews in computational neuroscience, including other government funders and philanthropy. As part of a very specific, highly interdisciplinary field, this program has backed research behind several modern AI advances, including brain-computer interfaces and spiking neural networks.

Another objection rests on whether the federal government can lawfully share this information. In the lightest-touch version, no disclosure is required. Agencies can email each applicant a private code, and the applicant then shares their own materials with a new potential funder. Neither the Privacy Act nor the Trade Secrets Act (18 U.S.C. § 1905) are implicated here. Indeed, the federal government already draws this line elsewhere: NIH and NSF publish the title, PI, abstract, and amount of every funded award.

We believe consent should be explicitly investigator-controlled. A PI would be able to opt in, and control access to abstract-level summaries, full proposals, and any panel reviews. A reporting layer would make a project’s existing support (from the government or philanthropists) visible, enabling a project to receive multiple gifts and for philanthropists to see where funding is most needed. None of this would require the government to spend significant capital or to endorse a particular funder. This proposal only requires federal agencies to make available evidence they already produce.

We can further reduce friction via two more expansive options. One option is to add an opt-in box to the NIH or NSF application, authorizing the agency to transmit the full proposal and reviews directly to the platform on the researcher’s behalf. Because that disclosure rests on the applicant’s written consent, it falls within the Privacy Act’s consent exception (§ 552a(b)) and would require no new system-of-records change. The only procedural step needed would be to modestly revise the agency’s approved information collection under the Paperwork Reduction Act.3

A second option, adopted by the NSF CRCNS program, is to disclose in the Request for Proposals itself that proposals and their reviews will be shared with outside funders.

Subscribe now

Two additional concerns should be addressed. The first is substitution: if philanthropy funds what the agencies are meant to support, does that hand Congress a rationale to cut public research budgets further? The best answer is that philanthropy and government largely have different views on prioritization, meaning there is natural complementarity to their co-existence. Rather than replacing public funding, philanthropy will most likely remain additive capital, reaching work that government agencies failed to fund.

The second concern is that simple metrics like percentile scores are noisy and are not reliable marks of quality, especially for a diverse set of funders with diverse motivations. This is correct, which is exactly why the summary of the agency assessment, along with detailed reviewer commentary and disagreement, should be included alongside any top-level percentile scores. In fact, we believe that sharing peer review information can uniquely enable more diverse prioritization of research by allowing funders to see with greater clarity how they fit into the larger funding ecosystem.

Our primary goal is to attract more capital into research. Through our proposed platform, philanthropy could complement federal funding and broaden the range of what gets funded. Where funders might normally duplicate each other’s evaluations in private, they could instead build on existing peer reviews and coordinate with each other through a shared platform. Besides bringing in more funding, treating peer review as public infrastructure would make the ecosystem more legible to everyone involved.

Shared infrastructure could enable the work of federal peer review to compress the timelines of other funders. Today, a proposal can wait the better part of a year for a funding decision, and many of the best and most ambitious ideas remain unfunded. In general, we should be building toward a world in which all the best ideas are funded, and the time from generating a very good idea to getting it funded goes to zero. This proposal could bring us one step closer to that distant goal.

This would not require the federal government to lower its standards in peer review, nor change what it chooses to fund. We should simply stop discarding the output of the largest and broadest scientific evaluation system ever built.

Subscribe now

In this lecture, Chad discusses how innovation and technology has shaped economic growth in the past, and how technology — in particular artificial intelligence — may influence growth in the future. I found the section toward the end of the lecture on the potential growth effects of AI and automation fascinating; one of the big insights from Chad’s modeling efforts is that the growth effects may lag considerably. So you might get profound abundance, but not until 50-100 years in the future. See below for a hint of what you’ll get in the lecture.

Subscribe now

Chad will have a chance to dig deeper into the economics of AI while influencing the technology’s future. In June, he announced that he was taking a leave of absence from Stanford University to conduct research on the economics of AI at Anthropic.

Here are the readings that go with this lecture:

• Bloom, Nicholas, Charles I Jones, John Van Reenen, and Michael Webb. “Are Ideas Getting Harder to Find?” American Economic Review 110, no. 4 (2020): 1104-1144.

• Jones, Charles I. “Growth and Ideas.” Handbook of Economic Growth 1B (2005): 1063-1111.

• Jones, Charles I. and Christopher Tonetti. “Past Automation and Future A.I.: How Weak Links Tame the Growth Explosion.” Stanford unpublished manuscript (2025).

• Jones, Charles I. “The Past and Future of Economic Growth: A Semi-Endogenous Perspective.” Annual Review of Economics 14, (2022): 125-152.

• Romer, Paul M. “Endogenous Technological Change.” Journal of Political Economy 98, no. 5 (1990): S71-S102.

Subscribe now

Thanks to William Higbie and Harry Fletcher-Wood for their support in producing this video and the transcript.

Subscribe now

Lecture Transcript:

The Past and Future of Economic Growth: A Semi-Endogenous Perspective

I’m going to present two sets of slides. First, an overview of how I see economic growth with some applications. Second, on AI and our economic future — a review of the recent work I’ve been doing.

The first thing I want to do is give you my view of how the economic growth process works. Here’s my favorite graph in economics.

This is a chart of GDP per person — average living standards in the US for the last 150 years. Once you plot it on a log scale, it’s a straight line with a slope of 2% per year. One of the big questions of the economic growth literature is, “How is sustained exponential growth at a stable rate for 150 years possible?” There are a bunch of growth models — a bunch of answers. But I want to give you my preferred answer.

I love simple models. Once you claim you understand something, it ought to be possible to strip out a bunch of ingredients that are unnecessary to show the core insight as simply as possible. You definitely don’t start there, there’s an arc — but it’s great to end up there.

• I want to do that.

• I want to talk about some growth accounting with this model.

• Then, I might skip through the future of growth: why it might speed up or slow down.

• That will take us naturally into the artificial intelligence material.

What’s the simplest possible model you can write down that gets at the core insight of why sustained exponential growth is possible? The key insight here is Paul Romer’s Nobel Prize for what I like to call “the infinite usability” of ideas. Romer pointed out that ideas are different from almost every other good in the economy. Most goods are objects. iPhones, airplane seats, surgeons, me, you, my computer, my office. Objects are rival. If I’m using it, you can’t use it at the same time. That rivalry is what gives rise to the fundamental scarcity at the heart of economics — economics is the study of how we allocate scarce resources.

Ideas are different. They are nonrival. I don’t like defining it as the negative of something else, so I prefer “infinitely usable.” Once you invent an idea, it’s technologically possible for it to be used by any number of people simultaneously without any depletion. You don’t run out of a given idea. Think about calculus, or HTML — the code that underlies web pages — or the chemical formula for a new drug. We invent the COVID vaccine and eight billion people can benefit simultaneously without any depletion. Contrast that with a barrel of oil or an apple: you or I can eat it. But we can all use calculus, computer code, or the latest ChatGPT. The code is nonrival.

The essence of Romer’s insight was, when you write down the production function — suppose A is the stock of ideas: the total number of ideas ever discovered. If we think of ideas as recipes in a cookbook, A is the number of recipes. Why, when you include ideas in a model, do you want the A outside the constant returns rather than inside?

This is a profound question. Your bias should be that everything goes inside the constant returns to scale. If we’re adding land, or splitting labor into skilled and unskilled, or splitting capital into equipment and structures, all that goes into the constant returns to scale. The default is that ideas should go in the constant returns to scale just like everything else, and you have to have a good reason for it not to. Griliches, when he was putting ideas in a model, put knowledge capital inside the constant returns to scale, exactly as he should have done if he didn’t have a good reason not to. Romer taught us the good reason not to, and why you want to pull the ideas outside the constant returns.

Let me talk about the constant returns to scale. Suppose we’re looking at Steve Jobs and Steve Wozniak in the garage in the ‘70s making the first personal computer. It’s the two of them, some screwdrivers, computer chips, and a keyboard. The reason you want to write constant returns to scale is because of the replication argument. If Jobs and Wozniak can make seven computers a week in their garage, one way to double production is to replicate it: create an identical garage, get two people identical to Jobs and Wozniak, and get the same computer parts. If we’ve replicated everything, we should produce the same amount. We’ve doubled our inputs and therefore doubled our output. That’s constant returns to scale. The replication argument is why I say the presumption should be constant returns to scale.

Yet once you think about ideas, that breaks. Why? Back to the Jobs and Wozniak example: there’s constant returns to scale after they’ve discovered the idea. The blueprints for building a computer are written down on a sheet of paper. The piece of paper is an object that needs to be replicated, but you don’t have to reinvent the design. You photocopy the design — you double the number of pieces of paper. But you don’t have to pay the years of work that went into getting the blueprint for the first computer. You have constant returns to scale to objects — that’s the replication argument — and therefore increasing returns to objects and ideas.

If you read the Romer 1990 paper, this core insight is in the first five pages, and he expresses it in some math just like this.

If A is the stock of ideas, you’ve got constant returns to K and L — they’re objects. Therefore, if the marginal product of knowledge is positive — if adding a new idea increases output — then there must be increasing returns to objects and ideas together. It is that simple. The replication argument and nonrivalry, the infinite usability of ideas, gives you constant returns to scale to objects and therefore increasing returns to objects and ideas. That’s the Nobel Prize. Everything else after that is dotting the i’s and crossing the t’s. That’s the essential insight in the way I view growth.

Now I want to write down a simple model to illustrate this core insight and how it delivers a theory of growth.

Notice I’ve dropped capital. Why? Solow taught us that capital is a bit of a sideshow — you don’t need it — so I’m going to take it out. In the first equation, there’s constant returns to labor — the only object — and A is the stock of ideas. With σ, there’s increasing returns to labor and ideas. The parameter σ measures the degree of increasing returns in the production of goods.

Then we also need a production function for ideas. Where does knowledge come from? This black box of where ideas come from is a big mystery. It turns out from the growth perspective you don’t have to solve that mystery. You can write down a standard production function for ideas and that works fine. There’s a fair amount of flexibility for how you write it down.

In the second equation, the number of new ideas — Ȧ, the flow of new ideas — depends on the number of people searching for ideas, R, and potentially on the stock of previous knowledge that’s been discovered, A.

Now I’ve got this φ exponent. What we’re saying is past knowledge could help or hurt you in producing ideas. There’s some flexibility there.

φ = 0 is the simplest possible model: one researcher discovers one new idea, and the model works great.

φ > 0 is standing on the shoulders of giants: past discoveries like calculus and information technology make me more productive at producing ideas.

But all graduate students are convinced that φ is negative. Why? φ < 0 is, “All the low-hanging fruit has been picked.” It says the more discoveries in the past, the harder it is to find new ideas today. It would be great to be an economist in the ‘50s — Solow, Samuelson, and Arrow had it easy. There was all this low-hanging fruit. Nobel prizes were hanging there. You just had to pick the apple. Graduate students today are climbing up the tree looking for the last remaining fruit, out on the branch precariously balanced, just to get the smallest of apples. It feels like ideas are getting harder to find in a strong way and that φ is negative.

It turns out the model works fine whether φ is positive, 0, or negative, which is a nice feature.

In recent work I found it helpful to rewrite this equation in terms of the growth rate of knowledge and define β to be 1 − φ. β captures the rate at which ideas are getting harder to find. Sometimes I like all my parameters to be positive. Both ways — the φ way and the β way — have lots of merits, but I go back and forth.

The next equation is the resource constraint. We’ve got L people in the economy. There’s population growth — I’ll come back to that. The main decision in this economy is how you take your number of people and divide it into production workers versus researchers. That’s the allocative decision. We could:

• Let markets decide.

• Write a utility function and let the social planner figure out the optimal allocation.

• Do a rule-of-thumb allocation — the one I’m going to follow here, and the easiest.

I’d encourage you, whenever you write down a model, just do rule-of-thumb allocations to start; you can do the optimal allocation or let markets allocate later. The thing that makes the model easiest to understand is some simple rule of thumb. Suppose we put a constant fraction s̄ — say s̄ is 10% — into research and 90% into producing goods. Then if you count equations and unknowns, the model’s closed and we can solve it. We can ask in particular: does this model generate sustained exponential growth at a constant rate, at least eventually?

[Student question.] Should I think about this rule of thumb as meaning skilled workers are unable to become unskilled workers?

In the simple model, there’s no such thing as skilled versus unskilled — there’s just a homogeneous labor. You could add skilled versus unskilled, and that would complicate the model, but the core result would still be the same. Like I’m forgetting about capital, I’m forgetting about human capital as well. In richer models — I’ll show you one at the end — I incorporate human capital. But for the simplest model you don’t need it, so I’ve left it out.

Remember when I said income per person grows at 2% per year — that’s about little y rather than big Y. In many growth models, we use lowercase letters to denote per capita variables. Little y is income or output per person. If you use my allocation, output per person depends on the stock of knowledge raised to some power times 1 − s̄.

[Student question.] I had a quick question on how to think about implementation costs within innovation. It seems a lot of implementation of new ideas are rival, and I’m struggling to think about something like calculus, which can be instantly transmissible to anyone who wants to use it, versus something like a new semiconductor innovation, which less developed countries wouldn’t be able to necessarily implement.

Even with calculus, it’s not like you show me a calculus book and I suddenly know how to do it. There’s a bunch of human capital — education — that has to be developed. It’s absolutely true that using ideas can require scarce resources. Imagine we’re making the COVID vaccine. It requires an incredibly sophisticated biotech factory and chemists to make it. Just because it’s technologically possible for many people to use the design simultaneously, doesn’t mean it’s not constrained by the scarce factors that are complementary to implementing it. In the simple model, those considerations are left out, but in the real world they’re definitely important.

[Student question.] Could you think about them being included in either β or σ here — just for conceptualization within the simple model?

I don’t think so. I think they’re different. Implementation is about rival stuff, and the φ in the production of ideas feels different to me. You’d have to complicate the model rather than use this one. But the first equation — income per person depends on the stock of ideas — Romer’s Nobel Prize is already there.

If we want the growth rate: on a balanced growth path, Ȧ/A is going to be constant. Whatever that constant is — along a balanced growth path, you can solve the idea production equation Ȧ = R·A^(−β) to get that A* (where the star denotes the balanced growth path) is some constant × R^(1/β). Where do ideas come from? From people. In the long run, the stock of knowledge is proportional to the number of people raised to some power, where the power is the inverse of the rate at which ideas are getting harder to find. A big β says ideas are getting hard to find quickly; a small β says they’re not getting as hard to find.

Now we put these two equations together, substitute the second into the first, and you get two useful results.

The first is that income per person is some constant × R^γ, where γ is my notation for σ/β. γ is a key measure of the degree of increasing returns to scale.

Notice what this equation says: income per person depends on the total number of people. Why? Income per person depends on the stock of ideas. The stock of ideas depends on the number of people. And we’re off and running.

Let me jump ahead quickly.

The equation I just derived says income per person depends on A to some power. The Solow model would say income per person depends on k to some power. Suppose I make both powers ⅓: α is ⅓ and σ is ⅓. The Solow model fails to deliver exponential growth, but the Romer model succeeds. Why? It has to do with the key difference between these two letters. A is the aggregate stock of ideas, and k would be capital per worker or per person. This is the nonrivalry of knowledge: each person benefits from every idea we create, whereas with machines you need a machine for every person.

Output per person depends on the number of ideas; the number of ideas depends on the number of people; so output per person depends on the number of people. If you take growth rates of this equation — logs and derivatives — the growth rate of output per person is the growth rate of the number of researchers × γ. So γn, where in the long run the number of researchers grows with population growth.

What this last equation says is the growth rate of income per person in the long run is a constant. This constant could be 2%: maybe n is 1% and γ is 2. I’ve given you a model that could give us constant exponential growth for 150 years. It has some interesting features — namely, population growth is important. If the population growth rate were 0, the exponential growth rate would be 0. We wouldn’t grow.

Why is population key? There are two ways to see it. The first is a narrow mathematical way. Romer’s key insight is that ideas are infinitely usable, and that gives rise to increasing returns to scale. What does increasing returns to scale mean? Bigger is more productive. Population growth makes us bigger, and the degree of increasing returns to scale turns the bigness of the population into per capita income. The growth rate is the product of the degree of increasing returns to scale, γ, × the rate at which scale is growing. The degree of increasing returns to scale is the ratio of how important ideas are in producing goods, σ, to the rate at which ideas are getting harder to find, β. γ is an overall measure of increasing returns to scale, including goods and ideas.

The other interpretation is that people produce ideas. We need the stock of ideas to grow. Getting aggregates to grow is easy; Solow taught us how. People produce ideas, so the more people, the more ideas. People produce computers, so the more people, the more computers. But it’s computers per worker that matters for productivity, whereas with ideas it’s the total stock. The COVID vaccine, calculus, or faster computer chips — you invent it once, eight billion people benefit.

You add one new idea and you can make an unlimited number of people more productive or better off, because the ideas are infinitely usable. Income per person depends on the total stock of knowledge, not ideas per person. It’s easy to make aggregates grow — population growth makes aggregates grow. That’s the essence of how Romer’s insight of nonrivalry leads to growth in a way that other models don’t.

Let me turn to this paper with Nick Bloom, John Van Reenen, and Mike Webb: Are ideas getting harder to find? This paper provides evidence for the view of growth I laid out. It documents a new stylized fact: there’s a robust sense in which ideas are getting harder to find.

There’s a word equation that many growth models have that says the growth rate equals the number of people looking for ideas times their research productivity times the rate at which they turn ideas into growth.

In this paper, we say, “Economic growth and the number of researchers are pretty easy to measure. So we can do growth accounting with the idea production function.” We measure growth and the number of researchers, and we can infer research productivity — the total factor productivity (TFP) of the idea production function — as a residual.

When we do that, everywhere we look — agriculture, Moore’s law, health innovations — economic growth is pretty stable or maybe even declining slightly. But the number of researchers is rising incredibly quickly. This says that research productivity — the rate at which we turn people into economic growth — is declining very robustly. That’s a sense in which ideas are getting harder to find. The way I wrote down the growth model is exactly this: growth is on the left-hand side times the number of researchers, and the TFP of the idea production function is A^(−β). For β > 0, ideas are getting harder to find. The evidence we document in lots of different places very much looks like the model I wrote down.

Let me show you some of that evidence. First, for the aggregate economy, what does TFP growth — the blue line — look like?

It was high in the ‘40s, ‘50s, and ‘60s, and it’s been declining ever since.

What about the number of researchers? That’s on the right-hand axis. If research productivity were constant, growth rates would be proportional to the number of researchers — the original Romer, Aghion-Howitt, and Grossman-Helpman models said that. They assumed research productivity was constant. What we’re showing is that research productivity is falling. That’s why I’ve made the scales similar: the model with constant research productivity says these two lines should lie on top of each other. Instead, growth is flat or declining, and the number of researchers has grown by a factor of 23 in the United States since 1930. If research productivity were constant, growth rates would have grown by a factor of 23, but they haven’t. Research productivity at the aggregate is declining — by more than a factor of 23, because the growth rate is declining.

My favorite example is Moore’s law. Moore’s law is a straight line on a log scale, very much like US income per person.

It’s been behind most of the rapid growth we’ve had in the last 25 years. It says the number of transistors we fit on a computer chip doubles every two years. For the last 50 years that’s been true.

Where does Moore’s law come from? Notice: constant slope on a log scale — constant exponential growth, doubling every two years. The growth rate is 70/2, or 35% per year. Moore’s law is not a law of nature — it doesn’t happen regardless of what we do. It’s viewed as a target by the industry, and we’re throwing more resources at these problems to stay on the line. If you measure the number of researchers devoted to pushing Moore’s law forward — Intel, AMD, Samsung, TSMC, and ASML, but also IBM, AT&T, and Motorola historically, or National Semiconductor and Fairchild Semiconductor in the ‘70s — it’s growing incredibly rapidly.

It takes about 18 times the number of researchers today to generate the doubling of chip density as it took in the ‘70s.

Again, growth is constant, the number of researchers is rising like crazy — so research productivity is falling and ideas are getting harder to find. We have to run faster — throwing ever more resources at these problems — to maintain constant growth. That’s what the growth model I started with says. We look at agriculture, medical innovations, Compustat data, the aggregate economy — you see this fact everywhere. We haven’t found a place where you don’t. You have to double researchers every decade to maintain constant growth. If we ever stopped doing that, growth would presumably slow down. That’s what the model says.

Let me do some growth accounting — this gets at the earlier question. I’ll give a broad overview. I can add capital, human capital, and misallocation, M, and do growth accounting with a model that includes the idea production function.

I’m using the balanced growth path version, where A = R^γ and R = sL — s is the fraction of people engaged in research, and L is the total population. When you write down this richer model, in the long run all growth is proportional to population growth, so it’s γn again, even with all these other ingredients. But along a transition path you can grow faster than γn.

Historically, how does growth accounting look in this richer model?

I’m writing out the terms. There is:

• A capital-output ratio,

• Educational attainment,

• The employment-population ratio — what fraction of people are working; and then,

• TFP growth.

TFP growth in turn has a misallocation and an idea term. The idea term has the research intensity and population growth.

In the long run, all of these terms, other than the last one, are zero:

• Educational attainment — what fraction of your time you spend in school — is bounded by 100%.

• What fraction of your population you put in research — the s variable — is bounded at 100%. It has to stop growing eventually.

But historically, s has been rising, and educational attainment has been rising. Historically, not all growth was due to population growth.

Let’s do the accounting. What you get is summarized in these pie charts:

• The capital-output ratio is pretty constant.

• Human capital per person rises at about a year per decade in the 20th century. Each year [of additional education], labor economists tell us, raises wages by 6%. It’s actually risen by less than a year per decade, so each decade we’re adding 5% to wages. Wages are output per person. If you divide by 10 you get 0.5% per year. Rising educational attainment accounted for a quarter of our 2% growth in much of the 20th century. But it’s already leveling out.

• Similarly, rising female labor force participation, the employment-population ratio: 2 percentage points. But the employment-population ratio has to level out at some point — it’s already leveled out.

These historical sources of growth are going away.

TFP growth is the residual, 1.3.