I really enjoyed this, as it exactly matches my thinking about how we should approach the study of science. I think measurable incentives are a bit overrated and more suited for endeavours where the relationship between input and output is linear. Science is the opposite of that, with Pareto outcomes. Great Science is transformative, rare and happens when serendipidity and the will and passion of a few meet. Another thing that is underappreciated is the importance of culture in Science -- this can be seen through the fact that the mentees of top scientists are much more likely to have distinguished careers themselves and the effect is huge. One could argue this is all because of nepotism, but I think something more than that is going on: small, imperceptible habits and ways of thinking are being passed on. This is not very legible from the outside and I think would be hard to quantify via the usual methods from Economics or Quantitative Social Science.
It's perhaps why I think anthropological, descriptive approaches are underrated. A while ago I wrote this piece https://www.writingruxandrabio.com/p/the-weird-nerd-comes-with-trade-offs and I called it "a metascience post of sorts", because I think this descriptive, observational approach is important.
Have you considered the distinction here between hard money and soft money positions? I would say that the strong researcher hypothesis is likely true for hard money positions where not getting grant doesn't mean you lose your position. I don't think you can influence intrinsic motivation for the top 0.1% of researchers but metascience interventions are likely to have good floor raising effects for workhorse science.
All to say, I agree that we should think carefully about how we evaluate proposed changes to incentives.
A big part of this IMO is that current organizations and institutions aren't able to handle or account for very human but "intangible", hard to quantify, qualitative factors and motivations. Part of this may be due to technocratic approaches (some critics might even say "technopoly") and influence of a sort of "scientism", and a priority placed on risk and mitigation of risk to organizations, therefore trying to focus on things that can be quantified and managed.
In the introduction to the recent book "Technofeudalism", the leftist commentator Yanis Varoufakis talks about the tension between "experiential value" and experiential labour in capital driven systems, vs. commodity labour, and how organizations have trouble quantifying and managing experiential labour, even though it's the uncommodifiable "effort, inspiration, goodwill" etc. which is "actually what makes the [institution] desirable".
Perhaps part of the solution is institutional organizations in general adopting a more humanist governance approach that works towards resilience but simultaneously is comfortable with uncertainty and understanding of intrinsic motivations and even transcendental values that motivate humans to pursue discovery and inquiry.
This makes an interesting point that funding might not be so effective at shaping the motivations of top researchers.
But as you allude to, other metascience interventions could be more effective. And to see how, we need to reframe of the Strong Researcher Hypothesis a bit.
To the extent science is cumulative, strong researchers are most effective if they operate in an ecosystem of trust, where they can build on credible research from other researchers, including weaker researchers. That's why metascience interventions to increase truth-seeking and honesty can help propel strong researchers, even though the immediate target of those interventions might be weaker researchers!
A good analogy here is actually capitalism. In the long run, strong companies are the only ones that matter. But for them to succeed, they need to operate in a world where other companies mostly follow the rules. How those rules are shaped exactly can make a huge difference.
I think the clearest role for pecuniary influence is at the point where proto-researchers are diverted from the field itself — we all know folks who had the résumé and personality for academic research but got poached by a private lab or company because of the general sense of misery of public academic work. Even small symbolic efforts that try and make grantwriting or incentive structures better for academics have the potential to bring the next Kariko out of Verily or OpenAI and to a public research university.
I really enjoyed this, as it exactly matches my thinking about how we should approach the study of science. I think measurable incentives are a bit overrated and more suited for endeavours where the relationship between input and output is linear. Science is the opposite of that, with Pareto outcomes. Great Science is transformative, rare and happens when serendipidity and the will and passion of a few meet. Another thing that is underappreciated is the importance of culture in Science -- this can be seen through the fact that the mentees of top scientists are much more likely to have distinguished careers themselves and the effect is huge. One could argue this is all because of nepotism, but I think something more than that is going on: small, imperceptible habits and ways of thinking are being passed on. This is not very legible from the outside and I think would be hard to quantify via the usual methods from Economics or Quantitative Social Science.
It's perhaps why I think anthropological, descriptive approaches are underrated. A while ago I wrote this piece https://www.writingruxandrabio.com/p/the-weird-nerd-comes-with-trade-offs and I called it "a metascience post of sorts", because I think this descriptive, observational approach is important.
> Through reforms to scientific publication, changes to granting, and other procedural tweaks
These reforms can alternatively be understood as ways to stop non-scientists from blocking/interfering with actual scientists.
Have you considered the distinction here between hard money and soft money positions? I would say that the strong researcher hypothesis is likely true for hard money positions where not getting grant doesn't mean you lose your position. I don't think you can influence intrinsic motivation for the top 0.1% of researchers but metascience interventions are likely to have good floor raising effects for workhorse science.
All to say, I agree that we should think carefully about how we evaluate proposed changes to incentives.
A big part of this IMO is that current organizations and institutions aren't able to handle or account for very human but "intangible", hard to quantify, qualitative factors and motivations. Part of this may be due to technocratic approaches (some critics might even say "technopoly") and influence of a sort of "scientism", and a priority placed on risk and mitigation of risk to organizations, therefore trying to focus on things that can be quantified and managed.
In the introduction to the recent book "Technofeudalism", the leftist commentator Yanis Varoufakis talks about the tension between "experiential value" and experiential labour in capital driven systems, vs. commodity labour, and how organizations have trouble quantifying and managing experiential labour, even though it's the uncommodifiable "effort, inspiration, goodwill" etc. which is "actually what makes the [institution] desirable".
Perhaps part of the solution is institutional organizations in general adopting a more humanist governance approach that works towards resilience but simultaneously is comfortable with uncertainty and understanding of intrinsic motivations and even transcendental values that motivate humans to pursue discovery and inquiry.
This makes an interesting point that funding might not be so effective at shaping the motivations of top researchers.
But as you allude to, other metascience interventions could be more effective. And to see how, we need to reframe of the Strong Researcher Hypothesis a bit.
To the extent science is cumulative, strong researchers are most effective if they operate in an ecosystem of trust, where they can build on credible research from other researchers, including weaker researchers. That's why metascience interventions to increase truth-seeking and honesty can help propel strong researchers, even though the immediate target of those interventions might be weaker researchers!
A good analogy here is actually capitalism. In the long run, strong companies are the only ones that matter. But for them to succeed, they need to operate in a world where other companies mostly follow the rules. How those rules are shaped exactly can make a huge difference.
I think the clearest role for pecuniary influence is at the point where proto-researchers are diverted from the field itself — we all know folks who had the résumé and personality for academic research but got poached by a private lab or company because of the general sense of misery of public academic work. Even small symbolic efforts that try and make grantwriting or incentive structures better for academics have the potential to bring the next Kariko out of Verily or OpenAI and to a public research university.