Back testing is useful as a demonstration and validation, but the real question is: what does your algorithm predict as the *next* breakthroughs, based on the current state of the literature?
(I would also be curious to know if you've run it further afield from biomedicine. Can it pick out the next superconductor?)
Great questions! Our paper concentrates on biomedical research, but if you look in the supplemental data, there are some signals from other fields.
Here’s one example: in our 2017 data there’s a signal predicting a breakthrough in carbon dioxide conversion catalysis. In January of this year (2026) a European team described a molecular catalyst capable of converting near-atmospheric concentrations of carbon dioxide into carbon monoxide, which can be combined with hydrogen to make a fuel called syngas: https://onlinelibrary.wiley.com/doi/10.1002/anie.8293935
Getting a comprehensive answer requires expanding our input data, something I’m very interested in doing. Stay tuned!
Highly interesting - is this method only using scientific awards as a response? Would there be other measures of success that could be used like citations outside of topic or policy document citations?
Thank you! Our model is trained on prize-winning topics because the concept of a breakthrough is fuzzy, and we theorized that a big prize is an indicator of strong consensus. When you’re teaching the algorithm, it’s important to be as clear as possible about the distinction between what you are and aren’t looking for.
But the model does return signals with more real-world impact than a prize. One is the HIV quality of life work I described. Our paper reports other, more recent signals that might ultimately fit into this category. Examples include the use of DEXA scans to quantify bone fragility through something called the trabecular bone score (signal in 2015, FDA approval in 2016), and the use of mobile phones to deliver healthcare (think teledoc appointments, also a 2015 signal).
Mapping to citations in policy documents would be another important way to demonstrate impact, and is a great direction for future research!
Really great work. In thinking it through, if you were to run that experiment and seek to leave ample room for self-organizing and reduce incumbency in the researcher pool (this is how I’m reading this, correct me if I am wrong), wouldn’t you want to do this via pull funding? Am I misinterpreting the chain of logic here?
Thank you! You’re absolutely right, pull funding would be a natural fit for this approach. A more traditional NIH/NSF-style solicitation could also be applied, although that might be a little less elegant.
Interesting read. What is the false positive rate on this signal? i.e. don't start with breakthroughs but with detecting that pattern and then asking whether a breakthrough or a major award resulted
Back testing is useful as a demonstration and validation, but the real question is: what does your algorithm predict as the *next* breakthroughs, based on the current state of the literature?
(I would also be curious to know if you've run it further afield from biomedicine. Can it pick out the next superconductor?)
Great questions! Our paper concentrates on biomedical research, but if you look in the supplemental data, there are some signals from other fields.
Here’s one example: in our 2017 data there’s a signal predicting a breakthrough in carbon dioxide conversion catalysis. In January of this year (2026) a European team described a molecular catalyst capable of converting near-atmospheric concentrations of carbon dioxide into carbon monoxide, which can be combined with hydrogen to make a fuel called syngas: https://onlinelibrary.wiley.com/doi/10.1002/anie.8293935
Getting a comprehensive answer requires expanding our input data, something I’m very interested in doing. Stay tuned!
Highly interesting - is this method only using scientific awards as a response? Would there be other measures of success that could be used like citations outside of topic or policy document citations?
Thank you! Our model is trained on prize-winning topics because the concept of a breakthrough is fuzzy, and we theorized that a big prize is an indicator of strong consensus. When you’re teaching the algorithm, it’s important to be as clear as possible about the distinction between what you are and aren’t looking for.
But the model does return signals with more real-world impact than a prize. One is the HIV quality of life work I described. Our paper reports other, more recent signals that might ultimately fit into this category. Examples include the use of DEXA scans to quantify bone fragility through something called the trabecular bone score (signal in 2015, FDA approval in 2016), and the use of mobile phones to deliver healthcare (think teledoc appointments, also a 2015 signal).
Mapping to citations in policy documents would be another important way to demonstrate impact, and is a great direction for future research!
Really great work. In thinking it through, if you were to run that experiment and seek to leave ample room for self-organizing and reduce incumbency in the researcher pool (this is how I’m reading this, correct me if I am wrong), wouldn’t you want to do this via pull funding? Am I misinterpreting the chain of logic here?
Thank you! You’re absolutely right, pull funding would be a natural fit for this approach. A more traditional NIH/NSF-style solicitation could also be applied, although that might be a little less elegant.
Very cool! I hope people use this thinking!
Interesting read. What is the false positive rate on this signal? i.e. don't start with breakthroughs but with detecting that pattern and then asking whether a breakthrough or a major award resulted
Starting with all 18 signals between 1994 and 1997, 17 can be traced to a major breakthrough. The 18th is Vioxx/Celebrex.
Cool!