Most similar is CT Open, which is also a trial outcome predictor. ForecastBench does the same but for general questions.
For 111 trials reported after the cutoff date, Opus/Sonnet/Haiku was too pessimistic (~59% expected vs. 73% observed) and was beaten by a plain regression on the registry features. Their ordering of the trials was good though, and combining with the regression got them 8% accuracy increase. Small sample size; thus, the online version
Are there other projects like this? Also, what results have you been able to find on historical data?
Most similar is CT Open, which is also a trial outcome predictor. ForecastBench does the same but for general questions.
For 111 trials reported after the cutoff date, Opus/Sonnet/Haiku was too pessimistic (~59% expected vs. 73% observed) and was beaten by a plain regression on the registry features. Their ordering of the trials was good though, and combining with the regression got them 8% accuracy increase. Small sample size; thus, the online version
This is very cool!