Author here. I tested Jev with a fair die 400 times without telling it the die result. The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%.
I used a die as an example of problems whose answers can't be known at all, unlike problems with correct answers like MMLU questions. In a more practical test, a document said "30% risk" and Jev returned 5%. The point of the post is that Jev is weak at some kinds of tasks, so you should check the calibration for your own use case.
saw this yesterday (when someone else posted your post), thanks for sharing such easy to understand cases which show Jev does not produce real world likelihoods for the choices
Author here. I tested Jev with a fair die 400 times without telling it the die result. The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%.
Code and data are here: https://github.com/KantaHayashiAI/jev-does-not-play-dice
Happy to answer questions!
Maybe it watched Rosencrantz & Guildenstern Are Dead[1] once too many.
[1]: https://www.youtube.com/watch?v=C_TfdNAXOwE
I found that in the play it's 92 heads in a row [1], and in my coin test Jev returned around 92%. Maybe Jev was actually trained on it.
[1]: https://en.wikipedia.org/wiki/Rosencrantz_and_Guildenstern_A...
why would you use ai for that though? how does that translate to other tasks?
I used a die as an example of problems whose answers can't be known at all, unlike problems with correct answers like MMLU questions. In a more practical test, a document said "30% risk" and Jev returned 5%. The point of the post is that Jev is weak at some kinds of tasks, so you should check the calibration for your own use case.
saw this yesterday (when someone else posted your post), thanks for sharing such easy to understand cases which show Jev does not produce real world likelihoods for the choices
related, https://www.alexmolas.com/2026/09/23/jev-cant-be-calibrated....
Thanks! I'm really glad that you read my article.