A Slinky on an Escalator Is a Weirdly Good Test of LLM Intelligence

(morethanamachine.com)

16 points | by reddmachine 8 hours ago ago

3 comments

  • reddmachine 8 hours ago ago

    I want to preface that I've done LLM evals in a proper setting before. This is far sillier. It's derivative of the "pelican riding a bicycle" test, something simple that shows how capable an LLM is at perceiving intent, dealing with physics and general aesthetics. I run this for every new model now and it gives me a decent idea of what to expect.

  • Rodmine 2 hours ago ago

    The setup needs more description. Can it have pre-installed npm, npx, chrome, playright etc.? It turns out my agent did not have access to it and it is setting all of that itself. Will update when done.

  • cyanydeez 2 hours ago ago

    would be better if it compared to open weights models.