I want to preface that I've done LLM evals in a proper setting before. This is far sillier. It's derivative of the "pelican riding a bicycle" test, something simple that shows how capable an LLM is at perceiving intent, dealing with physics and general aesthetics. I run this for every new model now and it gives me a decent idea of what to expect.
The setup needs more description. Can it have pre-installed npm, npx, chrome, playright etc.? It turns out my agent did not have access to it and it is setting all of that itself. Will update when done.
I want to preface that I've done LLM evals in a proper setting before. This is far sillier. It's derivative of the "pelican riding a bicycle" test, something simple that shows how capable an LLM is at perceiving intent, dealing with physics and general aesthetics. I run this for every new model now and it gives me a decent idea of what to expect.
The setup needs more description. Can it have pre-installed npm, npx, chrome, playright etc.? It turns out my agent did not have access to it and it is setting all of that itself. Will update when done.
would be better if it compared to open weights models.