Show HN: KradleBench – Multi-agent AI evals in Minecraft

(kradle.ai)

8 points | by jamest 5 hours ago ago

7 comments

  • vikrum 4 hours ago ago

    (vikrum from Kradle here, too)

    One of the most interesting things about this benchmark is that it encodes latency into the score as agents navigate the world where other agents are also doing things. Just because a model can score high when there is no time limit is not useful for many important workloads like active red/blue teaming. The dimension of "collaboration" in KradleBench adds color to another piece through this same prism: with a shared goal, and possibly with differing views of the world, how do they yield results greater than if it was just a single agent? We have bits and pieces of this in our day-to-day (multiple agents coordinating to create a pr, then multiple agents reviewing PRs against a code base), however, these also lack the time constraints.

    I'm excited to be pushing benchmarks forward. Having a safe place like Minecraft to ratchet up these types of tests is a lot of fun.

  • TheMrZZ 5 hours ago ago

    Florian from Kradle here - happy to answer any question! We're very excited about this release. We did not expect Astra 6.2 to score so high, especially with its current online reputation. Looks like it's better as an autonomous agent than a coding assistant!

  • stopachka 4 hours ago ago

    Great bench! Question: from playing with the models why do you think developers seem to prefer Opus / Fable, whereas Astra does much better here? What do you think Astra is lacking for them?

    • TheMrZZ 3 hours ago ago

      Great question. It seems that Astra is an excellent autonomous model (which is how we're testing them) but has very poor interactions with humans. Like using super complex vocabulary, asking questions way too deep for the user etc... It's annoying when using it as a tool, but does not cause issues within a video game.

      Some people also said it over-fixated on details. For coding, maybe? But from our PoV, it's actually the first model that got a lot of details correct in Minecraft. For example in our economy eval, Astra keeps improving its strategy through the game, when other models just do not grasp the complexity of the challenge. For ex Fable uses a very basic, bad strategy, and iterates at most once in the entire game.

      So maybe this attention to details (coupled with a great latency) make these models worse for coding but great for autonomous, real world tasks

  • gsoltis 4 hours ago ago

    It's wild that if this had been published a couple days ago, humans would've been in the lead. The state of the art is changing so quickly.

  • olivercameron 4 hours ago ago

    Who knew Minecraft would become so integral to AGI. Congrats on the launch!