Testing it against Kimi is potentially skewing the numbers massively. Kimi has a number of quirks that requires behaviours that e.g. Claude or GPT doesn't.
Harnesses that are built around needing to work with "weird" models will need to deal with that, such as Kimi's tendency to get stuck in tool-call loops.
Harnesses built to deal with e.g. Anthropic's models primarily, do not need to deal with that.
Claiming on the blog that this gives Kimi Code no home field advantage seems like a dicey assumption. I haven't dug into the newest Kimi Code much, but the older Kimi CLI included several tools that were clearly specifically aimed at working around that behaviour - when I copied their checkpoints and "dmail" mechanism into my own harness, the performance with Kimi improved dramatically, but it made zero difference against Anthropic models.
That doesn't make the data worthless - it's clear you shouldn't use Clade Code to work against Kimi. But it does significantly limit the utility of it.
Good points! That’s a known limitation of our v1.0 benchmark with Claude Code. For v1.1, we're expanding both harnesses and models to evaluate the full harness × model matrix. This will highlight interaction effects (showing which harnesses suit which models best) and let us publish a full compatibility matrix. We shared more details on our roadmap in our blog post (https://runta.com/blog/introducing-frontierharness-eval/).
That's great. It'd definitively be very interesting. Not least as an indicator of which harnesses to dig into the code of.
Kimi-cli is actually very interesting in that respect for the checkpoint / messaging mechanism I mentioned; it's basically almost like a model-initiated partial compaction of the end of a conversation to prune investigations that happened on the main agent loop - it'll be really interesting to see if it helps on any other models.
So oe thing that'd be really interesting to see when you expand to other models, would be if you mine the traces from those harnesses and generate stats on which tools get called. Seeing which models manage to take advantage of custom tools from which harnesses would be quite useful.
Afaik the reverse is also true. Some models are trained for specific harness setups so work better/more efficiently in that harness. If you give a model trained on a Read/Write/Bash tool setup a harness with only Bash where it needs to generate shell commands, I'd expect it to perform differently.
Related, there was a small debacle when Claude Code yeeted out a prompt change telling Opus to use Bash for everything. It was speculated it was an optimization to try to make it more efficient by encouraging it to chain a bunch of "tool use" commands into a giant shell command. However, they didn't communicate it clearly and it broke a bunch of setups that expected/hooked tool calls
There's also the (anecdotal?) tradeoff that dumb models tend to do better with smart tools and smart models tend to do better with dumb tools (smart tools -> easier to use but require more context usage for specs, dumb tools -> harder to use but more versatile and save context)
One issue I see with including something like Pi is that it's intentionally bare bones. I don't think anyone uses Pi without some basic custom extensions (subagents, check lists, etc), so this benchmark may not be representative of a realistic setup.
Would be interesting to see some sort of ablation test too. E.g. what parts of Codex contribute most to the perf, and can they be recreated more minimally in something like Pi.
It does seem a bit odd to say "We tested all of these bicycles with the same rider" when one of the test cases is actually a bare high-end frame with no components on it.
It's even weirder given that another of the test cases is essentially "We put parts on the frame (OhMyPi) and it went faster!"
I think it's still a useful data point. For example, omp, which is pi with some default extensions, scores worse. I do agree that adding more configurations of Pi would help though.
This is great. People always talk about how important the harness is and yet we have so few harness benchmarks. Agree with sibling commenter that ‘harness x model’ is needed.
People always talk about how Cursor harness has some secret sauce; would like to see how that one stacks up.
Thanks! V1.1 is set to add more harnesses and evaluate a broader range of models. We're moving to a full harness × model matrix to uncover interaction effects and release a complete harness-model compatibility map.
I am looking forward to your findings on home-field advantage. It's natural that Claude Code will work better with Anthropic models. If we know by how much, it will inform many of us.
I'm not sure i agree with these. I've been testing a simple prompt with different model and harness combinations to work out which one produces best results. I do this in '/goal' mode.
Prompt -
"Build a single-page Three.js sci-fi hangar with hovering drones, animated warning lights, emissive runway strips, and subtle volumetric-style fog planes. Include drone formation toggle and cinematic camera path. Output one self-contained HTML file with inline JavaScript."
OMP i would say comes out on this prompt the best out of all the harnesses i've tried. I suspect possibly a lot of this is to do with the base system prompt that each harness injects.
I saw that very much developing 3code. Even slight system prompt variations can have huge differences in results, so I have one system prompt per model family.
I also feel Copilot is not getting the attention it should considering its user base and its importance in the enterprise context. There is a lot to be learned and I found https://news.ycombinator.com/item?id=49256057 (mitm copilot investigation) interesting. Playing with Copilot and asking it what was the biggest part of its standard payload I was surprised to learn that VSCode extensions plug in their MCP servers without me being much aware of it.
I was excited until I saw cost was only provided as the median. Your provider will bill you for all your tasks and one can get back to that total from the mean by multiplying by the number of tasks. This isn't possible with the median and I suspect the median is likely below the mean so this understates the actual costs.
Claude Code may simply be best used with Anthropic's models and quite bad with Kimi. An alternate solution is to remove Claude Code from the diagram if its so far off from the others that it causes scaling problems.
I was looking at the results JSON and it looks like there is only one run of each task with each harness. Since these are disparate tasks using the median means that the headline cost is the cost of one specific task for each harness (or the mean of two it looks like in the case of Exo Harness; I couldn't spot the one task that lined up with the headline cost), but the same task isn't used for the headline cost for each harness. It's not the same as picking a task at random to use as the headline task, but its in the ballpark.
There really should be error bars on those measurements. With just 30 samples, the 95% confidence intervals for accuracy should all be more than 35 percentage points wide, so comfortably overlap. For cost it's harder to say, because outcomes aren't constrained to {0, 1}, but I also expect a lot of variability there.
Version 1.1 will add more harnesses and evaluate a broader set of models. Rather than holding the model fixed, we plan to evaluate the full harness × model matrix to reveal interaction effects (which harnesses work best with which models) and produce a harness-model compatibility matrix.
Do you have instructions on how to run a custom harness against this? There are none on the linked page. I want to run Dirac (https://github.com/dirac-run/dirac).
Interesting to see harness benchmarks for coding agents. The same problem exists for conversational and data agents but I dont see anyone benchmarking them yet. Seems like manual-spot checks are the norm
It would be interesting to try to control for system prompt although obviously there's some coupling to what the harness provides and the instructions the model receives on using it
Deepseek is fuckin fast! Seems like the bigger the task, the faster it gets, which is kind of unfortunate because nobody is going to be doing 17 benchmark passes on a $50-100 task. I'm assuming the brief pause before it avalanches out 16kb of text at upwards of 250tk/s (multiples beyond anything resembling a comfortable reading speed) is some sort of workload evaluator directing sessions to individual/multiple cards, occasionally waiting for what it thinks is best to become available.
I couldn't believe it at first, it shit out a damn fine multithreaded physics simulation fabric (integrated into a massive codebase, tests passing) in under an hour. Anything I could find online says they average like 80 but my logs average ~3x that.
>nobody is going to be doing 17 benchmark passes on a $50-100 task
Hopefully we'll see more of this as big companies try to optimize token usage where the cost of benchmarking is dwarfed by the potential savings across the org
I use routatic/proxy (actually my own fork with various changes/fixes http://github.com/nijave/proxy/ -- fyi the fork is all vibes...)
Some gateways/proxies/hosts also offer multiple endpoints (z.ai has both Anthropic endpoint and Completions endpoint)
Edit: Here's the PoC I made for work although it assumes you're using Cloudflare Access (VPN) with a Cloudflare AI Gateway to hand wave the auth away (auth to the LLM API gets auto injected by Cloudflare Access derived from the client's VPN connection) https://gist.github.com/nijave/6f5e5793a992619808135ce85fa49...
Claude Code supports the base URL env var so you could tell it to talk with any LLM API endpoint that receives the Anthropic style request format, e.g. DeepSeek.
Claude Code allows you to change the config such that you use the harness with different models - you can actually ask a coding agent to configure it for you
Testing it against Kimi is potentially skewing the numbers massively. Kimi has a number of quirks that requires behaviours that e.g. Claude or GPT doesn't.
Harnesses that are built around needing to work with "weird" models will need to deal with that, such as Kimi's tendency to get stuck in tool-call loops.
Harnesses built to deal with e.g. Anthropic's models primarily, do not need to deal with that.
Claiming on the blog that this gives Kimi Code no home field advantage seems like a dicey assumption. I haven't dug into the newest Kimi Code much, but the older Kimi CLI included several tools that were clearly specifically aimed at working around that behaviour - when I copied their checkpoints and "dmail" mechanism into my own harness, the performance with Kimi improved dramatically, but it made zero difference against Anthropic models.
That doesn't make the data worthless - it's clear you shouldn't use Clade Code to work against Kimi. But it does significantly limit the utility of it.
Good points! That’s a known limitation of our v1.0 benchmark with Claude Code. For v1.1, we're expanding both harnesses and models to evaluate the full harness × model matrix. This will highlight interaction effects (showing which harnesses suit which models best) and let us publish a full compatibility matrix. We shared more details on our roadmap in our blog post (https://runta.com/blog/introducing-frontierharness-eval/).
That's great. It'd definitively be very interesting. Not least as an indicator of which harnesses to dig into the code of.
Kimi-cli is actually very interesting in that respect for the checkpoint / messaging mechanism I mentioned; it's basically almost like a model-initiated partial compaction of the end of a conversation to prune investigations that happened on the main agent loop - it'll be really interesting to see if it helps on any other models.
So oe thing that'd be really interesting to see when you expand to other models, would be if you mine the traces from those harnesses and generate stats on which tools get called. Seeing which models manage to take advantage of custom tools from which harnesses would be quite useful.
Afaik the reverse is also true. Some models are trained for specific harness setups so work better/more efficiently in that harness. If you give a model trained on a Read/Write/Bash tool setup a harness with only Bash where it needs to generate shell commands, I'd expect it to perform differently.
Related, there was a small debacle when Claude Code yeeted out a prompt change telling Opus to use Bash for everything. It was speculated it was an optimization to try to make it more efficient by encouraging it to chain a bunch of "tool use" commands into a giant shell command. However, they didn't communicate it clearly and it broke a bunch of setups that expected/hooked tool calls
There's also the (anecdotal?) tradeoff that dumb models tend to do better with smart tools and smart models tend to do better with dumb tools (smart tools -> easier to use but require more context usage for specs, dumb tools -> harder to use but more versatile and save context)
One issue I see with including something like Pi is that it's intentionally bare bones. I don't think anyone uses Pi without some basic custom extensions (subagents, check lists, etc), so this benchmark may not be representative of a realistic setup.
Would be interesting to see some sort of ablation test too. E.g. what parts of Codex contribute most to the perf, and can they be recreated more minimally in something like Pi.
It does seem a bit odd to say "We tested all of these bicycles with the same rider" when one of the test cases is actually a bare high-end frame with no components on it.
It's even weirder given that another of the test cases is essentially "We put parts on the frame (OhMyPi) and it went faster!"
I'm not sure your analogy holds, because here the optimization metric is clear and unanimous: every one wants max pass rate at min costs.
With the bicycle, some may prefer comfort, others speed, others offroad, etc., so it would not be obvious which one is "best".
Not weird or odd in the least. I think it clearly shows that Pi barebones at least in this set of tests, preformed better and cheaper cost that OMP.
I think it's still a useful data point. For example, omp, which is pi with some default extensions, scores worse. I do agree that adding more configurations of Pi would help though.
I use Pi barebones.
I also prefer using vanilla Pi over Oh My Pi.
This is great. People always talk about how important the harness is and yet we have so few harness benchmarks. Agree with sibling commenter that ‘harness x model’ is needed.
People always talk about how Cursor harness has some secret sauce; would like to see how that one stacks up.
Thanks for making this and filling a real gap!
Thanks! V1.1 is set to add more harnesses and evaluate a broader range of models. We're moving to a full harness × model matrix to uncover interaction effects and release a complete harness-model compatibility map.
I am looking forward to your findings on home-field advantage. It's natural that Claude Code will work better with Anthropic models. If we know by how much, it will inform many of us.
I'm not sure i agree with these. I've been testing a simple prompt with different model and harness combinations to work out which one produces best results. I do this in '/goal' mode.
Prompt -
"Build a single-page Three.js sci-fi hangar with hovering drones, animated warning lights, emissive runway strips, and subtle volumetric-style fog planes. Include drone formation toggle and cinematic camera path. Output one self-contained HTML file with inline JavaScript."
You can see the outputs here - https://alvins82.github.io/hangar-harness-model-tests/ https://github.com/alvins82/hangar-harness-model-tests/
OMP i would say comes out on this prompt the best out of all the harnesses i've tried. I suspect possibly a lot of this is to do with the base system prompt that each harness injects.
Putting only the tasks and results in the repo is a poor decision. These conclusions would be far more credible if anyone could re-run the benchmark.
I saw that very much developing 3code. Even slight system prompt variations can have huge differences in results, so I have one system prompt per model family.
Is Github Copilot (integrated in VS Code) not a thing? I use it all the time and don't know what more I could wish for.
Disclaimer: I don't let agents run on huge tasks for hours. Almost all tasks I give them are done in under 30 min.
I also feel Copilot is not getting the attention it should considering its user base and its importance in the enterprise context. There is a lot to be learned and I found https://news.ycombinator.com/item?id=49256057 (mitm copilot investigation) interesting. Playing with Copilot and asking it what was the biggest part of its standard payload I was surprised to learn that VSCode extensions plug in their MCP servers without me being much aware of it.
I was excited until I saw cost was only provided as the median. Your provider will bill you for all your tasks and one can get back to that total from the mean by multiplying by the number of tasks. This isn't possible with the median and I suspect the median is likely below the mean so this understates the actual costs.
We initially tried using the mean (average), but a few extreme outliers caused Claude Code's cost to look far higher than it typically is.
Claude Code may simply be best used with Anthropic's models and quite bad with Kimi. An alternate solution is to remove Claude Code from the diagram if its so far off from the others that it causes scaling problems.
I was looking at the results JSON and it looks like there is only one run of each task with each harness. Since these are disparate tasks using the median means that the headline cost is the cost of one specific task for each harness (or the mean of two it looks like in the case of Exo Harness; I couldn't spot the one task that lined up with the headline cost), but the same task isn't used for the headline cost for each harness. It's not the same as picking a task at random to use as the headline task, but its in the ballpark.
There really should be error bars on those measurements. With just 30 samples, the 95% confidence intervals for accuracy should all be more than 35 percentage points wide, so comfortably overlap. For cost it's harder to say, because outcomes aren't constrained to {0, 1}, but I also expect a lot of variability there.
Good start, but needs to be harness x model to be useful. 3 top harnesses x 3 top models would be more interesting.
Version 1.1 will add more harnesses and evaluate a broader set of models. Rather than holding the model fixed, we plan to evaluate the full harness × model matrix to reveal interaction effects (which harnesses work best with which models) and produce a harness-model compatibility matrix.
With all the hype around the latest Gemini 3.8 Flash/Cyber release, will Antigravity CLI [1] be supported?
1/ https://antigravity.google/product/antigravity-cli
agy has fewer users than grok, both are ~1% based on some surveys I've seen, not every harness needs to be evaluated
hey, do you have a sample survey you could link? I don't have the slightest idea what I'd search for.
Do you have instructions on how to run a custom harness against this? There are none on the linked page. I want to run Dirac (https://github.com/dirac-run/dirac).
Their code and evalations are linked to on the blog post accompanying the results:
Blog Post: https://runta.com/blog/introducing-frontierharness-eval/
The tasks and methodology: https://github.com/runta-dev/frontier-harness-eval
+1, I'll also add Maki to the list: https://github.com/tontinton/maki
Same, I want to bench PasClaw on it.
Interesting to see harness benchmarks for coding agents. The same problem exists for conversational and data agents but I dont see anyone benchmarking them yet. Seems like manual-spot checks are the norm
It would be interesting to try to control for system prompt although obviously there's some coupling to what the harness provides and the instructions the model receives on using it
We'd rather not mess with system prompts, we just eval them as shipped to keep the results reproducible.
Kind of bizarre to see the article author here with [dead] posts… presumably because they are AI-generated.
I've been confused a lot why there isn't a benchmark to measure a harness's performance rather than the model's one. Now there it is.
Incredibile result for pi. The ratio quality / complexity of the harness make me think that all other harnesses are very bloated.
It's nice to see time reflected here.
Deepseek is fuckin fast! Seems like the bigger the task, the faster it gets, which is kind of unfortunate because nobody is going to be doing 17 benchmark passes on a $50-100 task. I'm assuming the brief pause before it avalanches out 16kb of text at upwards of 250tk/s (multiples beyond anything resembling a comfortable reading speed) is some sort of workload evaluator directing sessions to individual/multiple cards, occasionally waiting for what it thinks is best to become available.
I couldn't believe it at first, it shit out a damn fine multithreaded physics simulation fabric (integrated into a massive codebase, tests passing) in under an hour. Anything I could find online says they average like 80 but my logs average ~3x that.
>nobody is going to be doing 17 benchmark passes on a $50-100 task
Hopefully we'll see more of this as big companies try to optimize token usage where the cost of benchmarking is dwarfed by the potential savings across the org
DeepSeek is consistently the fastest model, and Kimi the slowest. GPT in the middle. Anthropic is probably on the slow side as well.
Agree with the exception of Haiku. Haiku is insanely fast if it works for your task
I just started using Hermes and it's pretty good. Guess I'll try Pi. I'm hesitant on DSH.
Very interesting. As models become commodities, the harness will be the next optimizing game.
Software is constrained when you write it. Agents have to be constrained while they run.
The Exo harness is the most interesting one in the benchmark, as it has the potential to complete tasks with fewer turns and cost.
Where is jcode?
Harness wars are the next browsers wars!!
But wow, opencode is that bad?!
Forgive my stupidity but how do you run Claude Code with non-Anthropic models?
I use routatic/proxy (actually my own fork with various changes/fixes http://github.com/nijave/proxy/ -- fyi the fork is all vibes...)
Some gateways/proxies/hosts also offer multiple endpoints (z.ai has both Anthropic endpoint and Completions endpoint)
Edit: Here's the PoC I made for work although it assumes you're using Cloudflare Access (VPN) with a Cloudflare AI Gateway to hand wave the auth away (auth to the LLM API gets auto injected by Cloudflare Access derived from the client's VPN connection) https://gist.github.com/nijave/6f5e5793a992619808135ce85fa49...
Here's an example for using it with GLM:https://docs.z.ai/devpack/tool/claude ... works similarly for other models
Claude Code supports the base URL env var so you could tell it to talk with any LLM API endpoint that receives the Anthropic style request format, e.g. DeepSeek.
The full list for the curious
https://code.claude.com/docs/en/env-vars#variables
Claude Code allows you to change the config such that you use the harness with different models - you can actually ask a coding agent to configure it for you
Codex is the best harness to me because of its GUI and subscription.