I've been working on the same problem, seems like we ran into a lot of the same bottlenecks.
You're definitely right the grading of tasks is very challenging to do. First of all, you have to think about what it is that you even care about measuring, and then figure out how to encode that in code/llm-as-a-judge graders. For example some of the problems I ran into going down this rabbit hole:
Just use the PR tests -> these encode assumptions about naming and behavior
Tell the agent what naming and behavior to use -> prompt becomes overspecified
Use an agent to write tests -> agents are bad at writing tests
Use an agent to modify the agent patch to fit gold test signature -> this is unfair to some agent patches
Use LLM-as-a-judge instead of tests -> LLM is still noisy and biased
all to say - this is somewhat of an open domain from what I can tell, and you just have to try your best to extract some signal from the noise
Thanks for your reply and it's interesting that we ran into the same issues too.
Its challenging even describing what good looks like. Due to the nature of problem solving, there is several ways to a solution.
I've found it difficult because there isn't many references on how companies are even doing this. The companies that post that they do evals generally just vague post about it.
It's not like we can look at the past 10 years of history and see a good implementation of this, the patterns are still emerging.
Have you personally had any success with your attempt?
I have had success with specific repos since I'm able to tune tasks to be more fair. However, I'm working on making a generalized solution where this can be done more automatically for new arbitrary repos, and am still working on getting the success rate of task creation up.
My guess is there's a lot of secrecy around this since, to my limited understanding, a large part of model gains in the past 1-2 years is from scaling up RL environment creation, and what we're describing is in essence creating RL environments from repos, so there's a lot of value in keeping your methodology private.
I've been trying to write more about this, and actually am working on a post going into more detail on some of the issues I faced. I also have this post with some high level detail https://www.stet.sh/blog/benchmark-agreed-with-frontier-code
I've been working on the same problem, seems like we ran into a lot of the same bottlenecks.
You're definitely right the grading of tasks is very challenging to do. First of all, you have to think about what it is that you even care about measuring, and then figure out how to encode that in code/llm-as-a-judge graders. For example some of the problems I ran into going down this rabbit hole:
Just use the PR tests -> these encode assumptions about naming and behavior
Tell the agent what naming and behavior to use -> prompt becomes overspecified
Use an agent to write tests -> agents are bad at writing tests
Use an agent to modify the agent patch to fit gold test signature -> this is unfair to some agent patches
Use LLM-as-a-judge instead of tests -> LLM is still noisy and biased
all to say - this is somewhat of an open domain from what I can tell, and you just have to try your best to extract some signal from the noise
Thanks for your reply and it's interesting that we ran into the same issues too.
Its challenging even describing what good looks like. Due to the nature of problem solving, there is several ways to a solution.
I've found it difficult because there isn't many references on how companies are even doing this. The companies that post that they do evals generally just vague post about it.
It's not like we can look at the past 10 years of history and see a good implementation of this, the patterns are still emerging.
Have you personally had any success with your attempt?
I have had success with specific repos since I'm able to tune tasks to be more fair. However, I'm working on making a generalized solution where this can be done more automatically for new arbitrary repos, and am still working on getting the success rate of task creation up.
My guess is there's a lot of secrecy around this since, to my limited understanding, a large part of model gains in the past 1-2 years is from scaling up RL environment creation, and what we're describing is in essence creating RL environments from repos, so there's a lot of value in keeping your methodology private.
I've been trying to write more about this, and actually am working on a post going into more detail on some of the issues I faced. I also have this post with some high level detail https://www.stet.sh/blog/benchmark-agreed-with-frontier-code