Author here. We build tooling for rewriting legacy systems, and wanted to test our approach on code we didn't pick, graded by someone else.
Setup: all 21 RepoMod-Bench cases, three unattended runs each, one model throughout. As a control we reran the paper's Codex setup on the same model, since ours is newer than the paper's.
Before anyone asks: we report the best of three runs (control as well); most of the jump from the paper's numbers is the model; two cases are regraded (missing a fixture); and we tuned on the initial three cases, then froze the workflow before running the rest.
Author here. We build tooling for rewriting legacy systems, and wanted to test our approach on code we didn't pick, graded by someone else.
Setup: all 21 RepoMod-Bench cases, three unattended runs each, one model throughout. As a control we reran the paper's Codex setup on the same model, since ours is newer than the paper's.
Before anyone asks: we report the best of three runs (control as well); most of the jump from the paper's numbers is the model; two cases are regraded (missing a fixture); and we tuned on the initial three cases, then froze the workflow before running the rest.
Everything is on Hugging Face, about 60 GB.