← Back to blog

Coding Agent Benchmarks: Why a Leaderboard Is Not Your Repo

What a coding agent benchmark measures, why the same model scores differently on every leaderboard, and how to grade an agent on your own tickets instead.

Search “coding agent benchmark” and the first page is three arXiv papers, two vendor leaderboard hubs, a GitHub catalog of benchmarks, and a handful of listicles that walk through SWE-bench, Terminal-Bench, and Aider’s polyglot set one paragraph at a time. The best of them, an explainer from a harness vendor, admits that scores disagree across scaffolds and that contamination is real, then recommends its own index. The rest print the headline numbers and leave you to work out why the model that tops one board is fifth on another.

An engineering leader does not type that query to learn what SWE-bench stands for. They type it because someone on the team has a leaderboard open and wants to pick an agent from it, and the leader suspects, correctly, that the leaderboard is measuring something other than their repository. This post explains what a coding agent benchmark actually measures, why the numbers move so much, which ones are worth reading in October 2026, and how to grade an agent on your own tickets instead. It ends with how YAGNI grades its Agents on the only benchmark that pays: the work a customer’s reviewer accepts.

What does a coding agent benchmark actually measure?

A score on a coding agent leaderboard is the output of four things, and the headline hides three of them.

The harness is the agent program: how it reads the repository, how it plans, which tools it has, how it compacts context, how many turns it may take. The model is the weights the harness calls. The task set is a fixed list of problems, usually real GitHub issues or scripted terminal jobs, with a frozen repository state for each. The grader is the test suite or verifier that decides whether an attempt counts: for SWE-bench, the tests that failed before the fix must pass afterwards and the tests that passed must keep passing.

A leaderboard row is a tuple of all four. When a vendor says “our model scores 80 on SWE-bench Verified,” the honest sentence is “our harness, running our model, on the 500 Verified tasks, graded by those tasks’ tests, scored 80.” Every word in that sentence is a variable someone chose, and the two that move the number most, the harness and the task set, are the two that have nothing to do with your codebase.

Why do the same model’s scores disagree across leaderboards?

Because the harness is doing more of the work than the leaderboard admits. A September 2026 paper, Coding Agents Have Converged, measured how far one model’s SWE-bench Verified score moves when only the scaffold changes: up to 29.8 points within a single model, against an 8.8-point spread across the top thirty entries on the public board, with no adjacent pair in that top thirty statistically separable. The variance between harnesses is three times the variance between the models everyone is arguing about.

The benchmark maintainers know this, and the better boards now say which one they are measuring. Terminal-Bench lists Model, Agent, Resolution Rate, Cost, and Tokens as separate columns, so every entry is a named harness running a named model, and a vendor’s own agent sits in the same table as a minimal open-source one. Artificial Analysis runs every model through the same mini-swe-agent harness on Terminal-Bench 4.0 and reports pass@1 averaged over three repeats per task; in that run Kimi K3.1 Pro leads at 67.2%, with Claude Sonnet 5.5 at 63.6% and Claude Opus 5.5 at 59.6%. Vals.ai does the same on SWE-bench Verified with cost per test beside the score: in its September 2026 run, Opus 5 resolved 97.0% at $1.29 a test and DeepSeek V4-Pro 96.4% at $0.10.

Those fixed-harness boards answer a narrow, useful question: which model, given an identical and deliberately simple agent, solves the most tasks. The open-submission boards answer a different one: which product, tuned by its own vendor against this task set, scores highest. Neither is wrong. Reading one as if it were the other is how a team picks a model on a 2-point gap that the harness it will actually run erases.

Which coding agent benchmarks matter in October 2026?

Five come up on every page in the SERP. The table reads each one the way a buyer should: what it holds fixed, and what to take from it.

Benchmark Task source Harness What the score tells you What it cannot tell you
SWE-bench Verified 500 engineer-screened GitHub issues from Python repositories Submitter’s own on the main board; a fixed minimal agent on the fixed-scaffold boards Floor on capability; close to saturated for frontier and open-weight models alike Anything about non-Python work, or about a repository the model has not seen
SWE-Bench Pro 1,865 harder tasks across 41 repositories, part public, part held out Mostly submitter’s own, 250-turn limit Spread between models on long, multi-file changes A clean number: OpenAI’s July 2026 audit estimated about 30% of the public tasks are broken, mostly by overly strict hidden tests
Terminal-Bench 4.0 66 containerized terminal tasks across software, ML, ops, and security Named per entry, with cost and tokens The hardest current set; the harness column is the right shape for a buyer Comparability with 2.1 scores (89 tasks, a different ceiling) that vendors still quote
Aider polyglot 225 Exercism exercises in six languages Aider, fixed Edit-format reliability and cost per run on small problems Anything about a real codebase; the exercises are toy-sized
LiveCodeBench Contest problems with release dates, to dodge training-set contamination None; measures the model, not an agent Raw reasoning on fresh algorithmic problems Whether the model can navigate a repository at all

Two things to read off the table. Only Terminal-Bench puts the harness and the cost in the row, and it is the one a buyer should look at first for that reason. And the two task sets with the most citations, Verified and Pro, are the two whose maintainers and auditors have said out loud that the tasks need repair: OpenAI stopped publishing SWE-bench Verified in February 2026 after finding broken tasks in the hard subset, and its Pro audit flagged 200 tasks by automated analysis and 249 by human annotation. A score that moved from the twenties to the eighties on a public split in under a year is measuring the split as much as the models.

What is wrong with the tasks themselves?

Three things, and each one is a way your repository differs from the leaderboard’s.

The grader only knows one answer. A benchmark task ships with the maintainer’s fix and the tests written for it. An agent that solves the issue a different way fails. When Faros audited the disagreement cases on SWE-Bench Pro, attempts that failed the tests but scored well on a rubric, it found 23% were divergent designs: functionally valid patches the tests rejected because they did not match the reference. A human reviewer would have merged some of those. A human reviewer would also have rejected patches the tests passed, because they did the right thing in the wrong place. The grader on a leaderboard has no opinion on taste, conventions, or blast radius, and your reviewer has nothing else.

The task mix is not your task mix. SWE-bench is Python. Terminal-Bench is scripted terminal work with a known end state. Aider is Exercism. Your backlog last month was a config flip, a flaky test, a three-service migration, a dependency bump that broke the build, and a feature nobody had specified. The leaderboard’s distribution of difficulty, language, and ambiguity was chosen to make a benchmark gradable, and gradable is the property your tickets least share.

The repository is not blank. Benchmark tasks are solved against a frozen snapshot with the issue text as the whole brief. Your tickets are solved against a codebase with a history, a Slack thread that explains why the retry count is five, and a decision from July that the agent was never told. An agent that scores 80 on a snapshot and guesses on your conventions is a 60 on your repository, and the leaderboard cannot know that. This is the gap context engineering for coding agents is about: the record the agent reads before it guesses matters more than the model it guesses with.

How do you benchmark a coding agent on your own repository?

Run your own exam, on your own tickets, with your own reviewer as the grader. The recipe below is the one we walk customers through, and it is built so you can rerun it when the models change, which they will every quarter.

  1. Pick twenty tickets from last month that span your real mix: a few one-line fixes, several ordinary features, two or three multi-file changes, and at least one you consider hard. Use closed tickets, so a reference fix exists, but do not show it to the agent.
  2. Fix everything except the variable you are testing. Same context files, same skills, same MCP servers, same branch. If you are comparing models, hold the harness. If you are comparing harnesses, hold the model. Changing both at once is how you get a number nobody can act on.
  3. Run the hard tickets three times. A single pass on a hard task is a coin flip. Three repeats tell you whether the agent reaches a mergeable pull request reliably or occasionally, and the gap between pass@1 and pass-every-time is the number your on-call engineer cares about.
  4. Grade as a reviewer, not as a test suite. For each attempt, record whether it reached a mergeable pull request, and then what the reviewer did with it: accepted as proposed, edited before merge, or sent back. Record the reviewer’s minutes. A patch that passes CI and costs forty minutes of rework is not a pass.
  5. Record what each attempt cost, from the meter the harness prints or the vendor console, and divide by the number of accepted changes. Cost per accepted change is the only number that survives contact with the finance team.
  6. Name an owner and a date. The sheet is worthless if nobody owns rerunning it. Put a person’s name on it and a date next quarter.

Here is the shape of the sheet, with four rows filled in. The numbers are invented to show the arithmetic, not a measurement.

Ticket Mergeable PR? Reviewer verdict Reviewer minutes Run cost
Config flag for webhook retries Yes Accepted as proposed 6 $0.40
Flaky integration test Yes Edited (wrong fixture) 25 $1.10
Three-service migration (run 1 of 3) No Sent back 30 $4.80
Three-service migration (run 2 of 3) Yes Edited (missing rollback) 45 $5.20

Two accepted-or-edited changes for $11.50 of runs and 106 reviewer minutes is a different number from “75% mergeable,” and it is the one that tells you whether to route the migration to a stronger model or keep it with a person. The deeper version of that arithmetic, with dated vendor prices and three 2026 studies that measured cost per completed task, is in the best open source LLM for coding.

What does grading on real work look like on a YAGNI Team?

The sheet above is a one-off. YAGNI keeps it running, because an Agent Team is managed like an engineering team: ownership, review, and a record, and the record is the benchmark.

Every line of work on a Team keeps a track record, aggregated per job: accepted as proposed, edited, sent back, auto-shipped, and reversed, over time and per rule. It sits beside the line on the Team page so the person deciding how much that line may do on its own reads the real number, “22 of the last 25 accepted as proposed,” rather than a vendor’s score on someone else’s repository. Misses stay on the record. A reversal later is one more count, not a demotion.

The grader is your reviewer, every time. A ticket moves through six stages, each owned by a named Agent: Bailey, the Product Manager, proposes it from the Jira or Linear backlog with evidence; Reeve, the Staff Engineer, plans it, reading the decision ledger first; a person approves the plan; Wright, the Builder, builds it to a pull request in a sandbox; Proctor, the Reviewer, posts one review a person can read; Fletcher (QA, in beta) walks the change in a browser and attaches the video; Harper, the Chronicler, reports what shipped. A person merges. Every one of those acts lands on the ticket’s case file with what it cost, which is the audit trail the leaderboard never had: what an AI agent audit trail should contain is exactly the sheet in the previous section, kept for you.

And the cost column is kept too. The Usage page shows spend per developer and per Agent on one rate card, beside a benchmark of what the same work would have cost parked at the frontier, and the savings between them. YAGNI routes each step across vetted US-hosted open-weight models at 60% or more under comparable frontier API rates, and the per-ticket arithmetic above is how you check that the floor held on your tickets, not on a leaderboard’s.

When is a leaderboard still worth reading?

Three times. When you are shortlisting, a fixed-harness board like Artificial Analysis or Vals.ai is the cheapest way to rule a model out. When a vendor quotes a number, the board tells you whether they quoted the fixed-scaffold run or their own tuned agent, and which version of the task set. And when a new model lands, the board tells you whether it is worth a seat in your own exam. What a leaderboard cannot do is pick the agent for you, because the thing it ranks is not the thing you will run.

The best coding agent is the one whose mistakes you can catch, trace, and afford, and that is a property of the review workflow and the record around the model, not of its score. A benchmark sets the floor on the model. Your reviewer sets the bar on the agent. Build the exam once, put a name on it, and let the leaderboard be an input rather than the decision.

If you want to run the twenty-ticket exam with us, book 30 minutes and bring one ticket you consider hard. We will run it on your repository against the same context your developers use and send back the sheet: mergeable or not, what Proctor flagged, what your reviewer would have changed, and what it cost. The managed version, where that sheet becomes a Team’s track record, is on the home page.