The Benchmark Ends Too Soon

A leaderboard asks, “Did the system pass this test?”

A CTO asks, “Did the system remove work without creating worse work somewhere else?”

Those are different questions.

Benchmarks are indispensable because they hold enough things fixed to compare systems. They become dangerous when a score escapes its denominator and turns into a claim about autonomy, productivity, or return on investment.

The benchmark usually ends at a patch and a test result. The product continues through review, integration, deployment, observation, maintenance, incident risk, and user value.

A score is a conditional statement

Write every result in full:

system S achieved metric M
on task distribution D
with environment E
under harness H
using budget B
scored by verifier V
at version and date T

Remove any term and interpretation widens.

SWE-bench made software-agent evaluation materially more realistic by pairing issue descriptions with real repositories and asking systems to produce resolving changes. That is a major improvement over isolated function synthesis. It still samples a specific historical world: selected repositories, issues, repository snapshots, test patches, environment construction, and a pass/fail resolution rule.

A system that resolves 60 percent of one benchmark has not proven it can resolve 60 percent of your backlog. Your distribution may contain migrations, UI judgment, private infrastructure, weak specifications, security constraints, approvals, flaky integration environments, or work whose correct answer is “do not change code.”

Benchmarks compress failure into one bit

A failed task can mean:

• the issue was misunderstood;
• the relevant files were not found;
• retrieved evidence was stale;
• a correct plan was executed incorrectly;
• the patch solved the visible test and broke an invariant;
• the harness failed to install dependencies;
• the agent exhausted budget one step before success;
• the verifier rejected good work;
• the test accepted an overfit patch;
• the task itself was ambiguous or infeasible.

These failures demand different fixes. A larger model may help semantic understanding but do nothing for a broken environment. More retries may rescue stochastic search but worsen an incorrect verifier. A new tool may improve navigation while expanding the attack surface.

Store the terminal score and the failure stage:

task_validity
environment_ready
localized
plan_admissible
mutation_valid
tests_pass
policy_pass
review_pass
integration_pass
value_observed

This turns evaluation into engineering. The score says whether the system improved. The taxonomy says what to improve next.

The verifier defines the product you optimize

If the only verifier is a test suite, the agent is trained and tuned to pass tests. Tests can be excellent executable specifications. They can also be incomplete, flaky, coupled to implementation details, or vulnerable to shortcuts.

For repository changes, use a verifier stack:

1. patch applies to the declared base revision;
2. changed paths and generated artifacts obey policy;
3. focused tests pass;
4. full relevant tests, type checks, lint, and build pass;
5. new behavior has direct tests and old behavior does not regress;
6. security and dependency policies pass;
7. the diff matches issue scope and repository conventions;
8. integration or preview evidence passes;
9. a human or independent system accepts high-consequence changes;
10. post-deployment outcomes remain healthy.

The HELM paper argues for transparent, multi-metric, multi-scenario evaluation rather than reducing model quality to one notion. Agent evaluation needs the same discipline at the system level. Accuracy, calibration, robustness, fairness, efficiency, and risk do not collapse honestly into one number without an explicit utility function.

Judge disagreement is a measurement, not a nuisance

Many outputs have no cheap exact-match scorer. Teams use model judges for design quality, support, relevance, style, or adherence to a natural-language requirement. This can scale evaluation, but it moves uncertainty into another model.

Zheng et al. study large models as judges for chat assistants and discuss agreement with human preferences as well as position, verbosity, and self-enhancement biases. Those findings are task and judge specific. The broader control rule is durable: a judge must be evaluated like the system it scores.

Build a calibration set with independent expert labels and difficult disagreements:

clear accept
clear reject
partially correct
correct but out of scope
plausible and unsupported
test-passing shortcut
safe refusal
unnecessary refusal

Then measure:

• false accept rate, especially for high-consequence slices;
• false reject rate and its extra inference or human cost;
• agreement by task type, language, length, and answer position;
• stability under harmless formatting and candidate order changes;
• sensitivity to judge prompt and model version;
• correlation with later executable or production evidence.

Do not resolve every disagreement by majority vote. Three correlated judges can repeat the same blind spot. Prefer heterogeneous evidence: executable checks for behavior, static rules for policy, grounded comparison for requirements, and human review for ambiguous consequence.

Keep the raw judge decision, confidence or rubric scores, cited evidence, and later adjudication. A changed human label should not overwrite history. It becomes a new event that can recalibrate the judge.

Stop the evaluator from seeing the answer key

Evaluation harnesses leak in subtle ways. A tool may expose hidden tests. A repository may contain the future patch in history. A file name may encode the target. A judge prompt may include the reference answer before asking for independent analysis. An agent may infer that network failure is part of the benchmark and exploit the harness rather than solve the task.

Threat-model the evaluator:

Leakage path	Control
Hidden tests readable in sandbox	Separate verifier boundary and credentials
Future commits in Git objects	Construct a clean repository snapshot
Reference patch visible to judge	Blind candidate review before comparison
Task IDs map to public solutions	Re-key tasks and use private forward sets
Harness errors counted as failures	Separate environment validity from agent outcome
Repeated submissions reveal tests	Limit feedback and rotate hidden checks

An evaluation should fail closed when the environment is invalid, but it should not silently count infrastructure errors as model incapability. Record invalid_environment, repair the task, and rerun all compared systems under the same version.

Evaluation itself needs regression tests

Changing a container image, dependency resolver, timeout, test command, network policy, or parser can move scores without changing the agent. Treat the harness as production software:

1. pin dependency and image digests;
2. include known-pass and known-fail sentinel tasks;
3. run the same candidate repeatedly to estimate harness flakiness;
4. hash task inputs, expected evidence, and scoring code;
5. version environment repairs rather than editing history;
6. replay a stable control system on every harness release;
7. block comparisons across incompatible versions or label them clearly.

Suppose a 200-task suite has a 3 percent independent harness-failure probability per run. The expected number of invalid executions is six:

200 × 0.03 = 6

If the measured system difference is four tasks, environment noise can dominate the claimed improvement. Repetition, paired runs, and explicit invalid-task handling matter more than an extra decimal place on the leaderboard.

Contamination changes what a benchmark measures

Public benchmark tasks, solutions, discussions, tests, and leaderboards can enter training or tuning data. Contamination does not imply every high score is memorization. It weakens the assumption that the test set is unseen.

LiveBench was designed around frequently updated questions from recent sources and objective scoring to limit contamination. A 2025 ICML study of contamination mitigation reports that benchmark contamination can inflate estimates and examines mitigation strategies. The practical lesson is not “discard benchmarks.” It is “record exposure risk and add private, time-split, and mutation-based evaluations.”

Use at least four task pools:

• public anchor: comparable to published work;
• private replay: historical tasks and incidents never made public;
• fresh forward set: tasks created after the evaluated system version was fixed;
• generated mutations: controlled variants that test mechanisms rather than memorized answers.

Keep the private set private. Repeated internal tuning contaminates it too. Rotate holdouts and record which experiments viewed which results.

Offline success does not equal developer productivity

In July 2025, Becker et al. reported a randomized controlled trial with 16 experienced open-source developers completing 246 tasks in repositories they knew. In that specific setting with early-2025 tools, allowing AI increased measured completion time by 19 percent even though participants expected a speedup. The authors explicitly scope the result to the tools, developers, repositories, and period studied. The paper should not be generalized into “AI always slows developers.”

It is valuable because it demonstrates a measurement gap. Benchmark capability, perceived speed, and observed task time can disagree.

The researchers later described selection problems in a follow-up experiment because some developers did not want to participate when tasks might prohibit AI. That February 2026 update is another important lesson: even a well-designed field experiment can lose identifiability when adoption changes who agrees to enter the control condition.

Production evidence needs multiple designs:

• randomized task-level rollout where feasible;
• stepped-wedge rollout across comparable teams;
• matched historical comparison with declared confounders;
• interrupted time series for stable processes;
• within-task shadow execution without affecting humans;
• qualitative review to explain metric movement.

Never report an observational before/after chart as causal without defending what else changed.

Evaluate the system, not just the model

An agent result depends on:

model
prompt and context policy
tool interfaces
repository and environment
loop controller
budgets and stop rules
candidate selection
verifiers
human interaction

Run ablations. Change one layer at a time. A model upgrade tested alongside a new prompt, longer budget, different tool, and repaired environment cannot attribute the gain.

At minimum compare:

Variant	Question
Same system, new model	Does inference quality change outcomes?
Same model, new interface	Did tools or context unlock capability?
Same system, larger budget	Is the gain bought by more search?
Same candidates, new verifier	Is selection the bottleneck?
Agent versus simple pipeline	Does autonomy earn its complexity?
Agent versus human baseline	Does it remove time, cost, or delay?

Report confidence intervals or uncertainty bands when sample sizes allow. For rare high-consequence failures, absence in a small sample is not evidence of safety.

Build slices before averages

An aggregate 70 percent success rate can conceal a deployment decision:

Task slice	Volume	Success	Mean cost	Review minutes
Dependency bumps	400	92%	$0.80	3
Local bug fixes	300	78%	$2.40	9
Cross-service changes	200	38%	$8.90	31
Data migrations	100	12%	$19.00	74

These values are illustrative. Weighted success is:

(400×.92 + 300×.78 + 200×.38 + 100×.12) / 1,000 = 69%

The average suggests “almost seven in ten.” The slices suggest autonomous dependency updates, assisted local fixes, and human-led migrations.

Useful slices include task type, repository age, languages, dependency graph size, issue clarity, changed-file count, consequence, tool path, context size, model, workflow version, and whether the task required hidden organizational knowledge.

The unit of evaluation is the verified outcome

For each slice calculate:

verified success rate
on-time verified success rate
cost per verified success
human minutes per accepted outcome
human minutes per exception
rework within 7, 30, and 90 days
rollback or incident rate
change lead time
abandoned compute and queue cost

A system can increase patch volume while reducing delivery throughput if review and rework grow faster. It can reduce developer typing while increasing waiting. It can pass more tasks by spending ten times the inference budget.

Make the economic comparison explicit:

automation value
= baseline labor and delay avoided
- inference, tools, infrastructure, and verification
- human steering and exception handling
- rework and failure loss
- migration and operating burden

Do not compare salary to token price. Compare one terminal outcome under both operating models.

An evaluation ladder for deployment

Move upward only when the lower evidence remains healthy:

1. Deterministic contract tests: schemas, policies, idempotency, state reducers.
2. Component evals: retrieval, routing, structured output, verifier calibration.
3. Task replay: fixed historical environments with full traces.
4. Adversarial slices: ambiguity, stale evidence, permission denial, tool failure, long context.
5. Fresh private holdout: unseen tasks frozen before experiment.
6. Shadow production: agent runs without authority; compare proposal with actual outcome.
7. Canary delegation: reversible low-consequence tasks under strict budgets.
8. Controlled expansion: randomized or phased rollout with baseline measurement.
9. Autonomous route: only slices with stable outcome, cost, and exception evidence.
10. Continuous audit: drift, contamination, new failure classes, and post-deployment rework.

Every stage needs promotion and rollback criteria written before results arrive.

Worked decision: the cheaper system that should not ship

Assume 1,000 monthly tasks. Two systems are evaluated on a representative private replay:

Metric	System A	System B
Raw benchmark pass	74%	68%
Production verifier pass	60%	64%
Mean run cost	$1.20	$1.80
Human review per candidate	8 min	5 min
Accepted outcomes	600	640

At $60 per human hour:

A run cost = 1,000 × $1.20 = $1,200
A review cost = 1,000 × 8/60 × $60 = $8,000
A total / accepted = $9,200 / 600 = $15.33

B run cost = 1,000 × $1.80 = $1,800
B review cost = 1,000 × 5/60 × $60 = $5,000
B total / accepted = $6,800 / 640 = $10.63

System A wins the raw benchmark and loses the outcome economics. The example is illustrative, but the decision pattern is common: a stronger headline score can produce harder-to-review work.

Builder playbook

1. Write the production decision the evaluation must support.
2. Define the target workload and sample it before selecting metrics.
3. State the full score condition: tasks, environment, harness, budget, verifier, version, date.
4. Build a failure taxonomy and label terminal traces.
5. Use public anchors, private replays, fresh forward tasks, and controlled mutations.
6. Evaluate the full system and run layer-by-layer ablations.
7. Report slice metrics, uncertainty, exclusions, and cost.
8. Compare against the current operating model, including review and rework.
9. Pre-register promotion, pause, and rollback thresholds.
10. Keep measuring after deployment because the benchmark ends too soon.

The best benchmark is not the one with the hardest tasks or the biggest leaderboard.

It is the smallest honest instrument that predicts the decision you are about to make.

References

• Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” ICLR 2024. The primary benchmark paper for repository-level issue resolution.
• Liang et al., “Holistic Evaluation of Language Models,” 2022 preprint and HELM project. A transparent, multi-scenario and multi-metric evaluation framework.
• White et al., “LiveBench,” ICLR 2025. A frequently updated, objective benchmark designed to limit contamination.
• Sun et al., “The Emperor's New Clothes in Benchmarking?” ICML 2025. A peer-reviewed study of benchmark contamination and mitigation strategies.
• Becker et al., “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” 2025 preprint. A randomized field study whose scoped result differs from participant forecasts.
• METR, “We are Changing our Developer Productivity Experiment Design,” February 2026. A first-party disclosure of selection effects that complicated a follow-up productivity experiment.
• Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” 2023 preprint. Primary work on model-judge agreement and bias modes.