Prerequisites III: Benchmarks

A benchmark is a measuring instrument: a set of tasks, a grading procedure, and an aggregation rule, used to compare systems. Paper 3 exists because instruments can be wrong in ways that mislead silently. This page covers what a benchmark actually measures, the two virtues that trade off against each other (discrimination and realism), the ordering and band checks Paper 3 builds from them, and the two slow failure modes: contamination and rot.

What a benchmark actually measures

A benchmark score is a claim of the form: on this task set, under this grader, this system succeeded at this rate. Every qualifier in that sentence carries weight.

The working rule for this site: a score without its task set, grader, and conditions is not a fact. It is an advertisement.

Discrimination versus realism

Two different virtues, often confused:

A benchmark can have either without the other. Trivially easy tasks copied from production are realistic and discriminate nothing. Synthetic puzzles can discriminate beautifully and predict nothing about production work. Paper 3's anchors test discrimination only, and its authors are explicit that this is a deliberate, limited choice: discrimination can be checked without ground-truth labels, by comparing systems whose relative strength is already known. Realism, in their study, required the expensive thing: human experts auditing a sample of tasks. That audit was small (30 tasks, 6 per cohort, one rater per task, so no inter-rater agreement could be estimated). It rated task quality 2.3 out of 3 on average and found the defect class the anchors could not see. Any benchmark claim should be interrogated on both axes separately.

Capability ordering

A capability ordering is a prior claim about relative strength: these systems, grouped into tiers, should rank in this order on any valid instrument for this task family. Paper 3 builds its tiers from release order and public results, groups near-equals into one tier, and deliberately leaves models within a tier unordered, because adjacent models are often indistinguishable at realistic sample sizes.

The ordering is an assumption, and the paper treats it as one. It is the yardstick the anchors check against, so its justification has to come from outside the benchmark being built: prior releases, public results, and, when reality disagrees persistently, revision of the tiers themselves. In the development cohort (C1), two models tied under three graders plus an execution-based check, and the response was to merge their tier rather than torture the benchmark into separating them. The ordering serves the measurement. The measurement does not exist to flatter the ordering.

Difficulty bands

If every model passes nearly every task, the instrument is saturated: no headroom, no signal. If every model fails nearly everything, it has collapsed to the floor: equally silent. A difficulty band is the informative middle, defined in Paper 3's study as every model's pass rate lying within [0.10, 0.85].

The band does quiet, essential work in that paper's controls. Two broken graders, one scoring self-reports and one grading without a reference, both preserved the capability ordering perfectly (5 of 5 pairs, on a 52-task control set) and were rejected by the band alone, because they saturated at 0.90 to 1.00. Ordering checks ask "is the ranking right?" Band checks ask "is anyone actually being measured?" A serious validation needs both, plus a variance check (repeated runs should roughly agree) when repetitions exist. Paper 3 defined the variance anchor and did not use it, because its study rounds were not repeated. The site flags that rather than smoothing it over, and the cyber version treats the unused anchor as an open problem rather than a solved one.

Contamination

Contamination is evaluation material leaking into the data a system was trained or searched on. Scores inflate; the instrument now measures exposure, not capability. It is the slow failure mode behind all three November papers, and it has several strains:

Defenses are structural: private task sources (Paper 3's tasks come from internal merged changes public models are unlikely to have trained on), pool separation, frozen test sets, and continuously refreshed task supplies, which leads to the last idea.

Live benchmarks

A static benchmark is an artifact, and artifacts rot: tasks leak, models saturate them, the world moves on. The response, exemplified by SWE-bench-Live (Microsoft, 2025, abstract-level), is to replace the artifact with a pipeline: an automated curation system that continuously builds new tasks from recent material (in its case, GitHub issues created since 2024, each with a reproducible environment; 1,319 tasks across 93 repositories at initial release). Models score substantially lower on the live benchmark than on the static one under controlled conditions, which is the contamination tax made visible.

Read Paper 3 as the same move one level deeper. SWE-bench-Live automates task collection. Benchmark Factory automates task collection and validation, with anchors standing in for part of the human curation a live pipeline still needs. And Paper 1 closes the loop from the other side: it ends by calling for dynamic, contamination-proof benchmarks precisely because its self-improving harnesses will otherwise evolve against a fixed, leaking target. The three papers need each other. The synthesis makes that dependency explicit.