Paper 3: Benchmark Factory

Shervin Shahidi, James Poon, Hang Xie, Kenneth O. Xu, Robin Håkanson, Vikram Nair, Jens Pillgram-Larsen, Nachiappan Nagappan (Meta), arXiv:2611.18890, submitted Nov 13, 2026.

Evidence on this page: paper-reported (checked) unless labeled. The complete main text, including Tables I and II and the discussion and limitations, was read in full. The reference list beyond item [6] was truncated in the rendering read, so related works named in the body are reported as named in the text only.

In one breath. Teams deploying coding agents need benchmarks built from their own code. Hand-curation does not scale, and a benchmark that is too easy, too hard, or too noisy silently distorts every comparison made on it. Benchmark Factory turns benchmark construction itself into an agent loop. A meta-agent pulls candidate tasks from an organization's merged code changes, runs models of known relative strength on them, and then accepts, repairs, or rejects each candidate against quality anchors. Anchors are checks, set in advance, on the models' aggregate pass rates; they need no per-task human labels. The grader and the expected model ordering are frozen and are explicitly not available as repairs.

Accepted benchmarks were then tested on task pools, model families, and effort levels that played no part in building them. They recovered known capability orderings out of sample:

Every deliberately broken benchmark and grader was rejected. That holds under a relaxed replay rule the authors fixed in advance, because the unchanged anchors also false-reject the genuine grader on a 52-task set.

The authors frame their own result honestly: anchors are necessary, not sufficient. Their small expert audit (30 tasks, one rater per task) found a defect class, under-specified tasks, that anchors structurally cannot catch.

The problem it names

Public suites like SWE-bench do not cover internal applications, frameworks, and operating systems. Hand-curation with per-task expert verification does not scale to every team and codebase. The obvious automation, generating a benchmark from your change history, creates a failure worse than having no benchmark. An unvalidated instrument misleads silently, and hand-checking is expensive enough that it usually gets skipped.

The authors state the central risk of their own method up front. A benchmark is accepted when the very models that define the expected ordering line up on it, and the loop repairs it until they do. So passing could reflect those models' quirks rather than coding capability. Everything in the evaluation design (the two hypotheses below) exists to bound that risk. Acceptance alone would be circular, and the authors say so in their introduction.

The mechanism

The artifact

A benchmark has four parts:

A task is one landed change. The agent gets the change's title and summary and works at its parent commit. If the change shipped tests, the task is SWE-bench-style and the tests grade the patch. Otherwise an LLM grader compares the agent's patch to the landed patch.

In the five studied cohorts, no fail-to-pass tests were recorded. Grading was therefore done by an LLM grader (GPT-5.5) scoring each patch against the landed patch, 0 to 100. Because the tasks come from private repositories, public models are unlikely to have seen the landed patches in training. That is a structural defense against contamination (see Benchmarks).

The loop

A driver invokes the meta-agent in a fresh context each iteration. The agent performs one step and keeps its state in a file, so long runs do not degrade. The paper names this the "Ralph" loop, after Huntley's write-up. Each candidate moves through three phases:

  1. Research. Explore production data, then write and validate the sourcing query and evaluation configuration. Research also splits the candidate, by hashing task identifiers, into a larger hill-climb subset and a 20% holdout.
  2. Launch. One run per model under test.
  3. Analyze. Separate infrastructure failures from genuine ones and check the anchors. Then accept the candidate, send it to repair, or reject it after K rounds and hand it to a human. The human decides which of three things is wrong: the domain does not separate the models, the pipeline is faulty, or the expected ordering is wrong.

Study parameters: rejection after K = 5 rounds; a second consecutive threshold drop required human approval; a human reviewed every acceptance.

One round is roughly 4 models times 120 tasks, about 480 agent runs, and finishes within a few hours depending on concurrency. The meta-agent itself takes only a few iterations per round. The cost is the agents under test doing the tasks. The paper reports cost in agent runs, not in tokens or compute.

The anchors

Anchors are checks on the models' pass rates, fixed before the runs. They are label-free in the sense that they need no per-task human labels. The tiers and checkpoints themselves are human inputs, and the paper says so.

The study ladder put Opus 4.7 in the top tier, {Opus 4.6, Sonnet 4.6} in the middle, and Opus 4.5 at the bottom. Cohort C5, built a month later, used Opus 4.8, then Opus 4.7, then {Opus 4.6, Sonnet 4.6}.

Benchmark sizes:

The holdout discipline

The holdout runs every round, and the meta-agent sees its results. But its membership is frozen by a hash that is re-checked every iteration. By rule it never gates acceptance and never drives a repair. It is reported at acceptance as a regression check. If a repair re-sources the whole pool, a new holdout may be drawn. C2 and C3 kept one holdout throughout, C1 re-drew once, and C4 re-drew three times.

The holdouts behaved. On these small holdouts (21 to 24 graded tasks), the top tier stayed above the bottom in every recorded case. C2 kept the full ordering. C3 had Sonnet 4.6 tie the top model. In C4, one pair fell 0.002 short of the margin.

The repair order, and what is NOT a repair

Repairs are sequenced by ascending power to tailor the benchmark to the models:

  1. Rewrite the sourcing query (toward larger or more recent changes, for example).
  2. Change task presentation (the prompt).
  3. Fix the pipeline (infrastructure failures go straight here).
  4. Inspect tasks that every model passes or every model fails. Drop a task only for a defect in the task, and encode the exclusion as a sourcing rule, so the population changes by a stated rule, not by cherry-picking.
  5. Move the pass threshold, last.

And the hard rule: the grader and the expected ordering are not repairs. They change only on evidence from outside the candidate (an execution-based check, for example), and are then frozen. Otherwise the loop could satisfy its anchors by tuning the measurement.

What the loop actually did

Five benchmarks were accepted and one candidate was rejected. Humans intervened in C1, C4, and C5. Four of the five accepted benchmarks needed repair. Across cohorts, the loop used 8 sourcing changes, 5 threshold changes, 3 infrastructure fixes, and 1 prompt change. C1 added 3 grader and ordering changes on top of that. C1 was the development cohort: it ran 6 rounds, past the five-round limit, under human supervision while the rules themselves were being revised.

Two cohort stories show the machinery working.

C2, the clean repair. In round 0, Opus 4.7 passed 70% and Sonnet 4.6 passed 68% (p = 0.66), far inside the margin. The meta-agent rewrote the sourcing query toward larger changes (3 to 6 files, at least 50 lines). That separated the two models (67% against 59%) by replacing 81 of the 96 tasks under a stated criterion, rather than re-scoring the same tasks. C2 was accepted after two rounds, about 960 agent runs. (C3, for contrast, took five rounds, about 2,400 runs.)

C5, the rejection. At first, no worker could check out all of C5's repositories, so the candidate was narrowed to five projects. It then kept the bottom-tier model above a middle-tier one across two rounds and on its holdout (combined p = 0.049, reported as descriptive). The inversion persisted, and the candidate was rejected as an operational decision. Once infrastructure covered the full population, a fresh attempt on the new dataset passed without repair. But it used updated tiers (Opus 4.8 in, Opus 4.5 out), so the inverted pair was never re-tested. Report C5 honestly: the rejection mechanism worked, and the follow-up did not re-examine the original anomaly.

Where threshold changes occurred (C3, C4), they served only the band. The ordering already held at the original threshold of 60, but the top model sat above the 0.85 ceiling. Raising the threshold to 75 lowered every model's rate and fixed only that. C3 confirmed on a fresh round. C4 re-thresholded stored grader scores and confirmed on a fresh holdout run.

H1: do accepted benchmarks behave out of sample?

After acceptance, each benchmark definition was frozen. A new pool of 42 to 150 tasks per cohort was drawn independently of the construction sets. By chance, 2 tasks in C2 and 58 in C4 overlapped with construction, and results are reported with and without them.

Two factors that played no part in construction were varied:

The tests were paired exact McNemar tests, one-sided, with directions fixed before the runs, plus cohort-stratified bootstrap 95% confidence intervals from 10,000 resamples. Runs producing no output counted as failures: 10% of GPT-5.4 runs, 7% of GPT-5.5 runs, and 12 to 13% at every effort setting.

Robustness checks the authors report:

Which leg bears the weight. The two contrasts are not equally strong, and the draft of this page did not say so.

The effort contrast is the robust leg. It is positive in every cohort, survives every reported cut, and is free of the grader self-preference confound, because both arms are the same Claude model.

The model-family contrast is real in the pool but fragile. It drops to p = 0.07 when a single cohort (C5) is left out. It has one contrary cell: GPT-5.5 trails in C3 by -4.8 points, a small (n = 42), near-saturated slice where four tasks separate the models (two-sided p = 0.63) and mean grader score slightly favors GPT-5.5. And it is the contrast exposed to self-preference: the study's grader is GPT-5.5, which the authors flag themselves.

The adversarial panel described the p = 0.07 as a Bonferroni-corrected result. It is the leave-one-cohort-out figure, and both pooled contrasts do survive Bonferroni. The authors state plainly that per-cohort samples are underpowered and that H1 rests on the pooled contrasts, all from one organization. The abstract's "9 of the 10 per-cohort comparisons pointed the same way" counts 4 of 5 for GPT plus 5 of 5 for effort.

Two further results matter.

Extra-high effort gives no gain over high (-2.2 points, not significant), despite 20% more turns and 31% more output tokens. That is a clean negative result about effort scaling on this task family.

The grader moves the level, not the order. This finding comes from two companion benchmarks built by the same loop for another organization: 105 and 150 tasks, three runs per setting, three graders.

Grader family moves the level, not the order. That is why thresholds must be re-calibrated when the grader changes, and why the grader is frozen with the benchmark.

H2: do the anchors reject broken benchmarks and graders?

All controls ran on re-drawn task sets. The grader controls used n = 52, from an October 2026 re-run of C2 with Opus 4.8 as the top tier. The trivial-changes control used n = 57. Every broken setup was rejected, under the replay rule described below. The details are where the value is, including the paper's own blind spots.

What H2 does not test. Every control above is a fixed broken setup, built by the authors and run once through the anchors. None is an optimizer actively searching for a benchmark or grader that passes the anchors while measuring the wrong thing. The t = 94 blind spot hints at what such a search could find. The adversarial panel's demand here is valid: run an arm that explicitly optimizes against the anchors, and report the false-accept rate as a function of δ. Until then, H2 shows that the anchors reject known kinds of breakage, not that they resist a motivated adversary.

The expert audit: the failure class anchors cannot catch

The audit is small, and every conclusion drawn from it carries that condition. It sampled 5% of the roughly 600 pooled tasks: 30 tasks (6 per cohort), re-run with Opus 4.8 and a bottom-tier model. Three Meta engineering experts did the rating, with one rater per task, so inter-rater agreement cannot be estimated. Raters saw the task, both patches, model names, and the grader's reasoning.

The failure analysis is singular and actionable: no PASS verdict was rated wrong, and all five wrong verdicts were FAILs on under-specified tasks. In each case, the description omitted details (enum values, item priorities) that the landed patch contained, so the grader penalized the agent for details the task never specified. Most arguable verdicts follow the same pattern. Four of the 6 poor tasks were unclear or under-specified, and the other 2 were over-specified.

The authors note that the anchors cannot catch this class: under-specification lowers every model's pass rate alike and leaves ordering and band intact. It needs a sourcing filter that removes or rewrites under-specified tasks, which they list as future work. Necessary, not sufficient, demonstrated on the authors' own accepted output.

On a sample of 30 with one rater, read the pattern as a strong hint, not a measured rate. (The adversarial panel read the 53% lower bound as "barely better than chance". That misreads the categories. Most of the remainder is "arguable", not wrong.) The panel's proposed fix is a good one: a blinded audit with at least 3 raters and at least 15 tasks per cohort, reporting agreement.

The incident that taught the loop

In the development cohort, Opus 4.5 scored 0% because its model identifier was invalid. That was an infrastructure failure, not a capability measurement. The band anchor caught the anomaly. But the meta-agent misread it as low capability and lowered the threshold from 80 to 60. The ladder compressed (three rates within 0.012), and an acceptance made before the gate checked for compression was retracted. Four rules followed, and they generalize to any loop that reads scores:

  1. Check run health before reading a score: turns, tool calls, tokens. A zero is a pipeline symptom until proven otherwise.
  2. Order repairs by their power to fit the measurement, threshold last.
  3. Freeze the grader and the expected ordering unless evidence from outside the candidate justifies a change.
  4. Keep a human checkpoint at every acceptance and rejection.

Capability-alignment bias. Anchors constrain aggregate pass rates. Individual tasks where a weaker model beats a stronger one are therefore retained, because repairs remove tasks only for defects. That is the opposite failure from item-flaw detection systems.

Tier merges. When C1 showed Sonnet 4.6 matching Opus 4.6 under three graders plus an execution-based check, the response was to merge their tier, not to re-source the benchmark.

A use-case demo (illustrative, labeled so by the authors). A 40-task benchmark for comparing two Claude Code skills was built in one round, within an hour, with a grader that reads trajectories. Baseline pass rates were Opus 4.8 76%, Opus 4.7 62%, and Opus 4.6 and Sonnet 4.6 54%. The benchmark and grader were fixed before any run. Two skills were tested:

It is a demonstration, not part of the H1 or H2 evidence.

Limitations and what NOT to conclude

The authors concede the following:

Future work they name: harder and higher-quality sourcing; filtering or rewriting under-specified tasks; and grading that separates useful extra work from scope drift. On that last point, the companion effort study found higher effort produced larger-than-reference patches that a scope check penalized. That is right when the extra changes are unwarranted, and unfair when they are useful work the reference lacks.

Three limits from the adversarial read, checked against the paper's numbers:

What not to conclude: