The Build Book: Seven Builds
Seven projects. Each one is the shared skeleton pointed at a program that already exists: a loop edits an artifact, each round starts from fresh context, state lives in files, and a frozen check the loop cannot edit decides what survives. None of them needs a new platform.
Every build names its frozen check before its loop, because the check is what makes the loop safe to run unattended.
How to read this page:
- Program design. Everything here is program design, taken from the build book draft of Nov 20, 2026. It is a plan, not a set of results.
- Times. Times are agent wall-clock hours for one dedicated agent.
- Compute floors. Evaluation runs, training runs, and inference at scale are listed separately. More agents do not shrink them, and most of them run on the M5 Max.
- Assumptions. Thresholds marked assumption are starting values to tune with data. They are not findings.
- Reviewer notes. Paragraphs labeled Reviewer note are editorial design suggestions added in review. They are not part of the draft and say nothing about how any program works today.
The theory behind each build is on the cyber version page. The doctrine is in the synthesis.
Build 1: Gym Anchor CI
Validates tasks and graders for the cyber range, run as continuous integration on every new task. Draws on Paper 3.
The idea. Every task entering the gym, authored or generated, passes an automated validation gate before anything is scored on it. The gate does four things:
- runs the task against a pinned ladder of solver configurations of known relative strength
- checks label-free anchors
- runs negative controls
- sends failures to a bounded repair loop (three rounds), then to a human
Every task that passes carries a validation record. So every score the gym publishes traces back to a grader that was itself tested.
What changes in transfer. Benchmark Factory sources tasks from merged code changes and validates them against model strength orderings on coding tasks. The gym sources tasks from authors and generators, and validates them against a pinned attacker ladder on cyber tasks. Ground truth there is exploit success or fix survival, not test passage.
The anchors carry over unchanged because they never depended on what the task is about. They need only solvers of known relative strength, and the gym already keeps those as its pinned ladder. Ladder composition is a gym program decision: this build uses the ladder, it does not define it.
Architecture.
- Task intake. A folder convention: task definition, environment spec, grader code, and manifest. Intake watches the gym repo for new or changed task folders.
- Ladder runner. Inspect AI jobs run each task against each rung of the pinned attacker ladder, weakest to strongest. Runs are sandboxed in the range, on the M5 Max.
- Anchor checker. A deterministic program, not a model. It computes ordering, margins, difficulty, and variance, and reports pass or fail per anchor with the numbers attached.
- Control suite. Per task, it builds three kinds of control:
- a trivial variant with the solution path pre-opened
- broken grader variants: always-pass, then always-fail
- a label-shuffled scoring run
If a control passes validation, the validator itself is flagged as broken and the task is held. - Repair loop. An agent gets the failing anchor, the ladder results, and the task folder, and may edit the task or the grader. The anchor checker and the ladder are outside its edit reach, enforced by directory permissions. - Validation ledger. One file record per task per validation run: anchors checked, numbers observed, controls run, repairs attempted, and verdict. This is the audit trail behind any published score.
Inputs and outputs.
- Inputs: task folders, the pinned ladder definition, and anchor thresholds. The thresholds live in a configuration file that humans own.
- Outputs:
- validated tasks in the gym index
- a ledger entry for each task
- escalation reports for tasks that run out of repair rounds
- a weekly roll-up of anchor failure modes: which anchors fail most often, and which task families produce broken graders
The frozen check.
- Ordering anchor. Ladder rungs order monotonically on task success rate, and the top rung beats the bottom rung by at least 25 percentage points.
- Assumption: 25 points is a starting margin, tuned after the first 50 tasks.
- Paper 3 used δ = 0.05 for about 100-task benchmarks of one task family. That is a policy setting, not a universal number.
- Difficulty band. Success across the full ladder falls between 5% and 95%.
- Variance bound. Five repeated runs of the middle rung agree within a standard deviation of 10 points (assumption).
- Negative controls.
- The trivial variant must fail the band.
- The always-pass and always-fail graders must fail the ordering anchor.
- A label-shuffled run may reproduce the correct ordering in no more than 5% of simulations.
- For comparison, Paper 3 measured at most 1.4% false accepts within five attempts. That figure applies only under its conditions: δ = 0.05, 96-task draws, 20,000 simulations, no repairs. The local 5% gate is looser on purpose, and the local ladder, margin, and task count are different, so the two numbers are not directly comparable.
- Grader freeze. The grader under validation is frozen during its own validation. A proposed new grader restarts validation from zero.
Build sequence and acceptance. About 30 agent-hours:
| Step | Agent-hours |
|---|---|
| Manifest schema | 3 |
| Ladder runner on Inspect AI, with range sandbox wiring | 6 |
| Anchor checker, unit-tested on synthetic result sets | 5 |
| Control generators | 5 |
| Repair loop with permission-scoped edits | 4 |
| Ledger and roll-up | 3 |
| End-to-end trial on 20 existing tasks, with threshold calibration | 4 |
Compute floor: the trial is an overnight M5 Max range run, roughly 8 to 12 machine-hours depending on ladder size. It recurs with every validation batch.
Acceptance:
- On 10 seeded known-bad tasks, the gate rejects at least 9 and escalates none silently.
- On 20 existing human-reviewed tasks, it passes at least 90% without repair, and every repair it makes shows in the ledger.
- A deliberately corrupted anchor checker is caught by the control suite within one run.
- Every validated task links to a complete ledger entry.
The risk. Anchors are necessary, not sufficient. A task can order the ladder perfectly and still be unrealistic or teach the wrong skill: the gate certifies discrimination, not quality. Human review stays in place for new task families, though not for new tasks within known families. The weekly roll-up exists so that gaming patterns surface early.
There is a second risk: the repair loop may learn to write tasks that please the anchors instead of tasks that test the skill. If repair success rates climb toward 100%, treat that as a warning, not a success.
Reviewer note. Paper 3 defined its variance anchor but never exercised it, because its rounds were not repeated. So the 10-point variance bound has no prior evidence behind it from the source paper. Calibrating it is original work. Log the observed spread of every repeated run from day one, so the bound can be set from data rather than defended from intuition.
Where it lives. The cyber range and gym program. It sits in front of the train-open task set and, with stricter thresholds, the holdout-private set. It feeds Proof of Defense: PoD tasks that passed Anchor CI carry their ledger entries into the published methodology.
Build 2: Retester Loop
A self-improving attacker harness for the range, with a SIM-style edit memory. Draws on Paper 1 and Paper 2.
The idea. The range's house attacker harness improves itself under discipline:
- one module at a time
- every proposed edit verified against a frozen evaluation set before it is kept
- every edit, kept or rejected, recorded with its rationale, predicted effect, and observed effect
The memory becomes a written record of what works against our own defenses. That record is dual-use. The edits that most improved the attacker are a prioritized list of what the Blue and Green sides must answer.
Full rewrites are forbidden. The rationale comes from Paper 1's Table IV, which is one run on 40 held-out tasks:
- The unconstrained full rewrite used 451.6k runtime tokens per task. The gated modular harness used 46.7k.
- The accuracy difference was one task: 17/40 versus 16/40 combined.
- The gated harness was ahead in-domain (8/20 versus 7/20). The rewrite was ahead on the cross-repository set (10/20 versus 8/20).
- The rewrite also degraded on a stronger model (30.0% versus 37.5% for gated).
So the rewrite bought, at most, a one-task combined edge, with nothing to show which change produced it, at roughly ten times the runtime tokens. That is an argument about cost and attribution, not proof that gating wins on accuracy.
What changes in transfer. The papers improved coding harnesses on coding benchmarks. This build improves an attack harness against defended range systems. The evaluation metric is the gym's own headline pair: attack success rate against defended targets, and cost per success from the attacker's side (tokens and wall time per successful compromise).
Architecture.
- Harness under improvement. The build adopts the harness's existing module boundaries, for example recon, exploit selection, tool use, findings persistence, and reporting. Assumption: the harness is modular enough to edit one piece at a time.
- Frozen evaluation set. Defended range scenarios, versioned and outside the loop's edit reach. Scenarios rotate only by human decision, on a schedule, never by the loop.
- Edit proposer. Reads the edit memory, the last evaluation results, and one target module. It proposes exactly one edit with a written prediction: which metric moves, by roughly how much, and why.
- Verifier. Runs paired evaluations of the edited and unedited harness on the frozen and held-out sets, and applies the keep rule. Runs on the M5 Max in the range sandbox.
- Edit memory. File-backed, using Paper 2's schema. The proposer must cite the records it relied on. Rejected edits stay in the memory, along with their failures.
- Transfer guard. A small held-out scenario set the proposer never sees described. If an edit improves the frozen set but degrades the held-out set, it is rejected and recorded as suspected overfit.
Inputs and outputs.
- Inputs: the attacker harness codebase, the frozen and held-out sets, and an edit budget (tokens and M5 Max hours per day).
- Outputs:
- an improving harness with a full edit lineage
- the edit memory as a research artifact
- a weekly digest of kept edits, rejected edits, and the defense implications of the kept ones, routed to Counterfleet as Red-side intelligence for Blue and Green
The frozen check.
- Keep rule. All three conditions must hold (assumptions: starting thresholds):
- attack success rate on the frozen set improves by at least 3 points
- no scenario family regresses by more than 1 point
- attacker cost per success worsens by no more than 10%
- Held-out guard. Degradation of more than 2 points rejects the edit, regardless of frozen-set gains.
- One module per edit. Enforced by diff inspection before evaluation. An edit that touches two modules is rejected unevaluated.
- Isolation. The evaluation sets and the keep rule live outside the proposer's edit reach. The proposer cannot see what the held-out scenarios contain.
Build sequence and acceptance. About 25 agent-hours:
| Step | Agent-hours |
|---|---|
| Module inventory and edit scoping | 3 |
| Evaluation set assembly from existing range scenarios | 5 |
| Verifier | 5 |
| Edit memory | 3 |
| Proposer with citation requirement | 4 |
| Digest reporting | 2 |
| Supervised first week | 3, plus human review of the first 20 keep and reject decisions before the loop runs unattended |
Compute dominates: two full evaluation runs plus one held-out run per edit, roughly 1 to 3 M5 Max machine-hours per edit depending on scenario size. The edit budget is therefore a compute decision, not an agent-time decision.
Acceptance:
- 30 edit cycles with zero keeps that violate the keep rule, auditable from memory.
- At least 25% of proposals rejected. A loop that keeps everything is verifying nothing.
- Measurable frozen-set improvement after 30 cycles, or a defensible record of why the easy gains were not there.
- The weekly digest produced unprompted for three consecutive weeks.
The risk. Paper 2's negative transfer: lessons from one context can degrade another. In the paper, transplanted harnesses started up to 5.48 points below seed on a new model. The paper offers no detector and no bound. The held-out guard and the per-family regression caps are the local counters. The memory records evaluation context, so future proposers can see where an edit helped and where it hurt.
A second risk is overfitting to the range's defended systems, which would mislead Blue and Green about real-world exposure. The counters: the frozen set rotates by human decision, and the digest states plainly that gains are range-relative.
A third risk is capability drift. A steadily improving house attacker is a sensitive artifact. It lives in the range, on Brandon's machines, under the range program's handling rules.
Reviewer note. The keep rule is a heuristic gate of exactly the class TRUST measured. In TRUST's own study, a score-greedy gate accepted inferior updates 45.59% of the time. That figure does not transfer as a number, but it does transfer as a warning. Paper 1's own validation split was 8 tasks, where one task moves the score by 12.5 points.
Before trusting a 3-point keep threshold, do two things:
- Size the frozen set so that 3 points is several tasks, not one.
- Or require repeated paired runs, and estimate the run-to-run spread of the unedited harness first.
An edit that clears 3 points by less than that spread is noise. Either record it as "not enough evidence," or send it to a second evaluation, rather than keeping it.
Where it lives. The cyber range and gym program, Red side. The digest feeds Counterfleet. The edit memory follows Build 5's schema.
Build 3: PoD Grader Factory
Anchor-validated scoring for Proof of Defense, with negative controls as a publication requirement. Draws on Paper 3.
The idea. Proof of Defense lives or dies on whether outsiders believe the scores. This build manufactures that belief mechanically. Every PoD grader is produced and validated through the Benchmark Factory pattern before it scores a public run, and the validation record ships with the results. Each published number arrives with its grader's test history beside it, so credibility becomes an artifact rather than a claim.
What changes in transfer. PoD graders score defense, not task solving. The known-strength ladder is a ladder of defender configurations, for example: undefended baseline, detection only, detection plus automated containment, and the full fleet. The relative strength of these rungs is established by construction and confirmed by the gym's attacker ladder. The anchor logic is identical to Paper 3's: if the grader cannot rank the ladder correctly, with margin, then the grader is wrong, not the defenders.
Architecture.
- Grader spec format. Each metric (fix survival under re-attack, cost per resolved incident, detection time, containment time) gets a versioned spec owned by a human. The spec says which events count, how they are extracted from range logs, and how the score is computed.
- Grader implementation. Agents build graders from the specs. The graders are then frozen.
- Ladder validation harness. Runs each grader against recorded and live runs of the defender ladder and checks ordering and margins per metric.
- Control battery. Fixed adversarial log sets, each with a known correct verdict.
- Publication packager. Assembles each release: results, grader versions, validation records, control results, and ladder definitions. Nothing ships without the full pack. The public methodology note is generated from the records, not written from memory.
Inputs and outputs.
- Inputs: metric specs, range run logs, the defender ladder, and the attacker ladder (which supplies re-attack verdicts).
- Outputs: registered frozen graders, validation records, publication packages, and the generated methodology note.
The frozen check.
- Ordering anchor per metric. The grader ranks the defender ladder in the constructed order on at least 90% of scored runs. The top and bottom rungs must be separated by a margin that is pre-registered in the spec before validation runs, never set after seeing results.
- Control battery. 100% pass required. This is the strictest gate in this book, deliberately so: one failed control blocks publication. The battery checks three things:
- an always-success grader must fail ordering
- shuffled defender labels must produce near-random rankings
- a run whose fix demonstrably fails re-attack must not be scored as survival
- Shuffle control. With defender labels shuffled, the grader's ranking may match the true ranking in no more than 5% of simulations.
- Stability. Re-scoring the same logs must produce identical scores. Any nondeterminism in a grader is a defect.
Build sequence and acceptance. About 29 agent-hours:
| Step | Agent-hours |
|---|---|
| Spec format and the first two metric specs (fix survival, cost per resolved incident) | 5, plus Brandon's review |
| Grader implementations | 6 |
| Ladder validation harness | 4 |
| Control battery, including the adversarial log sets | 6 |
| Publication packager and methodology generator | 4 |
| First full rehearsal | 4 |
Compute floor: roughly one overnight M5 Max range run per grader version.
Acceptance:
- Both initial graders pass the full battery.
- A grader intentionally seeded with a scoring bug is caught before registration. This must be demonstrated on purpose at least once.
- A reader who was not involved can trace every published number to a grader version and validation record in under ten minutes.
- Re-scoring is bit-identical across three runs.
The risk. The main risk is silent misdirection: a rigorous-looking grader that ranks things wrongly. For PoD, one wrong public ranking costs more than publishing nothing, which is why the battery is a hard gate rather than a report.
The second risk is ladder construction bias. If the defender ladder's true order differs from the constructed order, the anchors will punish correct graders. Ladder order is therefore confirmed independently by the gym's attacker ladder before validation starts. Disagreements between the two ladders are investigated as findings, not smoothed over.
Reviewer note. The control battery tests fixed, known-bad inputs. Paper 3's equivalent controls did the same, and its own admitted blind spot shows the limit of that approach: a self-report grader passes at exactly t = 94. A fixed battery cannot find the input nobody thought to write.
Before the first public release, schedule one adversarial pass. Give a Red-side agent the grader spec and the goal "produce logs this grader scores as a surviving fix when the fix fails re-attack." Then add every log set it finds to the battery. This is the gate-gaming test that no paper in the family ran (cyber version, open problem 7).
Where it lives. Counterfleet, as the scoring backbone of Proof of Defense. It consumes the gym's ladders and range logs and shares validation machinery with Build 1.
Build 4: Orange Task Foundry
Orange's threat-to-task pipeline, rebuilt as a Benchmark Factory loop over disclosed vulnerabilities. Draws on Paper 3. Depends on Build 1.
The idea. Orange turns disclosed threats into range training tasks, and today that is a craft process. The foundry turns it into a factory with a quality gate:
- For each disclosure, an agent drafts the complete task.
- A reproducibility check runs before any validation.
- Anchor CI (Build 1) gates entry.
- Failures drive a bounded repair loop.
Output stops being limited by authoring time and starts being limited by validation compute. That is the correct bottleneck, because an invalid task teaches a wrong lesson at scale.
What changes in transfer. Benchmark Factory's agent wrote queries over landed code changes. Orange's agent drafts tasks from disclosure feeds: advisories, CVE records, write-ups, and patch diffs where they are public. That source material is messier and its ground truth is thinner. A merged change carries its own context, while a disclosure carries prose of varying honesty. So the foundry adds one component the paper did not need: a reproducibility check. The drafted exploit path must actually execute in the range before anchor validation begins.
Architecture.
- Disclosure intake. Orange's existing feeds and watchlists, normalized into a candidate queue with source links and disclosure dates.
- Task drafter. Reads a disclosure and produces a complete gym-format task folder: environment build, vulnerable configuration, exploit sketch, grader, and a manifest with provenance (disclosure source, drafter model, date).
- Reproducibility runner. Builds the environment in the range, executes the exploit path, and confirms that the vulnerable state is reachable and that the grader observes it. The verdict is binary. A task whose exploit does not run is a broken task, not a hard one, and it never reaches the anchors.
- Anchor CI. Build 1, unmodified. The foundry is a client of the gate, not a second gate.
- Repair loop. Structured diagnoses from either the reproducibility runner or Anchor CI go back to the drafter. After three rounds, the candidate is parked with a written reason and stays visible for human salvage.
- Provenance ledger hook. Every shipped task records its disclosure source and validation history in the LAB2 provenance ledger format.
Inputs and outputs.
- Inputs: disclosure feeds, the gym task format, Anchor CI, and range build capacity.
- Outputs:
- validated tasks, routed to the train-open or holdout-private set by Orange's routing decision
- a list of parked candidates, each with its reason
- a quality statistic for each disclosure source
The frozen check.
- Reproducibility. Binary and non-negotiable.
- Anchor thresholds. Anchor CI thresholds apply unchanged. The foundry gets no relaxed gate for volume. If throughput suffers, the answer is more validation compute, not weaker anchors.
- Provenance completeness. Enforced at dataset build time. A task with no source link or no validation record cannot enter any training set.
- Fixed vulnerability. The repair loop may not change the vulnerability being modeled, only its environment and grader.
Build sequence and acceptance. About 25 agent-hours on top of Build 1:
| Step | Agent-hours |
|---|---|
| Intake normalization | 4 |
| Drafter | 6 |
| Reproducibility runner | 5 |
| Repair integration with Anchor CI | 3 |
| Provenance hook and dataset-build enforcement | 3 |
| Pilot on 30 recent disclosures | 4, plus range time |
Compute floor: the pilot costs 30 environment builds plus full anchor validation for every task that survives reproduction. Estimate two overnight M5 Max runs.
Acceptance:
- At least 40% of piloted disclosures become validated tasks without human authoring. This is a pre-registered target (assumption), set before the pilot from Orange's manual conversion experience. Adjust the number if needed, but keep the pre-registration.
- Zero tasks enter any training set without reproducibility and anchor records.
- Every parked candidate carries a specific, checkable reason.
- Per-source quality statistics are produced and match a manual recount on a 10-disclosure sample.
Parked candidates double as threat intel. The draft's reasoning is that disclosures which cannot be reproduced are often overstated. That is a design rationale, not a measured rate.
The risk. The first risk is repair drift: fixing a task until it validates by drifting away from the disclosed vulnerability, which produces a valid task about a different bug. Three counters guard against it: the pinned manifest, a reproducibility test specific to the disclosure, and the rule against changing the vulnerability.
The second risk is that disclosure sources of uneven quality poison the queue. The per-source statistics make that visible and routable.
The third is that anchors certify discrimination, not realism. A factory can mass-produce discriminating tasks that teach a narrow, disclosure-shaped view of offense. To counter that, human curation continues at the portfolio level (the balance of task families), not at the task level.
Where it lives. Counterfleet Orange, feeding the cyber range and gym and the LAB2 training sets. It depends on Build 1, and its provenance ties into the LAB2 ledger.
Build 5: Fleet Edit Memory
A SIM-style improvement memory shared across the agent fleet, for harness, prompt, and skill edits. Draws on Paper 2.
The idea. The fleet edits itself constantly: harness configurations, prompts, skills, routing rules. Today those edits are scattered across chats and machines, and their outcomes mostly go unrecorded, so the fleet repeats failed edits and loses successful ones.
This build creates one file-backed memory in the SIM schema. Every improvement edit is recorded twice:
- before it lands: what changes, why, and what is predicted
- after it lands: what was observed, and in what context
Agents proposing an edit must read the memory first and cite the relevant records. The memory becomes the fleet's institutional knowledge about modifying itself, and it compounds.
What changes in transfer. SIM studied one harness improving on one benchmark family per setting. The fleet is heterogeneous: different agents, models, harnesses, and task families. So context is a first-class field in every record (agent, model, harness version, task family), and retrieval filters on it.
An edit that helped a GLM lane on research tasks is not evidence about a Claude lane on coding tasks. The memory must make that distinction mechanically, not leave it to the reader's judgment.
Architecture.
- Record store. A directory of records, one file per edit, in a fixed schema: structured front matter plus a short narrative with a length cap. These are plain files in a repo, readable by every agent, with no database and no service to keep alive. This matches the papers' state-in-files pattern and Brandon's VM-survival rule. If volume outgrows it, the storage changes and the records do not (assumption).
- Write path. A small CLI or skill that opens a record with a prediction and closes it with an outcome. Target: under two minutes per edit.
- Read path. Retrieval keyed on a proposed edit's context (agent, model, harness version, task family, edit type). Negative outcomes are weighted at least equally with positive ones.
- Transfer guard. Retrieval across scopes is allowed but labeled cross-context. An edit justified mainly by cross-context records needs a local verification run before it lands.
- Audit. A weekly pass that checks for:
- edits without records
- records never closed
- re-proposals of edits already recorded as failures in the same context
Inputs and outputs.
- Inputs: every fleet improvement edit (harness config, prompt, skill, routing), plus local check results for each edit.
- Outputs: the memory, weekly audit reports, and a quarterly synthesis written from the records.
The frozen check. The check is procedural, and it is stated plainly so it can be enforced:
- No improvement edit counts as fleet knowledge until its record is closed with an observed outcome.
- Predictions are written before the edit lands. A record whose prediction was edited after observation is invalid, and the audit flags it.
- Where the edit type has a natural metric, the observed outcome must include it, measured the same way as the prediction's baseline.
The audit's failure list is the enforcement surface.
Build sequence and acceptance. About 22 agent-hours. This is the cheapest build in the book, and its compute floor is negligible.
| Step | Agent-hours |
|---|---|
| Schema and template | 3 |
| Write path | 4 |
| Retrieval helper with context filtering | 4 |
| Integration into the two highest-volume edit surfaces first (skills and harness configs) | 4 |
| Weekly audit agent | 3 |
| Backfill of the 20 most consequential recent edits from existing notes | 4 |
Backfilled outcomes are marked as reconstructed wherever they were not measured at the time.
Acceptance:
- After 30 days, at least 80% of improvement edits on integrated surfaces have closed records.
- At least half of new records cite a prior record.
- In testing, the audit catches a planted unrecorded edit and a planted post-hoc prediction edit.
- In the first 30 days, at least one repeated failed edit is prevented by citation, or the audit reports honestly that none was attempted.
Build 2's edit memory adopts this schema rather than inventing its own.
The risk. The main risk is Paper 2's negative transfer at fleet scale. A memory trusted uncritically imports lessons into contexts where they do not hold, and its confidence grows faster than its validity. The counters:
- mandatory context fields
- cross-context use that is labeled and gated
- equal weight for negative outcomes
- a quarterly synthesis required to report transfer failures, not just wins
A second risk is friction. Friction kills adoption, and friction complaints are a build defect, not an agent attitude problem.
A third is that the memory becomes a junk drawer. The schema is kept fixed and small, and the narratives are capped.
Reviewer note. Paper 2's Appendix F shows two accepted edits that fixed 0 of 4 predicted tasks. The edits were kept on their aggregate effect, while their diagnosis was wrong. A closed record with an outcome is therefore not enough. Have the weekly audit also report each prediction's hit rate, meaning how often the predicted effect actually occurred. Accepted edits whose predictions missed then become visible as "worked, for unknown reasons," rather than counting as confirmed knowledge.
Where it lives. Fleet infrastructure, cross-cutting. It serves T3-driven work on both Macs, the skill library, and harness configuration.
Build 6: Divinity Anchor Bench
Label-free anchors for benchmark construction in the Divinity Science pipeline. Draws on Paper 3. Lives entirely inside Divinity's boundaries, and shares patterns with Builds 1 and 3 but no data.
The idea. The Divinity pipeline trains Nemotron models on experimental-data environments with verifiable rewards, Litmus-Bench-style. Its benchmark problem is the inverse of the gym's. Tasks come from experimental science, where labels are expensive and sometimes contested, and a bad benchmark quietly steers training toward the wrong capabilities.
Candidate benchmark tasks are validated by anchors that need no gold labels. The ordering ladder is the pipeline's own checkpoint lineage: base model, SFT checkpoint, RL checkpoint, and a stronger reference model.
What changes in transfer. The ladder's relative strength is expected but must be confirmed, not assumed. Tasks also come from experimental data, where a task can be scientifically ill-posed in ways a coding task cannot. So this build adds a control the paper did not need: a shuffled-data control.
Architecture.
- Candidate intake. Tasks proposed by the pipeline team or generated from experimental datasets, in a standard format with grader code.
- Checkpoint ladder runner. Evaluates each candidate across the ladder, with repeated runs.
- Anchor checker. Checks ordering with margin, the difficulty band, and the variance bound. It follows Build 1's deterministic pattern but is implemented separately for Divinity's stack and data rules.
- Control battery. Broken grader variants, trivial task variants, and the shuffled-data control.
- Repair and escalation loop. Three rounds of agent repair on the task or grader, then human escalation to the pipeline team. Escalation notes double as a data-quality signal about the underlying experimental datasets.
Nothing from this build touches Counterfleet or LAB2 data, and nothing from those programs touches it.
Inputs and outputs.
- Inputs: candidate benchmark tasks, experimental datasets, and pipeline checkpoints.
- Outputs: validated benchmark tasks with records, a per-release validation report that accompanies any training result the pipeline reports internally, and escalation notes.
The frozen check.
- Ordering anchor. The ladder orders correctly, with the reference strongest and the base weakest, and the reference beats the base by a pre-set margin (assumption: 15 points to start, tuned after the first batch).
- Difficulty band. Success across the ladder falls between 5% and 95%.
- Variance bound. Repeated runs of one checkpoint agree within a set bound (assumption: 10 points).
- Shuffled-data control. Permute the experimental data attached to a task. Scores must fall to within 5 points of the weakest rung's unshuffled score. If they do not, the grader is reading something other than the experiment, and the task is rejected. This control has no analogue in Paper 3, because merged code changes cannot be shuffled the way experimental data can.
- Grader freeze. Graders are frozen during validation, as in Build 1.
Build sequence and acceptance. About 21 agent-hours:
| Step | Agent-hours |
|---|---|
| Task format alignment with the pipeline's existing environment format | 3 |
| Ladder runner | 5 |
| Anchor checker and control battery, including the shuffled-data control | 6 |
| Repair loop and escalation reporting | 3 |
| Trial on current benchmark candidates, with a written comparison against the team's manual review of the same candidates | 4 |
Compute floor: ladder evaluation is inference across four checkpoints with repetitions. That is hours per batch on the M5 Max, recurring with each benchmark release, and it comes out of the Divinity program's existing compute budget.
Acceptance:
- The trial flags at least the same problematic candidates that manual review flags. Any candidate the trial passes but manual review rejects gets a written post-mortem, which is added to the control battery.
- The shuffled-data control demonstrably catches a grader seeded to ignore experimental data.
- Validation reports are produced for one full training cycle without manual assembly.
The risk. Anchors certify discrimination, not experimental competence. A task can order the ladder by rewarding style, verbosity, or format compliance. The shuffled-data control narrows this risk without closing it, and human scientific review remains the authority for new task families.
The second risk is the ladder assumption. If an RL checkpoint is genuinely worse than its SFT parent on a real capability, the anchors will punish valid tasks for reporting it. So ladder order is re-confirmed each release on a small set of human-trusted tasks. Anchor failures that cluster on one rung are investigated as checkpoint findings first and task findings second.
Where it lives. The Divinity Science pipeline only.
Build 7: Data Factory Anchor Loop
Anchor-validated quality control for LAB2 synthetic training examples. Draws on Paper 3, with Paper 2 for the repair records.
The idea. LAB2's data factory generates worked examples for cyber model training, and today a schema validator checks them. Schema validity is a floor, not a quality bar. An example can be perfectly formatted and still pedagogically wrong, trivially easy, or impossible.
This build adds a behavioral validation layer:
- Generated batches are sampled and run against a ladder of solver models of known relative strength.
- Examples whose solve pattern violates the anchors are rejected or routed for repair.
- Batch-level anchor statistics become the factory's quality dashboard.
The factory stops measuring itself by volume and starts measuring itself by validated yield.
What changes in transfer. Paper 3's loop validated tasks by running models of known strength. This loop validates training examples the same way, because a training example is a task with a worked solution attached. Two things differ:
- The unit is an example, and the volume is far higher. So validation is sampling-based, with batch acceptance rules instead of per-item gates.
- One anchor has no analogue in the paper: solution consistency. The shipped solution must actually solve the example under the grader, verified by execution rather than by the generator's say-so.
Architecture.
- Batch sampler. Stratified by task family, difficulty claim, and generator prompt.
- Solver ladder runner. Runs sampled examples against solver models spanning LAB2's strength range, from current small cyber models up to a strong reference. The top rung is always external, to avoid circularity. It partially reuses gym infrastructure (assumption).
- Execution checker. Runs each sampled example's shipped solution against its grader. The result is binary.
- Anchor scorer. Records per-example solve patterns and computes per-batch violation rates against batch acceptance thresholds.
- Repair and routing. Violating examples go one of three ways: generator-prompt repair (with edit proposals recorded Paper 2-style), bounded one-round example repair, or rejection.
- Ledger hook. Batch statistics and routing decisions are written to the LAB2 provenance ledger, so every training set carries the validated-yield numbers of the batches it was built from.
Inputs and outputs.
- Inputs: generated example batches, the solver ladder, and anchor thresholds for each task family.
- Outputs:
- accepted batches with validated-yield statistics
- rejected examples with reasons
- generator-prompt edit proposals with recorded outcomes
- a quality dashboard broken down by family, prompt, and source material
The frozen check.
- Execution anchor. 100% of sampled examples must have shipped solutions that pass their own graders. Any failure rejects the example and flags the generator prompt that produced it. There is no sampling tolerance.
- Difficulty ordering. Within a family, the mid ladder rung must solve examples labeled harder at lower rates than examples labeled easier, by a margin (assumption: 15 points). Inverted labels mean the family's difficulty signal is noise, and the batch is held.
- Sanity band. An example solved by the weakest solver but labeled hard, or unsolved by the strongest but labeled easy, is a violation. A batch is accepted only if violations stay under 5% of the sample.
- Consistency bound. For the mid rung, repeated solver runs agree on solve or fail in at least 90% of repetitions.
- Freeze rules. The ladder and graders are frozen during a batch's validation. Generator prompts may be edited between batches, never during one.
Build sequence and acceptance. About 23 agent-hours:
| Step | Agent-hours |
|---|---|
| Sampler and execution checker | 4 |
| Solver ladder runner | 5 |
| Anchor scorer and batch logic | 4 |
| Routing and SIM-style repair records | 4 |
| Dashboard and ledger hook | 3 |
| Calibration run over the existing manufactured corpus (the 230-example seed corpus is the natural first subject) | 3, plus inference: at most one overnight M5 Max run, likely less |
Acceptance:
- The calibration run produces yield statistics that the LAB2 team judges plausible against what they know of the corpus. Any surprises are investigated and explained.
- In testing, a seeded batch with known-bad examples (broken solutions, inverted difficulty labels) is caught at the batch level.
- Execution anchor enforcement works automatically: a broken shipped solution in the sample is rejected, and its prompt is flagged.
- Two consecutive production batches ship with ledger entries containing validated-yield numbers.
The risk. Anchors measure discrimination and consistency, not pedagogical value. A batch can pass every anchor and still teach shallow pattern matching. The anchors are a quality floor and a regression alarm, not a curriculum, and curriculum decisions stay human.
The second risk is solver-ladder circularity. The counters are the always-external top rung and a ladder review whenever LAB2 checkpoints change.
The third risk comes from Paper 2: when outcomes are noisy at small samples, generator-prompt repair records can encode superstitions. So a prompt edit counts as knowledge, rather than hypothesis, only after batch-level evidence across at least two batches. The record schema has a field for exactly that distinction.
Where it lives. The LAB2 cyber model program, inside the data factory. It runs downstream of the existing schema validator and upstream of dataset builds.
Sequencing: the dependency order
The order is stated once here, so it does not become a recurring question:
- Build 1 (Gym Anchor CI) first. Builds 3 and 4 consume it, and it de-risks every score the gym produces from now on.
- Build 5 (Fleet Edit Memory) in parallel. It is the cheapest build in the book, has no dependencies, and every other build generates edits whose records belong in it.
- Build 4 (Orange Task Foundry) once Anchor CI is live, because Orange is the gym's highest-volume task source.
- Build 3 (PoD Grader Factory) once the gym ladders are stable under Anchor CI. PoD's credibility spend should happen exactly once, on validated machinery.
- Build 2 (Retester Loop) once the frozen evaluation sets exist and have survived a rotation decision. It is the highest-compute build, and its outputs need the most interpretive care.
- Build 7 (Data Factory Anchor Loop) when the next LAB2 data batch is scheduled, so calibration runs against real production pressure.
- Build 6 (Divinity Anchor Bench) on the Divinity pipeline's own schedule, inside that program's boundaries.
Total: roughly 175 agent-hours across all seven builds, plus recurring compute floors, mostly ladder evaluation runs on the M5 Max. The recurring pattern is the point. Each build is the same skeleton, so the second costs less than the first, and the edit memory (Build 5) records what each build taught the fleet about running the next one.