The Build Book: Seven Builds

Seven projects. Each one is the shared skeleton pointed at a program that already exists: a loop edits an artifact, each round starts from fresh context, state lives in files, and a frozen check the loop cannot edit decides what survives. None of them needs a new platform.

Every build names its frozen check before its loop, because the check is what makes the loop safe to run unattended.

How to read this page:

The theory behind each build is on the cyber version page. The doctrine is in the synthesis.

Build 1: Gym Anchor CI

Validates tasks and graders for the cyber range, run as continuous integration on every new task. Draws on Paper 3.

The idea. Every task entering the gym, authored or generated, passes an automated validation gate before anything is scored on it. The gate does four things:

Every task that passes carries a validation record. So every score the gym publishes traces back to a grader that was itself tested.

What changes in transfer. Benchmark Factory sources tasks from merged code changes and validates them against model strength orderings on coding tasks. The gym sources tasks from authors and generators, and validates them against a pinned attacker ladder on cyber tasks. Ground truth there is exploit success or fix survival, not test passage.

The anchors carry over unchanged because they never depended on what the task is about. They need only solvers of known relative strength, and the gym already keeps those as its pinned ladder. Ladder composition is a gym program decision: this build uses the ladder, it does not define it.

Architecture.

If a control passes validation, the validator itself is flagged as broken and the task is held. - Repair loop. An agent gets the failing anchor, the ladder results, and the task folder, and may edit the task or the grader. The anchor checker and the ladder are outside its edit reach, enforced by directory permissions. - Validation ledger. One file record per task per validation run: anchors checked, numbers observed, controls run, repairs attempted, and verdict. This is the audit trail behind any published score.

Inputs and outputs.

The frozen check.

Build sequence and acceptance. About 30 agent-hours:

Step Agent-hours
Manifest schema 3
Ladder runner on Inspect AI, with range sandbox wiring 6
Anchor checker, unit-tested on synthetic result sets 5
Control generators 5
Repair loop with permission-scoped edits 4
Ledger and roll-up 3
End-to-end trial on 20 existing tasks, with threshold calibration 4

Compute floor: the trial is an overnight M5 Max range run, roughly 8 to 12 machine-hours depending on ladder size. It recurs with every validation batch.

Acceptance:

The risk. Anchors are necessary, not sufficient. A task can order the ladder perfectly and still be unrealistic or teach the wrong skill: the gate certifies discrimination, not quality. Human review stays in place for new task families, though not for new tasks within known families. The weekly roll-up exists so that gaming patterns surface early.

There is a second risk: the repair loop may learn to write tasks that please the anchors instead of tasks that test the skill. If repair success rates climb toward 100%, treat that as a warning, not a success.

Reviewer note. Paper 3 defined its variance anchor but never exercised it, because its rounds were not repeated. So the 10-point variance bound has no prior evidence behind it from the source paper. Calibrating it is original work. Log the observed spread of every repeated run from day one, so the bound can be set from data rather than defended from intuition.

Where it lives. The cyber range and gym program. It sits in front of the train-open task set and, with stricter thresholds, the holdout-private set. It feeds Proof of Defense: PoD tasks that passed Anchor CI carry their ledger entries into the published methodology.

Build 2: Retester Loop

A self-improving attacker harness for the range, with a SIM-style edit memory. Draws on Paper 1 and Paper 2.

The idea. The range's house attacker harness improves itself under discipline:

The memory becomes a written record of what works against our own defenses. That record is dual-use. The edits that most improved the attacker are a prioritized list of what the Blue and Green sides must answer.

Full rewrites are forbidden. The rationale comes from Paper 1's Table IV, which is one run on 40 held-out tasks:

So the rewrite bought, at most, a one-task combined edge, with nothing to show which change produced it, at roughly ten times the runtime tokens. That is an argument about cost and attribution, not proof that gating wins on accuracy.

What changes in transfer. The papers improved coding harnesses on coding benchmarks. This build improves an attack harness against defended range systems. The evaluation metric is the gym's own headline pair: attack success rate against defended targets, and cost per success from the attacker's side (tokens and wall time per successful compromise).

Architecture.

Inputs and outputs.

The frozen check.

Build sequence and acceptance. About 25 agent-hours:

Step Agent-hours
Module inventory and edit scoping 3
Evaluation set assembly from existing range scenarios 5
Verifier 5
Edit memory 3
Proposer with citation requirement 4
Digest reporting 2
Supervised first week 3, plus human review of the first 20 keep and reject decisions before the loop runs unattended

Compute dominates: two full evaluation runs plus one held-out run per edit, roughly 1 to 3 M5 Max machine-hours per edit depending on scenario size. The edit budget is therefore a compute decision, not an agent-time decision.

Acceptance:

The risk. Paper 2's negative transfer: lessons from one context can degrade another. In the paper, transplanted harnesses started up to 5.48 points below seed on a new model. The paper offers no detector and no bound. The held-out guard and the per-family regression caps are the local counters. The memory records evaluation context, so future proposers can see where an edit helped and where it hurt.

A second risk is overfitting to the range's defended systems, which would mislead Blue and Green about real-world exposure. The counters: the frozen set rotates by human decision, and the digest states plainly that gains are range-relative.

A third risk is capability drift. A steadily improving house attacker is a sensitive artifact. It lives in the range, on Brandon's machines, under the range program's handling rules.

Reviewer note. The keep rule is a heuristic gate of exactly the class TRUST measured. In TRUST's own study, a score-greedy gate accepted inferior updates 45.59% of the time. That figure does not transfer as a number, but it does transfer as a warning. Paper 1's own validation split was 8 tasks, where one task moves the score by 12.5 points.

Before trusting a 3-point keep threshold, do two things:

An edit that clears 3 points by less than that spread is noise. Either record it as "not enough evidence," or send it to a second evaluation, rather than keeping it.

Where it lives. The cyber range and gym program, Red side. The digest feeds Counterfleet. The edit memory follows Build 5's schema.

Build 3: PoD Grader Factory

Anchor-validated scoring for Proof of Defense, with negative controls as a publication requirement. Draws on Paper 3.

The idea. Proof of Defense lives or dies on whether outsiders believe the scores. This build manufactures that belief mechanically. Every PoD grader is produced and validated through the Benchmark Factory pattern before it scores a public run, and the validation record ships with the results. Each published number arrives with its grader's test history beside it, so credibility becomes an artifact rather than a claim.

What changes in transfer. PoD graders score defense, not task solving. The known-strength ladder is a ladder of defender configurations, for example: undefended baseline, detection only, detection plus automated containment, and the full fleet. The relative strength of these rungs is established by construction and confirmed by the gym's attacker ladder. The anchor logic is identical to Paper 3's: if the grader cannot rank the ladder correctly, with margin, then the grader is wrong, not the defenders.

Architecture.

Inputs and outputs.

The frozen check.

Build sequence and acceptance. About 29 agent-hours:

Step Agent-hours
Spec format and the first two metric specs (fix survival, cost per resolved incident) 5, plus Brandon's review
Grader implementations 6
Ladder validation harness 4
Control battery, including the adversarial log sets 6
Publication packager and methodology generator 4
First full rehearsal 4

Compute floor: roughly one overnight M5 Max range run per grader version.

Acceptance:

The risk. The main risk is silent misdirection: a rigorous-looking grader that ranks things wrongly. For PoD, one wrong public ranking costs more than publishing nothing, which is why the battery is a hard gate rather than a report.

The second risk is ladder construction bias. If the defender ladder's true order differs from the constructed order, the anchors will punish correct graders. Ladder order is therefore confirmed independently by the gym's attacker ladder before validation starts. Disagreements between the two ladders are investigated as findings, not smoothed over.

Reviewer note. The control battery tests fixed, known-bad inputs. Paper 3's equivalent controls did the same, and its own admitted blind spot shows the limit of that approach: a self-report grader passes at exactly t = 94. A fixed battery cannot find the input nobody thought to write.

Before the first public release, schedule one adversarial pass. Give a Red-side agent the grader spec and the goal "produce logs this grader scores as a surviving fix when the fix fails re-attack." Then add every log set it finds to the battery. This is the gate-gaming test that no paper in the family ran (cyber version, open problem 7).

Where it lives. Counterfleet, as the scoring backbone of Proof of Defense. It consumes the gym's ladders and range logs and shares validation machinery with Build 1.

Build 4: Orange Task Foundry

Orange's threat-to-task pipeline, rebuilt as a Benchmark Factory loop over disclosed vulnerabilities. Draws on Paper 3. Depends on Build 1.

The idea. Orange turns disclosed threats into range training tasks, and today that is a craft process. The foundry turns it into a factory with a quality gate:

  1. For each disclosure, an agent drafts the complete task.
  2. A reproducibility check runs before any validation.
  3. Anchor CI (Build 1) gates entry.
  4. Failures drive a bounded repair loop.

Output stops being limited by authoring time and starts being limited by validation compute. That is the correct bottleneck, because an invalid task teaches a wrong lesson at scale.

What changes in transfer. Benchmark Factory's agent wrote queries over landed code changes. Orange's agent drafts tasks from disclosure feeds: advisories, CVE records, write-ups, and patch diffs where they are public. That source material is messier and its ground truth is thinner. A merged change carries its own context, while a disclosure carries prose of varying honesty. So the foundry adds one component the paper did not need: a reproducibility check. The drafted exploit path must actually execute in the range before anchor validation begins.

Architecture.

Inputs and outputs.

The frozen check.

Build sequence and acceptance. About 25 agent-hours on top of Build 1:

Step Agent-hours
Intake normalization 4
Drafter 6
Reproducibility runner 5
Repair integration with Anchor CI 3
Provenance hook and dataset-build enforcement 3
Pilot on 30 recent disclosures 4, plus range time

Compute floor: the pilot costs 30 environment builds plus full anchor validation for every task that survives reproduction. Estimate two overnight M5 Max runs.

Acceptance:

Parked candidates double as threat intel. The draft's reasoning is that disclosures which cannot be reproduced are often overstated. That is a design rationale, not a measured rate.

The risk. The first risk is repair drift: fixing a task until it validates by drifting away from the disclosed vulnerability, which produces a valid task about a different bug. Three counters guard against it: the pinned manifest, a reproducibility test specific to the disclosure, and the rule against changing the vulnerability.

The second risk is that disclosure sources of uneven quality poison the queue. The per-source statistics make that visible and routable.

The third is that anchors certify discrimination, not realism. A factory can mass-produce discriminating tasks that teach a narrow, disclosure-shaped view of offense. To counter that, human curation continues at the portfolio level (the balance of task families), not at the task level.

Where it lives. Counterfleet Orange, feeding the cyber range and gym and the LAB2 training sets. It depends on Build 1, and its provenance ties into the LAB2 ledger.

Build 5: Fleet Edit Memory

A SIM-style improvement memory shared across the agent fleet, for harness, prompt, and skill edits. Draws on Paper 2.

The idea. The fleet edits itself constantly: harness configurations, prompts, skills, routing rules. Today those edits are scattered across chats and machines, and their outcomes mostly go unrecorded, so the fleet repeats failed edits and loses successful ones.

This build creates one file-backed memory in the SIM schema. Every improvement edit is recorded twice:

Agents proposing an edit must read the memory first and cite the relevant records. The memory becomes the fleet's institutional knowledge about modifying itself, and it compounds.

What changes in transfer. SIM studied one harness improving on one benchmark family per setting. The fleet is heterogeneous: different agents, models, harnesses, and task families. So context is a first-class field in every record (agent, model, harness version, task family), and retrieval filters on it.

An edit that helped a GLM lane on research tasks is not evidence about a Claude lane on coding tasks. The memory must make that distinction mechanically, not leave it to the reader's judgment.

Architecture.

Inputs and outputs.

The frozen check. The check is procedural, and it is stated plainly so it can be enforced:

The audit's failure list is the enforcement surface.

Build sequence and acceptance. About 22 agent-hours. This is the cheapest build in the book, and its compute floor is negligible.

Step Agent-hours
Schema and template 3
Write path 4
Retrieval helper with context filtering 4
Integration into the two highest-volume edit surfaces first (skills and harness configs) 4
Weekly audit agent 3
Backfill of the 20 most consequential recent edits from existing notes 4

Backfilled outcomes are marked as reconstructed wherever they were not measured at the time.

Acceptance:

Build 2's edit memory adopts this schema rather than inventing its own.

The risk. The main risk is Paper 2's negative transfer at fleet scale. A memory trusted uncritically imports lessons into contexts where they do not hold, and its confidence grows faster than its validity. The counters:

A second risk is friction. Friction kills adoption, and friction complaints are a build defect, not an agent attitude problem.

A third is that the memory becomes a junk drawer. The schema is kept fixed and small, and the narratives are capped.

Reviewer note. Paper 2's Appendix F shows two accepted edits that fixed 0 of 4 predicted tasks. The edits were kept on their aggregate effect, while their diagnosis was wrong. A closed record with an outcome is therefore not enough. Have the weekly audit also report each prediction's hit rate, meaning how often the predicted effect actually occurred. Accepted edits whose predictions missed then become visible as "worked, for unknown reasons," rather than counting as confirmed knowledge.

Where it lives. Fleet infrastructure, cross-cutting. It serves T3-driven work on both Macs, the skill library, and harness configuration.

Build 6: Divinity Anchor Bench

Label-free anchors for benchmark construction in the Divinity Science pipeline. Draws on Paper 3. Lives entirely inside Divinity's boundaries, and shares patterns with Builds 1 and 3 but no data.

The idea. The Divinity pipeline trains Nemotron models on experimental-data environments with verifiable rewards, Litmus-Bench-style. Its benchmark problem is the inverse of the gym's. Tasks come from experimental science, where labels are expensive and sometimes contested, and a bad benchmark quietly steers training toward the wrong capabilities.

Candidate benchmark tasks are validated by anchors that need no gold labels. The ordering ladder is the pipeline's own checkpoint lineage: base model, SFT checkpoint, RL checkpoint, and a stronger reference model.

What changes in transfer. The ladder's relative strength is expected but must be confirmed, not assumed. Tasks also come from experimental data, where a task can be scientifically ill-posed in ways a coding task cannot. So this build adds a control the paper did not need: a shuffled-data control.

Architecture.

Nothing from this build touches Counterfleet or LAB2 data, and nothing from those programs touches it.

Inputs and outputs.

The frozen check.

Build sequence and acceptance. About 21 agent-hours:

Step Agent-hours
Task format alignment with the pipeline's existing environment format 3
Ladder runner 5
Anchor checker and control battery, including the shuffled-data control 6
Repair loop and escalation reporting 3
Trial on current benchmark candidates, with a written comparison against the team's manual review of the same candidates 4

Compute floor: ladder evaluation is inference across four checkpoints with repetitions. That is hours per batch on the M5 Max, recurring with each benchmark release, and it comes out of the Divinity program's existing compute budget.

Acceptance:

The risk. Anchors certify discrimination, not experimental competence. A task can order the ladder by rewarding style, verbosity, or format compliance. The shuffled-data control narrows this risk without closing it, and human scientific review remains the authority for new task families.

The second risk is the ladder assumption. If an RL checkpoint is genuinely worse than its SFT parent on a real capability, the anchors will punish valid tasks for reporting it. So ladder order is re-confirmed each release on a small set of human-trusted tasks. Anchor failures that cluster on one rung are investigated as checkpoint findings first and task findings second.

Where it lives. The Divinity Science pipeline only.

Build 7: Data Factory Anchor Loop

Anchor-validated quality control for LAB2 synthetic training examples. Draws on Paper 3, with Paper 2 for the repair records.

The idea. LAB2's data factory generates worked examples for cyber model training, and today a schema validator checks them. Schema validity is a floor, not a quality bar. An example can be perfectly formatted and still pedagogically wrong, trivially easy, or impossible.

This build adds a behavioral validation layer:

The factory stops measuring itself by volume and starts measuring itself by validated yield.

What changes in transfer. Paper 3's loop validated tasks by running models of known strength. This loop validates training examples the same way, because a training example is a task with a worked solution attached. Two things differ:

Architecture.

Inputs and outputs.

The frozen check.

Build sequence and acceptance. About 23 agent-hours:

Step Agent-hours
Sampler and execution checker 4
Solver ladder runner 5
Anchor scorer and batch logic 4
Routing and SIM-style repair records 4
Dashboard and ledger hook 3
Calibration run over the existing manufactured corpus (the 230-example seed corpus is the natural first subject) 3, plus inference: at most one overnight M5 Max run, likely less

Acceptance:

The risk. Anchors measure discrimination and consistency, not pedagogical value. A batch can pass every anchor and still teach shallow pattern matching. The anchors are a quality floor and a regression alarm, not a curriculum, and curriculum decisions stay human.

The second risk is solver-ladder circularity. The counters are the always-external top rung and a ladder review whenever LAB2 checkpoints change.

The third risk comes from Paper 2: when outcomes are noisy at small samples, generator-prompt repair records can encode superstitions. So a prompt edit counts as knowledge, rather than hypothesis, only after batch-level evidence across at least two batches. The record schema has a field for exactly that distinction.

Where it lives. The LAB2 cyber model program, inside the data factory. It runs downstream of the existing schema validator and upstream of dataset builds.

Sequencing: the dependency order

The order is stated once here, so it does not become a recurring question:

  1. Build 1 (Gym Anchor CI) first. Builds 3 and 4 consume it, and it de-risks every score the gym produces from now on.
  2. Build 5 (Fleet Edit Memory) in parallel. It is the cheapest build in the book, has no dependencies, and every other build generates edits whose records belong in it.
  3. Build 4 (Orange Task Foundry) once Anchor CI is live, because Orange is the gym's highest-volume task source.
  4. Build 3 (PoD Grader Factory) once the gym ladders are stable under Anchor CI. PoD's credibility spend should happen exactly once, on validated machinery.
  5. Build 2 (Retester Loop) once the frozen evaluation sets exist and have survived a rotation decision. It is the highest-compute build, and its outputs need the most interpretive care.
  6. Build 7 (Data Factory Anchor Loop) when the next LAB2 data batch is scheduled, so calibration runs against real production pressure.
  7. Build 6 (Divinity Anchor Bench) on the Divinity pipeline's own schedule, inside that program's boundaries.

Total: roughly 175 agent-hours across all seven builds, plus recurring compute floors, mostly ladder evaluation runs on the M5 Max. The recurring pattern is the point. Each build is the same skeleton, so the second costs less than the first, and the edit memory (Build 5) records what each build taught the fleet about running the next one.