The Cyber Version
In one breath. Cybersecurity is a natural home for the self-improving loop skeleton, for a structural reason: cyber has some of the most honest verifiers in applied AI. A proof-of-concept either triggers the vulnerability or it does not. A patch either survives re-attack or it does not. The measuring instrument the November papers work hard to build for coding, a check that is expensive to argue with, is close to native in this domain. This page works out the theory of that fit, then the practice: how each paper maps onto the programs described in the build book (the cyber range and gym with its train-open and holdout-private task sets, Counterfleet and its Proof of Defense measurement, the Orange threat-to-task pipeline, the LAB2 cyber model program and its data factory, and the Divinity Science training pipeline), and finally the open problems the papers leave that a cyber program is unusually well positioned to answer.
How to read the labels on this page. Claims about the papers carry the paper's own numbers and conditions. Statements about Brandon's programs are program design, taken from the build book draft; nothing beyond it is asserted about any program. Extensions, proposals, and projections are marked speculation or research proposal.
Theory: why cyber fits the skeleton
Exploit success is close to a binary verifier. The skeleton's scarcest resource, in every paper, is a trustworthy verdict. Paper 3 grades patches with an LLM judge, because none of its five cohorts recorded fail-to-pass tests, and then has to build negative controls, grader probes, and an expert audit (30 tasks, single rater) to show the judge behaves. CyberGym (UC Berkeley, 2025; abstract-level, see the reading path) shows the cyber contrast: 1,507 real vulnerabilities from OSS-Fuzz, and the agent's proof-of-concept either reproduces the bug or it does not. Crash or no crash. No rubric, no judge drift, no self-preference confound. A domain whose ground truth is an execution event starts where Paper 3 is trying to get to.
One caveat keeps this honest. "Binary" describes the verdict on a single run, not the run itself. Exploits can be nondeterministic, environments flaky, and detection metrics timing-sensitive, so the same exploit against the same target can succeed on one run and fail on the next. An executable verdict removes judge disagreement; it does not remove run-to-run variance. That is why the variance anchor matters more in cyber than in coding, and why it appears below as an open problem rather than a solved one.
Re-attack is a strong check, but only frozen if it is pinned. The strongest frozen check in the November papers is a hash-frozen holdout plus a read-only grader. Cyber defense has a stronger verdict available: a fix is judged by re-running the attack. Proof of Defense already names fix survival under re-attack as a headline metric, with the gym's attacker ladder supplying re-attack verdicts (program design). A patch that closes the reported instance but leaves the vulnerability class open can fail a re-check that varies its approach.
There is a tension here the draft of this page skipped. An adaptive re-attacker that changes its approach between evaluations is a powerful check, but it is not a frozen one: if the re-checker improves, the standard a fix is judged against moves, and two fixes evaluated a month apart were not graded by the same instrument. The skeleton's rule still applies. To serve as the frozen check, the re-attacker must be pinned and versioned: a named attacker configuration, frozen for the duration of an evaluation, recorded in the result, and replaced only by a deliberate, logged decision that restarts comparisons. An improving attacker is valuable; it simply has to be released into the gate the way a new grader is, by version, not continuously. (Doctrine, from the papers' frozen-check rule.)
Threat intel is a sourcing stream. Paper 3's factory needs a renewable supply of candidate tasks and finds one in merged code changes. Cyber has another: the disclosure stream. New vulnerabilities, advisories, write-ups, and patch diffs arrive continuously, each one a candidate task with real-world provenance. Contamination, the slow poison of static benchmarks (see Benchmarks), is structurally lower for fresh disclosures, because they postdate most training data at disclosure time. A benchmark factory fed by the disclosure stream is a live benchmark by construction, the thing SWE-bench-Live builds for GitHub issues and Paper 1 asks for in its conclusion. (Speculation, in the sense that no paper in this family has tested a disclosure-fed factory.)
The capability ladder already exists. Paper 3's anchors presuppose solvers of known relative strength on the target task family, which for novel cyber tasks is the hard prerequisite. The gym already maintains a pinned attacker ladder of solver configurations of known relative strength; its exact composition is a gym program decision (program design). The anchors become applicable the moment that ladder is treated as an instrument and frozen accordingly: versioned, recorded with every validation, and changed only by decision.
Defense is measured on outcomes, which are anchor-shaped. The metrics the programs name are attack success rate and cost per resolved incident (the gym's headline pair), plus fix survival under re-attack, detection time, and containment time (the PoD metrics) (program design). Every one of them is an aggregate over runs, exactly the level at which anchors operate. Paper 3's lesson, that aggregate relational checks can validate an instrument without per-item labels, applies with unusual directness: a grader that cannot rank a ladder of defender configurations of known relative strength is wrong, and that can be known before it scores anything that matters. The same lesson carries Paper 3's limit: anchors are necessary, not sufficient, and they certify discrimination, not realism.
Practice: the mapping, program by program
The cyber range and gym: task and grader validation
The gym takes in tasks: authored scenarios, generated variants, and tasks converted from disclosures, routed into a train-open set and a holdout-private set (program design). Its central production risk is Paper 3's central problem: an invalid task, too easy, too hard, mis-graded, or under-specified, teaches a wrong lesson and corrupts every score computed on it, silently.
The mapping is direct. Every task entering the gym passes an anchor gate: the pinned attacker ladder must order correctly on it, with margin; its success rates must sit inside a difficulty band; repeated runs must agree within a variance bound, exercising the variance anchor Paper 3 defined but never used, because its rounds were not repeated. Negative controls run with it: a trivial variant must be rejected, a deliberately broken grader must be caught, a label-shuffled scoring run must fail to reproduce the ordering. Failing tasks enter a bounded repair loop, three rounds, then human escalation with a written diagnosis. Passing tasks carry a validation ledger entry, so any published score traces to a grader that was itself tested. This is Build 1 in the build book, and it is first in the dependency order because every other build consumes validated tasks or validated graders.
The under-specification failure class transfers with special force. Paper 3's audit (30 tasks, one rater per task) found all five wrong verdicts were FAILs on tasks whose descriptions omitted details the reference contained. Cyber tasks can have the same defect in a more dangerous form: an under-specified defended environment, or an exploit path described loosely, produces verdicts that punish solvers for the task author's omissions. The anchors cannot catch it here either, because under-specification lowers every rung's rate alike. The build book therefore treats a specification filter as a separate concern, human-seeded and control-tested, rather than pretending the anchor gate covers it.
Counterfleet and Proof of Defense: grader credibility
Proof of Defense is a public measurement claim: this defense, against this attacker ladder, produced these outcome numbers. Its credibility problem is precisely Paper 3's H2 problem: why should anyone believe the graders? The build book's answer (Build 3) is to manufacture the belief mechanically (program design). Every PoD grader is validated before it scores a public run: it must rank a defender ladder of known relative strength correctly, with margins pre-registered in the spec; it must pass a control battery with a 100% requirement (an always-success grader must fail ordering, shuffled defender labels must produce near-random rankings, a run whose fix demonstrably fails re-attack must not be scored as survival); re-scoring identical logs must be bit-identical. The validation record ships with the results, so a skeptical reader can trace a PoD number to a grader version and its test history.
The skeleton's separation of writable and read-only maps onto the program sides. Red is the attacker side; Blue and Green are the defense sides that consume Red's findings (program design). The papers' invariant says the scoring function must sit outside the edit reach of all of them. Paper 1's tampering failure mode (agents weakening assertions, editing scoring scripts, deleting logs, in audits the paper cites) is what that separation exists to make structurally impossible.
Speculation, labeled as such: a mature PoD program will likely want an explicitly independent scoring function, organizationally separate from the sides it scores, whose own verdicts are periodically anchor-tested against known-outcome runs: a grader factory auditing the auditors, because Paper 3's deepest lesson is that no instrument in the chain is exempt from validation. How often such a function would wrongly accept a change is unmeasured. TRUST's 45.59% figure does not answer it: that number is the accepted-inferior-update ratio of a score-greedy gate as implemented in TRUST's own study (HotpotQA, DROP, APPS; two model families), not of human or institutional review. What TRUST does license is the question: every gate has a false-acceptance rate, and it should be estimated rather than assumed to be zero.
Orange: disclosed threats to tasks, as a sourcing loop
Orange turns disclosed threats into range training tasks, and today that is a craft process (program design). Read through Paper 3, Orange is a sourcing loop missing its validation half. The foundry version (Build 4) completes it: for each disclosure, an agent drafts the full task (environment, vulnerable configuration, exploit sketch, grader, provenance manifest); a reproducibility runner executes the drafted exploit in the range before anchor validation begins, because a task whose exploit does not run is not a hard task but a broken one; Anchor CI validates; failures drive a bounded repair loop with one hard constraint the paper's factory did not need: the repair loop may not change the vulnerability being modeled, only its environment and grader, or the factory will drift into manufacturing valid tasks about different bugs than the ones disclosed.
Parked candidates, disclosures that cannot be reproduced, are retained with reasons. The build book treats them as a threat-intel signal in themselves, on the reasoning that unreproducible disclosures are often overstated ones (program design rationale, not a measured rate). The output bottleneck moves from authoring time to validation compute, which is the correct bottleneck for a pipeline whose errors teach at scale.
LAB2: data factory quality
The LAB2 cyber model program has a data factory that generates worked examples for cyber model training, checked today by a schema validator; its existing seed corpus is 230 examples (program design). Schema validity is a floor, not a quality bar: an example can be perfectly formatted and trivially easy, impossible, or pedagogically wrong. Build 7 applies Paper 3's inversion: sample each generated batch, run the examples against a solver ladder of known relative strength (with an external reference model always on top, against circularity), and score the patterns. The shipped solution must pass its own grader under execution, 100% of the sample, no tolerance. Difficulty labels must order: examples labeled harder must be solved at lower rates, with margin. Violations route to rejection, bounded repair, or generator-prompt edits recorded Paper 2-style, with rationale and outcome, so the factory accumulates knowledge about which prompts and source materials produce valid examples. The factory's dashboard stops reporting volume and starts reporting validated yield.
Research proposal: the harness and weight alternation. The two spaces page describes a general practitioner pattern, not from these papers and not a claim about any current program: hold a harness fixed, train weights into it with verifier rewards, then resume harness improvement around the improved model. A cyber model program that also ran a gated harness loop (Build 2) on the same task family would be one of the few places that alternation could be measured. Paper 2's negative transfer results are the operating warning for both directions: when the model changes under the harness, expect some inherited edits to start below seed, and let the record, not intuition, say which ones; when the harness changes around the model, the frozen evaluation set is what answers whether the model's gains survived. The alternation cadence (how many harness generations per training run) is an empirical question no paper in this family answers.
Divinity: experimental-data benchmarks
The Divinity pipeline trains Nemotron models on experimental-data environments with verifiable rewards, Litmus-Bench-style (program design). Its benchmark problem is the inverse of the gym's: labels are expensive and sometimes contested, and a bad benchmark quietly steers training toward the wrong capabilities. Build 6 applies the anchor pattern with the pipeline's own checkpoint lineage as the ladder (base model, SFT checkpoint, RL checkpoint, stronger reference), and adds one control Paper 3 did not need: a shuffled-data control. Permute the experimental data attached to a task; a valid task's scores must collapse. If they do not, the grader is reading something other than the experiment, and the task is rejected.
The anchor caveat applies with extra force in science: anchors certify that a task discriminates checkpoints, not that it measures real experimental competence, and a ladder whose assumed order is wrong (an RL checkpoint genuinely worse than its SFT parent on some capability) will punish valid tasks for reporting it. The build therefore re-confirms ladder order on human-trusted tasks per release, and treats anchor failures clustered on one rung as a checkpoint finding first and a task finding second. Divinity's data and builds stay inside Divinity's boundaries; what transfers is the pattern, not the data.
Extending the research: open problems a cyber program can answer
The papers leave specific, named gaps. A cyber program running the builds above would be positioned to answer several of them, and the answers could be publishable contributions rather than internal notes. Everything in this section is a research proposal, not a result.
- Anchors for exploit-success graders. Paper 3 validates LLM graders that score patch similarity. Cyber graders can have execution ground truth, which permits a stronger anchor family: degenerate-submission probes with known correct verdicts (the reference exploit must pass, a no-op must fail, another vulnerability's exploit must fail, a partial exploit must score between), run as a standing battery rather than a one-off study. Paper 3's own probe (142 tasks: landed patch 100%, no-op 0%, another task's patch 0%, padding 0%, half-patch 8 to 28%) is the template. Open question: what false-accept and false-reject rates do anchor suites achieve when the grader is executable? The coding evidence cannot say; the grader class is different.
- The variance anchor, exercised. Paper 3 defined it and never used it. Cyber evaluation is variance-rich: environment flakiness, exploit nondeterminism, timing-sensitive detection metrics. A range that runs repeated ladder evaluations as routine CI would generate real data on how variance anchors behave, what bounds are achievable, and which task families are irreducibly noisy. Open question: is a task with unboundable variance invalid, or merely expensive to measure?
- Transfer guards and the missing bound on negative transfer. Paper 2 documents negative transfer and shows records mitigate it, but it provides no detector and no bound: transplanted harnesses started 0.60 to 5.48 points below seed cross-model and 2.80 points below seed in one cross-dataset case (72.92 versus 75.72), and inheritance lost to a fresh restart on the newest model generation (94.8 versus 95.6), with no stated condition for when not to inherit. The Retester Loop's held-out guard (an edit that improves the frozen set and degrades a never-described held-out set is rejected as suspected overfit) is a candidate detector. Open question: how well do held-out guards predict cross-model and cross-environment transfer, compared to record review alone, and can a program state in advance when an inherited artifact should be dropped? The range can run that comparison directly, because it controls both the frozen and held-out sets.
- Statistical gating for fleet changes. TRUST measured a score-greedy gate accepting inferior updates 45.59% of the time in its experiments (HotpotQA, DROP, APPS; abstract-level), against 2.78% for its statistical procedure. Defense fleets accept configuration, detection, and harness changes continuously, on evidence often no stronger than a validation split. Open question: what is the measured false-acceptance rate of a working security program's change process, and what does a statistical gate cost in evaluation runs per accepted change at fleet scale? Neither number has been published for security operations, as far as this site found.
- Under-specification filters. Paper 3 names the failure class and lists the filter as future work. Disclosure-derived cyber task corpora are likely to be rich in it, and the foundry's parked-candidate records would be labeled examples of exactly this defect. Open question: can a specification-completeness check (does the task statement determine the graded properties?) be automated to the point of catching most of the class before human audit?
- Co-evolution cadence. The harness and weight alternation above, with full records. Open question: the shape of the returns curve. This is Paper 1's declared unsolved problem (model and harness co-evolution), and a program that runs both loops on the same task family is one of the few places it can be measured.
- Adversarial testing of the gates themselves. No paper in this family attacked its own gate with an optimizer. Paper 1's Stage-3 gate rests on design argument and one illustrative rejection. Paper 2's in-loop selector was never tested for bias or sycophancy; its Appendix F shows it accepting two edits that fixed 0 of 4 predicted tasks, on aggregate effect. Paper 3's H2 controls are fixed broken setups, not an adversary searching for a way through, and the authors' own t = 94 blind spot (a self-report grader passes at exactly that threshold) shows such a way can exist. TRUST controls error against noise; an adversarial proposer is a separate question, as far as its abstract states. Cyber is the domain where adversarial testing is native: red-teaming a gate is the same skill as red-teaming a defense. Open question: what is a gate's false-accept rate when the proposer is explicitly optimizing to pass it without doing the work, and which gate designs degrade gracefully under that pressure? A range that pointed a Red-side proposer at Anchor CI or a PoD grader, as a scheduled exercise, would produce the first such numbers in this family.
The through-line of all seven: the November papers built their evidence on coding tasks because coding has the benchmarks. Cyber has unusually honest verifiers and a native adversary. A program that runs the skeleton in cyber, with the records kept honestly and the controls run routinely, would not merely be applying this research. It would be positioned to supply part of the evidence the research is missing. (Speculation.)