The Complete Synthesis

Evidence on this page: comparison cells for Papers 1 to 3 are paper-reported (checked against full text). TRUST cells are abstract-level. The doctrine section is the site's own inference, labeled as doctrine.

In one breath. The four papers are one architecture, described from four angles. A proposer edits an artifact around a frozen model, in fresh context, with state in files. A frozen check the proposer cannot reach decides what survives. Predictions are recorded before outcomes. A checkpoint, human or statistical, guards the boundary. Each paper contributes one discipline:

A practitioner who implements all four has the complete machine. Implementing any three leaves a hole the family has named but not measured. And one hole is open in all four: none of the four papers attacks its own gate with an adversary.

The shared skeleton, as one architecture

Name the components once, precisely. Every system in this family is an assembly of these parts:

  1. The artifact. Code or configuration around a frozen model: harness modules, a memory file, a benchmark definition. Never the weights. (The two spaces page covers the weight-space alternative and how the two compose.)
  2. The proposer. Generates candidate changes. Reads traces, diagnoses, and memory. It is rewarded for imagination and is therefore never trusted with the verdict.
  3. The selector or gate. Screens candidates, before execution where possible (Paper 2's selector), and applies the acceptance rule after evaluation (Paper 1's Stage 3, TRUST's statistical procedure).
  4. The frozen check. The verdict-rendering apparatus the proposer cannot edit: held-out task sets in read-only storage, a grader and expected ordering frozen before runs, quality anchors fixed a priori. See Verifiers and Graders.
  5. The state file. The loop's entire memory, in files: raw trajectories, scores, and the edit ledger with rationale, pre-registered prediction, observed outcome, and provenance. It is read in full at the start of each round and appended honestly, failures included.
  6. The fresh-context driver. Invokes each step with an empty context, so the loop's knowledge is exactly what the state file says, and a human can inspect it at any time. Paper 3 names this the Ralph loop.
  7. The checkpoint. One of three kinds:
  8. a human reviewing every acceptance and rejection (Paper 3);
  9. a statistical procedure with a controlled false-acceptance rate (TRUST);
  10. in the weaker designs, a heuristic gate whose error rate is unmeasured (Paper 1).

Two cross-cutting disciplines bind the parts:

Paper 1's invariant generalizes: the mutable system must never control its own permission boundaries, its grading harness, or its audit logs.

The comparison

Paper 1: Harnesses Paper 2: SIM Paper 3: Benchmark Factory TRUST (abstract-level)
What may be edited Harness code, one module at a time (context, control, action interface, verification) Harness code, via a proposer-selector loop, across a setting change A benchmark definition: sourcing query, prompt, pipeline, threshold (last). Grader and ordering are not editable The system under recursive self-improvement; the abstract does not specify the artifact type further
What carries across changes Archived harness variants, branchable by later search The source harness plus the SIM ledger: root causes, attempts, rationale, pre-registered predictions, observed outcomes, provenance The frozen benchmark definition, plus the validation record: anchors passed, holdout behavior, control results Not stated at abstract level. The procedure controls a rate across rounds, so it must track decision history (site inference)
The frozen check Stage-3 gate: executes, non-regression on held-out validation (N=8 in the experiment), token budget; evaluators in read-only storage Held-out sets quarantined from every loop decision; predictions written before evaluation Quality anchors (ordering with margin δ, difficulty band, variance), and a hash-frozen holdout that never gates or drives repairs Online multiple-hypothesis testing controlling the false-evolution rate across rounds
Where the human sits Prescribed at the safety boundary: no privilege escalation across self-edits; evaluators isolated Outside the loop; continuation proceeds from the record without a human re-deriving it At every acceptance and rejection; adjudicates every K-round failure; approves repeated threshold drops Above the procedure, presumably setting the target error rate (site inference)
Headline evidence One run per method, held-out SWE-bench Lite: combined, rewrite 17/40 vs gated 16/40; in-domain, gated 8/20 vs rewrite 7/20; OOD, rewrite 10/20 vs gated 8/20; runtime 451.6k vs 46.7k tokens/task; prompt-only 1/40, one task below seed Paired study, coding only, 10 source runs: +2.36 points, 95% CI [1.05, 3.68], p=0.003, 9 of 10 pairs; cross-model margins +3.60 to +11.33, single runs Out-of-sample ordering recovery: effort +6.2 points (p=0.002, n=551, 5 of 5 cohorts); model family +5.3 points (p=0.004, n=552; p=0.07 without C5); all broken controls rejected under a relaxed replay rule fixed in advance Accepted-inferior updates: 45.59% for a greedy gate vs 2.78% for TRUST, on HotpotQA, DROP, APPS with two model families
Named weakness Single run, no intervals; heuristic gate with unmeasured false-acceptance rate; no adversarial gate test One replicated domain; carrying and updating memory confounded; negative transfer unbounded; selector reliability untested Anchors certify discrimination, not realism; under-specified tasks pass; variance anchor unused; audit n=30, single rater; GPT leg fragile Narrow evidence base; evidence costs evaluation runs; guarantee is against noise, not adversaries (as far as the abstract states)

Read the table by columns and each paper is a system. Read it by rows and the family's open joints appear. The "frozen check" row holds four different mechanisms: human cost falls, and statistical sophistication rises, from left to right. The "human" row shows the family steadily trying to move the human from inside every decision to above the procedure, where the human sets budgets and adjudicates only what the procedure cannot.

Evidence strength at a glance

Which claims bear weight, and on what. "Repeated" means independent repetitions of the optimization itself, not repeated rollouts.

Claim Rests on n Repeated? Status
Gated edits cost about a tenth of a full rewrite's runtime tokens (Paper 1) Table IV token means 40 tasks No, one run Large gap, one run; ratio about 9.7 (site arithmetic)
Gated and rewrite reach similar held-out accuracy (Paper 1) Table IV 40 tasks (two splits of 20) No Within one to two tasks on every split; untested
Unenforced rules do not stop the behavior they forbid (Paper 1) Prompt-only arm 40 tasks No Direction clear (unverified exits 3/20 to 6/20); magnitude one run
Gated harness transfers best to a stronger model (Paper 1) Part B 40 tasks No 15/40 vs 12/40; one run
Carrying SIM beats harness-only continuation (Paper 2) Paired study 10 source runs Yes, coding only Established in one domain and one transition; robust to its largest pair (site arithmetic)
The same holds across domains and models (Paper 2) Six cross-dataset, four cross-model settings One run each No Descriptive; consistent direction, no variance
Inheritance can lose to a fresh restart (Paper 2) Generational chain One chain No Counterexample on Argon (94.8 vs 95.6)
Accepted benchmarks rank unseen configurations correctly (Paper 3) H1 pooled contrasts 551 to 552 tasks Pooled over 5 cohorts Effort leg robust; model-family leg fragile to dropping C5
Anchors reject broken setups (Paper 3) H2 controls 52 to 142 tasks Simulated nulls (20,000) Holds under a pre-fixed relaxed rule; unchanged anchors also false-reject the real grader at n=52
Accepted tasks are mostly sound (Paper 3) Expert audit 30 tasks Single rater Wide intervals (correct verdicts 69%, CI 53 to 83%); raters unblinded
A statistical gate cuts false acceptance (TRUST) Abstract figures Not read Not read Abstract-level only

Where they disagree, and what is left open

How much to trust the gate. Paper 1 treats a non-regression gate as sufficient discipline and reports no false-acceptance accounting. TRUST measured a score-greedy gate, in its own study, accepting inferior updates 45.59% of the time. The two do not strictly contradict each other. Paper 1's gate adds execution checks and a budget, and TRUST's figure belongs to its own baseline on its own benchmarks. But the tension is real.

Paper 1's most persuasive gate evidence is the rejected turn-cap edit, and that was a non-regression call on an 8-task validation split, where one task is 12.5 points (site arithmetic). The size of the drop is not reported, and the edit also breached the token cap. A split that small is exactly where TRUST's concern bites hardest. Small true regressions, and lucky small gains, are invisible at that resolution, and they accumulate across rounds. Treat every heuristic gate in this family as carrying an unknown false-acceptance rate until it is measured.

What the record is for. Paper 2 shows that records improve continuation. It does not test whether records improve first-setting optimization, and its design confounds carrying memory with continuing to update it. Is the ledger's value mostly in transfer, or also in ordinary runs? Plausibly the latter, by preventing repeated failed proposals, as the Appendix F repair-loop case suggests. But it is an open empirical question the paper cannot answer from its own design.

How bad inheritance can get. Paper 2 documents transplanted harnesses starting below the plain seed in every cross-model domain:

It offers no bound on how far below seed a transplant can start, no detector that flags a harmful inheritance before rounds are spent, and no stated condition for when not to inherit. "Measure the start before you celebrate" (doctrine 5 below) is the only available guard, and it is a practice, not a result.

Whether discrimination is enough. Paper 3 is admirably blunt that anchors are necessary, not sufficient, and its audit names the defect class they miss. The audit was 30 tasks, single rater. But the field's incentive gradient runs the other way: anchor validation is cheap and automatable, and expert audit is neither. Unless audit sampling scales with the factories, the gap between what the anchors certify and what a benchmark means will widen. The paper lists a sourcing filter for under-specified tasks as future work, and no paper in the family has built it. (The build book treats that filter as a named component in the cyber builds, because cyber task specifications have the same failure shape.)

The unused anchor. Paper 3's variance anchor is arguably the one most relevant to flaky real-world evaluation. It was defined and never exercised, because rounds were not repeated. Run-to-run variance is where LLM-graded, environment-dependent evaluation actually lives. The family's entire evidence on variance control is one grader-agreement study on companion benchmarks: π = 0.81 self-agreement, and up to 1 verdict in 6 flipping between identical runs. No anchor test exists. Treat variance as an open problem, not a solved one.

Nobody attacked the gate. This is the open problem all four papers share:

None of them runs the arm a security engineer would run first: a proposer rewarded for passing the gate without doing the work. The cyber version argues that this is the experiment a cyber program is best placed to run.

Co-evolution. All three main papers freeze the weights. Paper 1 names model and harness co-evolution as unsolved. The staged composition on the two spaces page (adapt the harness, train into the frozen harness, re-adapt) is practitioner doctrine, not a result from these papers, and it is labeled as such there.

Interfaces. Paper 2's continuation requires a compatible harness interface across the change, and real model transitions sometimes break interfaces. The family's current answer to a broken interface is to start over and keep the ledger, which at least makes the restart informed. That is a reasonable posture, not a solution.

The adversarial read

Before this site was finalized, the three papers were put to a skeptical panel: one model (Perplexity, Nemotron 3.5 Super VL Thinking), two threads, a single origin. It was asked for strongest criticisms and reviewer verdicts. The panel is worth reading for what it got right, and worth studying for what it got wrong.

What it got right, and where this site now carries it:

What it got wrong:

The lesson is the site's thesis, applied to the site. The panel was a check, but it was a single model checking from a single origin, with no check on itself. It produced valid criticisms and a confident false claim in the same voice. The defense was the same one the papers use: an independent reading of the source, kept outside the panel's reach, decided which of its claims survived.

The doctrine that falls out

For a practitioner running agents in production, the four papers compress into working rules. These are the site's synthesis, stated as doctrine: inferences from the papers, not quotations from them.

  1. Name the frozen check before you build the loop. If you cannot state what renders the verdict, where it lives, and why the proposer cannot edit it, you do not have a self-improving system. You have a system that edits itself and reports success.
  2. Edit small, gate hard. One module, one hypothesis, one verification run. In Paper 1's single run, a full rewrite spent about ten times the runtime tokens to finish one task ahead overall, behind in-domain, and three tasks behind after a model change. (Paper 1)
  3. Enforce in code, never in prose. A rule the runtime cannot block is a suggestion the model will eventually ignore. In Paper 1's one run, unverified exits rose from 3/20 to 6/20 under a rule that forbade them. (Paper 1, the RULES.md result)
  4. Keep the ledger, with predictions. Record rationale, pre-registered prediction, observed outcome, and provenance, for kept and rejected changes alike. The ledger is the part of the system built to outlive the next model change, and in Paper 2's one repeated study continuing with it beat continuing without it in 9 of 10 pairs. (Paper 2)
  5. Carry the reasons with the artifact. When the setting changes, transplant the record along with the artifact, never the artifact alone. Budget for the possibility that the artifact starts below seed, and measure the start before you celebrate the inheritance. (Paper 2)
  6. Validate the instrument before trusting its readings. Anchors, negative controls, label shuffles, degenerate-submission probes. A check that never rejects anything is not a check. (Paper 3)
  7. Order repairs by their power to fit the measurement. Fix sourcing before presentation, presentation before pipeline, and the threshold absolutely last. The grader and the expected ordering are not repairs. (Paper 3)
  8. Check run health before reading any score. A zero is a pipeline symptom until proven otherwise. The most expensive misreading in Paper 3 came from treating an invalid model identifier as low capability. (Paper 3, the C1 incident)
  9. Price your gate's error rate. In TRUST's experiments, a score-greedy gate accepted inferior updates 45.59% of the time. Your gate's rate is unknown until you measure it, and a small validation split makes it worse. If the loop runs unattended, do one of three things: measure the rate, adopt a statistical gate, or keep a human at acceptance. Do not assume it is zero. (TRUST)
  10. Attack your own gate. Before trusting a loop unattended, run a proposer rewarded for passing the check without doing the work, and see what gets through. No paper in this family has published that experiment. (Speculative doctrine, grounded in the shared gap above; the cyber version develops it)
  11. Keep the human at the questions the check cannot answer. Realism, specification quality, whether the task family still represents the real work. Gates reject the provably bad, and judgment handles the rest. The system's job is to route judgment to exactly those questions. (All four papers, in their different assignments of the human)

The cyber version applies this doctrine where it fits best. Cyber is a domain whose ground truth is an executable event, whose threat stream never stops sourcing new tasks, and whose central measurement problem, proving a defense worked, is exactly an instrument-validation problem.