Glossary
Plain definitions, one to three sentences each, with a pointer to where the term does its work on this site. Terms are alphabetical.
Anchor. An a-priori relational check on aggregate outcomes that validates a measuring instrument without per-item ground truth: an ordering that must hold, a difficulty band results must sit inside, a variance bound runs must respect. Anchors are necessary, not sufficient, conditions for validity. See Paper 3 and Benchmarks.
Calibration. How well a system's stated confidence or expected ordering matches reality: whether the stronger system actually wins, whether a gate's claimed error rate is its measured error rate. A benchmark or gate can rank correctly and still be miscalibrated in level. See Paper 3 (grader family moved the level by about 11.5 points, not the order, on two companion benchmarks) and TRUST.
Capability ordering. A prior claim, grouped into tiers, about the relative strength of systems on a task family, justified from outside the instrument being built (release order, public results). Anchors check a benchmark against it; persistent disagreement is grounds to revise the tiers, not to torture the benchmark. See Benchmarks and Paper 3.
Contamination. Evaluation material leaking into training data, search archives, or a loop's feedback, so scores measure exposure rather than capability. Defenses are structural: private or fresh task sources, pool separation, frozen test sets. See Benchmarks.
CRSI (Continual Recursive Self-Improvement). Paper 2's setting: continuing an existing harness-optimization run after the benchmark changes (cross-dataset) or the policy model changes (cross-model), with the harness interface preserved. The research question is what should carry over.
Difficulty band. The informative middle range for pass rates, set in Paper 3's study as every model scoring within [0.10, 0.85]. Above the band an instrument is saturated; below it, collapsed to the floor. On a 52-task control set the band alone rejected two broken graders that preserved ordering perfectly. See Benchmarks.
Discrimination. An instrument's ability to separate systems of known different strength. Distinct from realism (tasks resembling real work). Anchors test discrimination because it can be checked without ground-truth labels; realism still needs human audit. See Benchmarks.
False-evolution rate. The fraction of accepted updates that are actually inferior to what they replaced: TRUST's target quantity. In its experiments (abstract-level), a score-greedy gate scored 45.59% and TRUST's statistical gate 2.78%. A rejected inferior edit is a correct rejection, not a false evolution; the error is accepting one. See TRUST.
Fresh context. Starting each loop iteration with an empty context window, reading state from files and writing results back to files. The loop's knowledge is then exactly what is written down, inspectable and persistent. Paper 3 names the pattern the Ralph loop. See The Loop.
Gate. A binary acceptance test a candidate change must pass to be adopted: it executes, it does not regress on held-out data, it stays in budget. Gates reject; scores rank. A gate's false-acceptance rate is a measurable property, and TRUST measures it for one class of gate. See The Loop and Paper 1.
Goodhart's law. When a measure becomes a target, it ceases to be a good measure. In this family it appears as benchmark gaming, evaluator tampering, and threshold tuning. SpecBench operationalizes it as the gap between visible-test and held-out-test pass rates. See Verifiers and Graders.
GRPO (Group Relative Policy Optimization). The standard weight-space RL optimizer of the current era, from DeepSeekMath: sample a group of outputs per prompt, score them, update the policy toward the above-average ones using the group's mean as the baseline, no separate critic network. The method harness-space approaches define themselves against. See Two Spaces.
Grader. The component that renders a verdict on work: a score or a pass/fail. Graders may be executable (tests, crash checks) or judged (a model scoring against a reference or rubric). Paper 3's controls show broken judged and self-report graders failing in measurable ways. See Verifiers and Graders.
Harness. All the runtime software around a frozen model: context assembly and compaction, control loop, tool and action interface, verification hooks. The family's shared claim: harness quality is a first-order performance variable, independent of weights. See The Loop and Paper 1.
Held-out set. Tasks used only to measure, never to optimize. Every paper quarantines one, and Paper 1's failure taxonomy includes a mode (benchmark cheating and overfitting) for what happens when separation leaks. See Verifiers and Graders and Paper 2.
Holdout. Paper 3's specific form: a 20% slice of a benchmark candidate, frozen by a hash of task identifiers, re-verified every iteration, which runs every round but never gates acceptance and never drives a repair. It is reported at acceptance as a regression check. See Paper 3.
Label shuffle. A negative control that destroys the true ordering while preserving task difficulty and noise, by permuting which system produced which result. If a validation procedure still "finds" the ordering under shuffling at any real rate, it is manufacturing signal. Paper 3 measured at most 1.4% pass within five attempts, at margin δ = 0.05 on 96-task draws. See Paper 3.
LoRA (Low-Rank Adaptation). Small trainable matrices layered onto a frozen base model: the lightweight end of weight-space improvement, cheap to train and swap. Trained inside a frozen harness with verifier rewards, a LoRA adapter becomes harness-shaped on purpose (a practitioner pattern, not a result from these papers). See Two Spaces.
Margin (δ). In Paper 3's ordering anchor, the minimum pass-rate gap required between any model in a higher tier and any in a lower one (δ = 0.05 in the study). It is a policy choice that relaxes a significance test, not a noise bound: δ = 0.03 raised the label-shuffle false-accept rate to 3.3 to 4.9% within five attempts, while δ = 0.05 false-rejected the real grader on a 52-task set. See Paper 3.
Meta-agent. An agent whose work product is an artifact other agents are then measured or run with: Paper 3's benchmark builder is the type specimen. The meta-agent operates one level above the systems under test, which is why its own checks must live outside its reach. See Paper 3.
Negative control. A deliberately broken input run through a validation procedure to confirm the procedure rejects it: a trivial task, a shuffled grader, a no-op submission. A check that never rejects anything is not a check. Fixed controls show a check rejects known-broken inputs; they do not show it resists an optimizing adversary. See Paper 3 and Verifiers and Graders.
Negative transfer. An artifact optimized in setting A starts below the unoptimized baseline in setting B. Paper 2's motivating measurement: an Opus-evolved harness scoring 68.41 on Gemini (agentic workspace domain) against the plain seed's 73.89. The paper documents it but offers no detector and no bound on how far below seed a transplant can start. See Paper 2.
Orchestrator. The driver that invokes loop steps, manages fresh contexts, and owns the schedule and the state files: Paper 3's driver program, Paper 1's outer loop. The orchestrator is deliberately dumb; judgment lives in the proposer, discipline in the gate (doctrine). See The Loop.
Pass@1. The fraction of tasks solved in a single attempt. It is Paper 1's metric and Paper 2's coding metric; Paper 3 uses thresholded grader pass rates, and Paper 2's other domains use pooled accuracy and rubric scores. Pass@k (any of k attempts) hides per-attempt cost and can mask structural failure, as Paper 1's Best-of-3 null result shows. See Paper 1.
Pre-registration. Recording a prediction, threshold, margin, or analysis plan before seeing outcomes, so success cannot be retroactively redefined. Paper 2 pre-registers per-edit predictions; Paper 3 fixes its anchors and its relaxed replay rule in advance. See Paper 2.
Problem-Only Memory. Paper 2's crucial control: the inherited harness plus root-cause labels only, with no attempts, predictions, or outcomes. It ranged from -1.82 to +2.40 points relative to carrying the harness alone across six cross-dataset settings, which is why the site says the active ingredient is attempt-level evidence, not diagnosis. See Paper 2.
Proposer and selector. The loop's two separated roles. The proposer generates candidate changes and should be imaginative; the selector screens and accepts, and should be boring. Merging them makes the system grade its own homework. No paper tested its selector's reliability adversarially. See The Loop and Paper 2.
Repair order. Paper 3's sequencing of allowed benchmark repairs by ascending power to tailor the benchmark to the models: sourcing query, then task presentation, then pipeline fixes, then task inspection, then the pass threshold last. The grader and the expected ordering are not repairs at all. See Paper 3.
RLVR (Reinforcement Learning with Verifiable Rewards). Post-training on prompts whose answers a deterministic verifier can check, using the binary verdict as the reward. Named by Tulu 3. The pattern, if you can check it mechanically, you can optimize against it, is the one all three November papers import into artifact space. See Verifiers and Graders and Two Spaces.
RSI (Recursive Self-Improvement). The general pattern: a system improves the process by which it improves, at minimum by editing the artifacts that govern its own future behavior. Papers 1 and 2 are RSI over harness code; the Darwin Gödel Machine is RSI over the optimizer itself. See The Reading Path.
Sourcing query. Paper 3's task-selection instrument: a query over landed code changes that defines the candidate task population. The first and least powerful repair in its repair order, because changing the population by a stated rule is more honest than changing the scoring. See Paper 3.
Test-time compute. Inference-time search, retries, best-of-N, revision, spent instead of improving the system. The control condition every harness-improvement claim must beat at matched compute. It helps when failures are independent; structural failures repeat identically. See The Reading Path, entry 16 (Snell et al.), and Paper 1.
Transfer. An artifact, record, or capability carrying from the setting it was optimized in to a new model, benchmark, or task family, with its value intact. Paper 2's taxonomy: infrastructure transfers directly, mechanisms transfer with recalibration, skeletons transfer and get extended, some components are purely setting-dependent. See Paper 2.
Verifier. The component that renders the verdict a loop optimizes against. Executable verifiers (tests, exploit triggers, re-attack) are treated as outranking judged graders, which outrank self-reports; Paper 3's broken-grader controls are consistent with the lower part of that ranking but do not measure the whole of it. See Verifiers and Graders.