← Back to blog

· 20 min read

Harnesses that improve themselves: how an agent can rewrite its own runtime without breaking it

Notes on a new framework for self-improving agent harnesses: four editable parts, a diagnose, propose, validate, deploy loop, why gated one-module edits beat full rewrites, a runnable toy, and a Stop hook you can use today.

  • paper-notes
  • ai-agents
  • agent-harness
  • self-improving-agents
  • claude-code
  • evals
  • swe-bench
  • llm

If you run a coding agent for long enough, you end up maintaining two things: the model choice, and everything around it. The system prompt, the CLAUDE.md, the hooks, the tool wrappers, how much of a big file gets pasted back into context, when the loop may stop. Most of my time on the second pile is reading a failed session, guessing what went wrong, and adding a line somewhere.

A new paper argues that this pile is where most of the reliability lives, and that the guessing can be automated without the agent wrecking itself. It is Harnesses That Improve Themselves: A Unifying Framework for Self-Improving Agent Harnesses by Deep Shah and Nehal Kathrotia, both listed with Google, submitted on 17 November 2026. It is mostly a framework paper: it maps about fifteen recent systems that rewrite their own runtime code onto one architecture, then runs a small SWE-bench Lite experiment to test the design rules it pulls out.

These are my notes on how it works, a toy version of the loop you can run, and what I would change in a Claude Code style setup because of it.

The problem: a harness tuned for one model breaks on the next

The paper calls the runtime around the model the agent harness, and gives three reasons it matters as much as the weights. When the same model runs inside different harnesses, its main failure modes (quitting before testing, looping on failed commands, overflowing context, skipping verification) follow the harness, not the model family, and tokens per solved task can vary by an order of magnitude. A good harness props up weak models on short tasks and gives strong models the state tracking they need on long ones. And no single harness suits every model: one tuned for a model can flip leaderboard rankings with another, and hand-maintained harness releases often stall or regress older tasks.

That last point is the one I recognise. You fix the agent for today's model, the model changes, and half your fixes become dead weight or start hurting.

Earlier automated work tuned prompts, demonstrations or multi-agent call graphs but treated the inner runtime (compactors, tool wrappers, loop controllers, stop checks) as fixed. Self-improving harnesses treat that runtime as code the system may edit.

What the paper proposes: four editable parts and a four-stage loop

The framework has two halves. The inner loop is one task: a frozen model talks to a live environment through four code-editable components. The outer loop runs across many tasks and rewrites those components, in four stages.

The outer loop in four steps, left to right: diagnose failures, propose a patch to one module, validate it on held-out tasks and a token cap, deploy it, then repeat.
The outer loop. Stage 3 is the gate that decides what ships. Simplified from the paper's Fig. 1.

The four components, in the paper's words reduced to what they do:

  • Context and State: assembles instructions, separates working memory from longer-lived experiential memory, and masks or compacts stale observations so long runs do not overflow.
  • Agent Control: turn budgets, retries, detecting and breaking loops, switching between planning and execution, handing work to subagents.
  • Action Interface: the agent-computer interface. It turns model output into sandboxed shell commands, file edits or GUI actions, and truncates or formats what comes back.
  • Verification and Recovery: intercepts the stop signal and runs tests, linters or visual checks, then routes failures back into context for repair instead of letting the run end.

Section II also writes the goal down as a constrained optimisation, which is the cleanest part of the paper. The harness is a four-tuple of code modules, the model is frozen, and the objective is the expected verifier pass rate, subject to a token budget and a validation gate. Here it is as a data shape:

eq1.txtText
h = (h_ctx, h_ctrl, h_act, h_ver)          # the harness: four code modules
M_theta                                    # the model, weights frozen
R_x(tau) in {0, 1}                         # external, read-only verifier for task x
C(tau)                                     # runtime tokens of trajectory tau

maximise   E[ R_x(tau) ]        over harnesses h
subject to E[ C(tau) ] <= B_tok            # per-task token budget
           G_val(h, h0) = 1                # Stage 3 gate vs the starting harness h0:
                                           #   modified code paths actually ran, and
                                           #   no regression on a held-out val split

The important part is subject to. Unconstrained maximisation over a small dev set is how you get a harness that memorises the dev set. The modular version approximates this one module at a time, then runs a merge pass to resolve conflicts.

How the outer loop works, stage by stage

  1. Stage 1: Diagnose failures

    Run the harness on a dev split, keep full trajectories, and find recurring bottlenecks by clustering verifier errors or diffing passing and failing runs. One failed run is not evidence.

  2. Stage 2: Propose edits

    An optimiser (the task model itself, or a stronger coding model) turns a failure cluster into a code or hook patch.

  3. Stage 3: Validate and filter

    Screen the candidate on validation tasks. Reject anything that regresses, burns tokens, or wins only on dev-set quirks.

  4. Stage 4: Deploy or route

    Merge the patch, archive it for later rounds to branch from, or route each task to a harness variant.

The paper names three design choices that separate the surveyed systems. Who proposes the edit: the task model itself, or a separate stronger coder (needed for small or domain-specific agents). What code can change: just the task harness, or the optimiser too, as in the recursive Darwin and Mendel Gödel Machines, which costs more compute and needs strict sandboxing. Where the evaluator lives: always outside the agent's write scope, so it cannot pass by editing the grader or the logs.

Underneath: telling a harness bug from a model limit

Section III is the part I found most useful, because it answers the question I get wrong most often when reading failed sessions: was that the model, or my setup?

The paper's rule is a divergence check between what the model reasoned and what actually happened. If the reasoning trace names the right file or the right fix (or the task passes on at least one retry), but the run still fails because the edit tool rejected the patch format, the shell lost its working directory, compaction dropped an earlier constraint, or the loop exited before running tests, then the harness is at fault. If the tool calls ran cleanly, observations stayed intact, and every retry still fails on the domain logic, Stage 1 abstains: no harness patch, because spending search compute on a model knowledge gap is waste.

It lists four ways the surveyed systems attribute failures:

  1. Failure clustering: group failed runs by their compiler or test error, which filters out one-off slips.
  2. Contrastive diffing: diff a passing and a failing run of the same task.
  3. Backward causal tracing: walk back from the stack trace to the earliest turn that wrote or misread the broken value.
  4. Self-reports: the agent says where tools held it back, and a verifier keeps only claims backed by trace events.

This matters because of an ablation the paper cites from the Meta-Harness work: an optimiser that saw only scalar scores or short summaries stalled near baseline, while one with a searchable folder of raw trajectories, prior harness code and diffs found recurring bottlenecks. Reflecting on single failures is also how instruction files fill up with rules for problems that never repeat. One cited paper is titled "Why Does CLAUDE.md Keep Growing?"

On the patch itself, the paper summarises rule ablations from other work: explicit negative boundaries ("never exit without running the tests") beat step-by-step procedural advice, and plain-text safety rules are not enforced unless a deterministic hook blocks the tool call at runtime. Small lifecycle hooks (pre-turn, post-tool, pre-stop) are how the surveyed systems keep self-edits from touching the core loop.

What these loops keep inventing

Across coding, terminal, web, GUI and robotics benchmarks, the optimisers keep finding the same five families of fixes:

  1. Pre-stop verification: check the diff and run tests before the agent may exit. The paper calls blocking unverified exits one of the largest single jumps in solve rate.
  2. Empty-turn recovery: catch a turn with no tool call or a malformed action and inject a continuation prompt.
  3. Loop breakers: counters on repeated failing commands that force a change of strategy.
  4. Environment and memory persistence: carry the working directory and PATH across stateless shell calls, and keep working memory apart from reusable experience.
  5. State-commit guards: confirm a click changed the screen, or a robot's state is consistent, before moving on.

If you have built agents by hand, all five look familiar. They are what people add after reading enough failed runs.

Why full rewrites overfit, and the four safeguards

Section IV argues the unit of change should be one module, not the whole harness. Observation filtering, compaction, turn limits and completion checks share assumptions, so changing them together breaks credit assignment and lets one edit quietly break another (a new output filter trims the stack trace a new verification check needs). The paper cites a ModularRSI ablation where editing all modules at once did worse than the untouched seed on held-out terminal tasks, and an RRSI result where unrestricted evolution topped its dev split but gained almost nothing out of distribution while more than doubling tokens per task.

The fix is four safeguards, spread across the loop:

  1. Evolve modules separately, then run a merge pass that reconciles runtime conflicts using failure traces.
  2. Keep edits small and general: each patch traces to a recurring failure cluster, edit budgets shrink over generations, task-specific hints are rejected.
  3. Enforce acceptance checks: no regression on evolution and validation splits, and proof the new code path actually executed. Code that never runs can pass by luck.
  4. Separate dev and test pools completely, and freeze the harness before out-of-distribution testing.

Section V adds a finding I did not expect, from a cross-scale study it cites: once a model clears a basic instruction-following bar (the example is 32B+ models), the patches it proposes are about as good as a frontier model's, because diagnosing traces is easier than solving the task. Structural checks also transfer across models far better than model-specific prompt wording.

The experiment: 142 lines of Python, one module at a time

Section VII uses 60 SWE-bench Lite tasks: 40 from sympy split into dev (12), validation (8) and held-out test (20), plus 20 from django as a held-out cross-repository test. Each run is in a clean sandbox with git history stripped, graded externally. Every method starts from the same 142-line seed harness:

seed harnessText
context_mgr.py   prompt + raw turn history (no compaction)
control.py       12-turn limit + JSON parser
action_aci.py    bash, view_file, edit_file, finish
verify.py        pass-through stop hook (lets every finish through)

The task agent is Gemini 3.8 Flash, frozen. The optimiser is Gemini 4 Carbon (falling back to Gemini 4 Argon on timeouts). Five methods are compared: the seed once, best of three seed runs, prompt-only reflection that writes learned rules into a RULES.md, an unconstrained rewrite of all four files with no validation or token cap, and modular gated evolution: one module per step, Stage 3 validation, checks that new lines executed, and a 110k-token runtime cap.

The four module edits in the paper's run: context_mgr.py compaction accepted, action_aci.py smaller file views accepted, control.py turn limit 12 to 18 rejected, verify.py blocking untested exits accepted.
The four proposed module edits. The control.py change improved dev accuracy and still got rejected. Source: Section VII.

The rejected edit is the interesting one. Raising the turn limit from 12 to 18 made dev accuracy jump, the kind of change you would happily ship after a good afternoon. Validation dropped and tokens broke the cap, so the gate threw it out. The unconstrained rewrite made the same kind of move (it raised the cap to 28) and nothing stopped it.

Bar chart of held-out solve rate and tokens per task for four methods on Gemini 3.8 Flash. Modular gated reaches 40.0 percent at 46.7k tokens, the ungated full rewrite 42.5 percent at 451.6k.
Held-out results across both repositories, 40 tasks. Best of three is left out: 5.0% at 237.6k tokens. Numbers from Table IV, Part A.

The numbers that stood out to me:

  • Retries do not fix a harness bug. Best of three more than doubled tokens (237.6k per task) and stayed at 5.0%, because context overflowed the same way on every retry.
  • A rule in a text file did not stop the behaviour it banned. The first rule in RULES.md forbade finishing with an empty or untested diff. The model still exited early: unverified cross-repo exits went from 15% to 30%, and accuracy fell to 2.5%. The same idea as a Python check in verify.py took unverified exits to 0.
  • The full rewrite scored highest (42.5%) at 451.6k tokens per task, with one rollout over 2.8M tokens.
  • Modular gated got 40.0% at 46.7k tokens per task: 8x the seed's accuracy at nearly 10x fewer tokens than the rewrite (116.8k vs 1,062.6k per solved task). The paper says the savings pay back its 100 search rollouts in under 100 deployment tasks.
  • Transfer to a stronger model, with no re-tuning: on Gemini 4 Argon, the modular harness got 37.5% at 64.8k tokens, the full rewrite fell to 30.0% at 284.7k, prompt-only got 15.0%, and the seed 10.0%.

A toy version you can run

To make the loop concrete I wrote a small simulation. This is not the paper's code, and its numbers mean nothing outside the toy. The fake model has fixed habits: it quits early a quarter of the time, re-reads files once context passes 4,000 tokens, and about one task in five is beyond it (a model limit). The harness is four knobs, one per component. The outer loop clusters failures, proposes one module patch from a playbook (standing in for an LLM optimiser), gates it on a validation split with a token cap and a did-the-code-run check, and deploys it if it passes.

harness_loop.pyPython
# harness_loop.py: a toy version of the diagnose -> propose -> validate -> deploy loop.
# Standard library only. The "model" is a fake with fixed failure habits, so it runs offline.
import random
from collections import Counter
from dataclasses import dataclass, replace

# ---- the harness: four editable modules, as plain config + flags -----------------
@dataclass(frozen=True)
class Harness:
    ctx_keep_turns: int = 99         # Context & State: how many past observations stay raw
    ctrl_max_turns: int = 12         # Agent Control: turn budget
    act_view_window: int = 0         # Action Interface: 0 = view_file returns the whole file
    ver_block_unverified: bool = False  # Verification & Recovery: no finish without diff + test

@dataclass
class Task:
    id: int
    file_tokens: int   # size of the file the agent has to read
    edits_needed: int
    model_can_solve: bool  # False = a model knowledge gap, not a harness bug

def make_tasks(seed, n):
    r = random.Random(seed)
    return [Task(seed * 100 + i, r.choice([1200, 2400, 4200]), r.choice([1, 2, 3]),
                 r.random() > 0.2) for i in range(n)]

# ---- the frozen "model": deterministic habits, no API ------------------------------
CONFUSION_TOKENS = 4000  # past this much context, the fake model starts re-reading files
QUIT_EARLY = 0.25        # chance per turn that it calls finish before it is done

def run(h: Harness, task: Task):
    """Inner loop. Returns (solved, tokens, trajectory signature list, hooks that fired)."""
    r = random.Random(task.id)   # same randomness for every harness version: a paired test
    history, tokens, events, fired = [], 0, [], set()
    read, edits, tested = False, 0, False
    for turn in range(h.ctrl_max_turns):
        if turn >= 12: fired.add("ctrl")     # a turn the seed's 12-turn cap would not allow
        # Context & State: older observations get compacted to a 60-token summary
        ctx = sum(t if i >= len(history) - h.ctx_keep_turns else 60
                  for i, t in enumerate(history))
        if h.ctx_keep_turns < 99 and len(history) > h.ctx_keep_turns:
            fired.add("ctx")
        tokens += 800 + ctx + 120            # system prompt + context + model output
        if r.random() < QUIT_EARLY and turn >= 2 and not (edits and tested):
            if h.ver_block_unverified:
                fired.add("ver"); history.append(80); events.append("finish_blocked"); continue
            events.append("finish_unverified"); return False, tokens, events, fired
        if ctx > CONFUSION_TOKENS and r.random() < 0.6:
            action = "reread"
        elif not read: action = "view"
        elif edits < task.edits_needed: action = "edit"
        elif not tested: action = "test"
        else: action = "finish"
        if action in ("view", "reread"):
            obs = h.act_view_window or task.file_tokens
            if h.act_view_window: fired.add("act")
            history.append(obs); read = True; events.append(action)
        elif action == "edit":
            edits += 1; history.append(150); events.append("edit")
        elif action == "test":
            tested = True; history.append(400)
            events.append("test_pass" if task.model_can_solve else "test_fail")
            if not task.model_can_solve:
                tested = False   # it keeps trying edits that cannot work
                edits = 0
        else:
            events.append("finish")
            return task.model_can_solve, tokens, events, fired
    events.append("turn_cap")
    return False, tokens, events, fired

def evaluate(h, tasks):
    res = [run(h, t) for t in tasks]
    solved = sum(s for s, *_ in res)
    avg_tok = sum(tok for _, tok, *_ in res) / len(tasks)
    fired = set().union(*(f for *_, f in res))
    return solved / len(tasks), avg_tok, res, fired

# ---- Stage 1: diagnose failures by signature, abstain on model limits -------------
def diagnose(tasks, results):
    clusters = Counter()  # one failed run can carry several signatures
    for task, (solved, _, ev, _) in zip(tasks, results):
        if solved: continue
        if ev.count("reread") >= 2: clusters["context_overflow"] += 1
        if ev[-1] == "finish_unverified": clusters["unverified_exit"] += 1
        if ev[-1] == "turn_cap" and not {"reread", "test_fail"} & set(ev):
            clusters["turn_cap"] += 1
        if "test_fail" in ev and "reread" not in ev:
            clusters["model_limit"] += 1      # tools were fine; do not patch the harness
    return clusters

# ---- Stage 2: propose ONE module patch per round (a stub optimizer) ---------------
PLAYBOOK = {  # signature -> candidate patches, in the order an optimizer might try them
    "context_overflow": [("control: raise turn cap 12 -> 18", "ctrl", {"ctrl_max_turns": 18}),
                         ("context: compact obs older than 2 turns", "ctx", {"ctx_keep_turns": 2}),
                         ("action: windowed view_file (400 tok)", "act", {"act_view_window": 400})],
    "unverified_exit":  [("verify: block finish without diff + test", "ver", {"ver_block_unverified": True})],
    "turn_cap":         [("control: raise turn cap 12 -> 18", "ctrl", {"ctrl_max_turns": 18})],
}

# ---- Stage 3: the gate --------------------------------------------------------------
B_TOK = 40_000  # per-task runtime token budget

def gate(base, cand, cand_fired, hook):
    if hook not in cand_fired:                 return False, "new code never ran on val"
    if cand["val"] < base["val"]:              return False, "regressed on val"
    if cand["tok"] > B_TOK:                    return False, f"avg tokens {cand['tok']:,.0f} > cap"
    if cand["dev"] <= base["dev"] and cand["val"] <= base["val"]:
        return False, "no gain"
    return True, "accepted"

def main():
    dev, val, test = make_tasks(1, 20), make_tasks(2, 20), make_tasks(3, 40)
    h, tried = Harness(), set()
    print(f"seed harness: {h}")
    for rnd in range(1, 7):
        d_score, d_tok, d_res, _ = evaluate(h, dev)
        v_score, v_tok, _, _ = evaluate(h, val)
        base = {"dev": d_score, "val": v_score, "tok": v_tok}
        clusters = diagnose(dev, d_res)
        print(f"\nround {rnd}: dev {d_score:.0%}  val {v_score:.0%}  tok/task {v_tok:,.0f}")
        print("  clusters:", dict(clusters.most_common()))
        target = next(((sig, p) for sig, _ in clusters.most_common() if sig != "model_limit"
                       for p in PLAYBOOK.get(sig, []) if (h, p[0]) not in tried), None)
        if not target:
            print("  no harness patch left to try: stop"); break
        sig, (name, hook, change) = target
        tried.add((h, name))   # a rejected patch may be retried once the harness changes
        cand_h = replace(h, **change)
        cd, _, _, _ = evaluate(cand_h, dev)
        cv, ct, _, fired = evaluate(cand_h, val)
        ok, why = gate(base, {"dev": cd, "val": cv, "tok": ct}, fired, hook)
        print(f"  propose [{sig}] {name}")
        print(f"  candidate: dev {cd:.0%}  val {cv:.0%}  tok/task {ct:,.0f}  -> {why}")
        if ok: h = cand_h      # Stage 4: deploy (merge into the default harness)

    for label, hh in (("seed", Harness()), ("evolved", h)):
        s, tok, _, _ = evaluate(hh, test)
        print(f"\nheld-out test, {label:7}: solved {s:.0%}  avg tokens {tok:,.0f}", end="")
    print(f"\nfinal harness: {h}")

if __name__ == "__main__":
    main()

Standard library only, Python 3.9 or newer, no API key. Here is the real output:

Bash
$ python3 harness_loop.py
seed harness: Harness(ctx_keep_turns=99, ctrl_max_turns=12, act_view_window=0, ver_block_unverified=False)

round 1: dev 40%  val 25%  tok/task 32,570
  clusters: {'unverified_exit': 12, 'model_limit': 2, 'context_overflow': 1}
  propose [unverified_exit] verify: block finish without diff + test
  candidate: dev 70%  val 65%  tok/task 63,556  -> avg tokens 63,556 > cap

round 2: dev 40%  val 25%  tok/task 32,570
  clusters: {'unverified_exit': 12, 'model_limit': 2, 'context_overflow': 1}
  propose [context_overflow] control: raise turn cap 12 -> 18
  candidate: dev 40%  val 25%  tok/task 41,474  -> avg tokens 41,474 > cap

round 3: dev 40%  val 25%  tok/task 32,570
  clusters: {'unverified_exit': 12, 'model_limit': 2, 'context_overflow': 1}
  propose [context_overflow] context: compact obs older than 2 turns
  candidate: dev 45%  val 25%  tok/task 13,728  -> accepted

round 4: dev 45%  val 25%  tok/task 13,728
  clusters: {'unverified_exit': 11, 'model_limit': 2}
  propose [unverified_exit] verify: block finish without diff + test
  candidate: dev 80%  val 75%  tok/task 19,910  -> accepted

round 5: dev 80%  val 75%  tok/task 19,910
  clusters: {'model_limit': 3}
  no harness patch left to try: stop

held-out test, seed   : solved 35%  avg tokens 25,092
held-out test, evolved: solved 85%  avg tokens 21,622
final harness: Harness(ctx_keep_turns=2, ctrl_max_turns=12, act_view_window=0, ver_block_unverified=True)

Four things in that run line up with the paper's argument, even at toy scale:

  1. Round 1: the verify hook is the right fix for the biggest cluster, but blocking early exits makes runs longer, and with uncompacted context they blow the token cap. A good patch can fail because another module is not ready for it.
  2. Round 2: raising the turn cap is the lazy fix, and the token cap rejects it, like the paper's control.py edit.
  3. Rounds 3 and 4: compaction barely moves accuracy but cuts tokens from 32.6k to 13.7k. With that headroom, the verify hook rejected in round 1 now passes. Module order matters, so retry rejected patches after the harness changes.
  4. Round 5: only model_limit failures remain, and diagnosis abstains instead of inventing a patch.

Little of this is "AI". The hard parts are the failure signatures, the paired comparison (same random seed per task across harness versions, so differences come from the harness), and the gate.

What I would take into a coding-agent setup

You do not need an automated optimiser to use this. Most of it applies to the harness you already edit by hand: CLAUDE.md, skills, hooks, tool configs.

1. Move rules that matter out of CLAUDE.md and into hooks

The RULES.md result is the clearest lesson in the paper. A rule that says "do not finish with an untested diff" did not stop the model from doing it. A ten-line check in verify.py did. In Claude Code the equivalent of the paper's verify.py is a Stop hook: it runs when the agent tries to end its turn, and exit code 2 blocks the stop and sends stderr back to the model.

.claude/hooks/pre_stop.pyPython
#!/usr/bin/env python3
# .claude/hooks/pre_stop.py: a pre-stop verifier. Runs when the agent tries to finish.
import json, os, subprocess, sys

event = json.load(sys.stdin)
if event.get("stop_hook_active"):
    sys.exit(0)  # we already blocked once and the agent is retrying; never loop forever

diff = subprocess.run(["git", "diff", "--stat", "HEAD"], capture_output=True, text=True).stdout
if not diff.strip():
    sys.exit(0)  # no code changed (it answered a question), nothing to verify

cmd = os.environ.get("PRE_STOP_TEST_CMD", "npm test --silent")
tests = subprocess.run(cmd, shell=True, capture_output=True, text=True)
if tests.returncode != 0:
    tail = (tests.stdout + tests.stderr)[-1500:]
    print(f"You changed code but `{cmd}` fails. Fix it before finishing.\n{tail}", file=sys.stderr)
    sys.exit(2)  # exit code 2 blocks the stop and sends stderr back to the model
.claude/settings.jsonJSON
{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "python3 .claude/hooks/pre_stop.py"
          }
        ]
      }
    ]
  }
}

Two differences from the paper's version, on purpose. An empty diff is allowed through, because in an interactive session the agent often just answers a question. And the stop_hook_active check lets it stop on the second attempt, so a broken test suite cannot trap it in a loop. I ran the script against a scratch repo (clean tree, failing tests, passing tests, the retry case) to check the exit codes.

2. Only add a rule after the same failure twice

The paper's first failure mode is prompt bloat: single-run reflection adds rules for slips that never repeat, and nothing gets pruned. Its safeguards: require repeated failures, prefer short "do not" rules, prune rules with no gain. I keep a small ledger next to the CLAUDE.md so each rule carries its evidence and how it is enforced. This shape is mine, not the paper's:

docs/agent-rules.yamlYAML
- rule: "Never finish with code changes that fail the test suite"
  evidence: [session 2026-11-03, session 2026-11-09]   # two separate failures, not one
  enforced_by: .claude/hooks/pre_stop.py               # a hook, not just text
  last_seen_helping: 2026-11-18                        # stale for a month? delete it
- rule: "Do not edit files under migrations/ unless the task says so"
  evidence: [session 2026-11-12, session 2026-11-14]
  enforced_by: text only
  last_seen_helping: 2026-11-14

3. Change one module at a time, and gate on tokens too

When a setup feels off, the temptation is to rewrite the system prompt, change compaction and add three tools at once. Change one component, re-run a fixed set of tasks, and keep it only if nothing regressed and tokens stayed in budget. Keep tuning tasks separate from judging tasks, even if that is 10 and 10 pulled from your own failed sessions.

4. Keep the grader, the logs and the memory out of reach

The paper's worst failure modes are the loop gaming its own evaluation: reading git history to recover the real fix, hard-coding visible test outputs, weakening assertions, deleting its own logs, or planting rules in memory that later disable checks like HTTPS verification. Its Table III boils this down to one invariant: the mutable harness must never control its own permission boundaries, grading harness or audit logs, and never grant itself more access than its parent had. If your agent edits its own skills or memory, keep the eval script and logs where it cannot write, and review its memory diffs like code.

Where this leaves me

The framing I am keeping: the harness is software, so treat its changes like software changes. Diagnose from many traces, not one. Change one module. Gate on held-out tasks, on cost, and on whether the new code ran. Keep the evaluator out of reach. The automated loop is early and the paper's experiment is small, but the manual version works today. And the RULES.md result alone was worth the read: if a behaviour matters, a line of text asking for it is not enough. Put it in code that runs every time.