Mutation harness

The mutation harness (src/core/eval/mutation-harness.ts) is the execution primitive for a kind: mutation invariant: it drives the configured coding agent to seed one small fault at a time into the project's own source, runs the project's own test command as the deterministic oracle, and classifies each seeded fault killed (the oracle now fails) or survived (the oracle still passes) — no external mutation-testing framework. It is the single place a mutation invariant's seed/detect/classify/revert sequence is implemented; nothing else in the codebase seeds or reverts a mutant.

Wired into evaluateInvariant. evaluateMutation (src/core/eval/invariant-evaluator.ts) runs this harness and folds its per-mutant outcomes into an InvariantOutcome with budget/threshold semantics: any survived mutant is a hard fail, and fewer than threshold evaluated mutants is unevaluable — mirroring how runWebLifecycle shipped standalone before web-deterministic-fold wired it into judgeCase — see eval invariant manifest.

evaluateMutation persists every mutant's diff and oracle output under .ratchet/evals/runs/<runId>/artifacts/invariants/<id>/ and memoizes the reduced outcome alongside it, keyed by (run.runId, invariant.id), so a survived mutant is replayable from disk and a repeated evaluation of the same run never spawns the agent a second time.

Overview

Contract

export async function runMutationHarness(
  invariant: MutationInvariant,
  cwd: string,
  deps: MutationHarnessDeps = {}
): Promise<MutationHarnessOutcome>;

• invariant — a resolved MutationInvariant (src/core/eval/invariants.ts): test (the user's own test command, the oracle), budget (positive integer ceiling on seed attempts), and threshold (not enforced by this harness — see Agent-neutrality note below).
• cwd — the working directory every git and test command runs in, and the cwd the spawned agent is launched in.
• deps — the injectable seams below; every field defaults to a real implementation, so a bare runMutationHarness(invariant, cwd) call is production behavior and tests override individual seams with fakes.

Sequence

1. Fail-closed precondition — deps.bash (default realBashRunner) runs the cleanliness probe git status --porcelain -- . ':(exclude).ratchet/evals/runs'. A non-zero exit (not a git repository, or git unavailable), a thrown call (git binary missing), or non-empty stdout (uncommitted changes) all return { kind: 'unusable-working-tree', reason } immediately — no agent is spawned and no test command runs. The probe excludes ratchet's own transient run directory (.ratchet/evals/runs): eval run persists the run record there before the invariant gate runs, so in a repo that tracks .ratchet/ that freshly-written record would otherwise count as a dirty tree and the mutation invariant could never evaluate. Any uncommitted path outside that directory still marks the tree unusable. (The pathspec is single-quoted because the probe runs through bash -c.)
2. Green-baseline precondition — checkOracleBaseline runs invariant.test through deps.bash once on the clean, unmutated tree and requires exit 0 before any mutant is seeded. This is the load-bearing anti-vacuity check: the oracle scores a mutant killed on a non-zero exit, so a suite already red on the clean tree would score every seeded mutant killed regardless of the fault, yielding zero survivors and a silently vacuous pass. A non-zero exit returns { kind: 'oracle-not-green', reason } immediately — no agent is spawned. The baseline run reverts unconditionally in a finally (the same scoped git reset --hard HEAD && git clean -fd -e .ratchet/evals/runs), so anything the test command writes cannot leak into the first mutant's git diff --cached and the tree is left exactly as clean as it was found. A baseline oracle that throws (its binary is missing) is deliberately not caught: it propagates to the caller as "could not run at all", distinct from "ran and was red".
3. Seed — for each of up to invariant.budget attempts, the harness builds a spawn request the same way judge.ts's buildVoteRequest does: RATCHET_EVAL_AGENT_CMD, when set, stands in for the agent binary (deterministic e2e testing); otherwise resolveAdapter(deps.agentName) resolves the configured coding agent's adapter and buildRequest builds the request, which deps.spawner (default realSpawner) runs. The override gate and spawn-request construction live in one shared helper (buildAgentSpawnRequest in src/core/batch/engine/agent.ts), shared with the batch engine and the judge, so the three spawn seams cannot drift apart. An active RATCHET_EVAL_AGENT_CMD is loud and auditable: eval run prints the one-line notice ⚠ agent overridden by RATCHET_EVAL_AGENT_CMD atop the scorecard, emits agentOverride: true in --json, and stamps via: "env-override" on the persisted run record. The instructions (buildSeedInstructions) ask the agent to make exactly one small, discrete edit to a non-test source file and to not run the test suite itself.
4. Detect — git add -A (stages tracked and untracked changes, so a new file the agent created is not silently invisible) followed by git diff --cached captures the fault as a unified diff. An empty diff means the agent made no change this attempt: nothing is recorded as a mutant, the oracle is never run for it, and the loop moves to the next attempt.
5. Run the oracle, classify — a non-empty diff runs invariant.test through deps.bash, the same seam judgeCheck and evaluateDeterministic already shell out with. exitCode === 0 classifies the mutant survived; any non-zero exit classifies it killed. The mutant's index (the 0-based attempt number), diff, outcome, and testResult are recorded.
6. Revert, unconditionally — the per-attempt body (seed → stage → diff → oracle) runs inside a try, with git reset --hard HEAD && git clean -fd -e .ratchet/evals/runs in the matching finally. The revert therefore runs on every path — after a mutant is classified (killed or survived), after an empty-diff attempt (a harmless no-op), and even when the spawner or oracle throws (the tree is reverted before the error propagates). This restores both tracked and untracked state before the next attempt, so every attempt after the first runs against the unmutated project and a seeded fault can never survive a mid-attempt failure in the user's working tree. The clean excludes ratchet's own transient run directory (-e .ratchet/evals/runs), matching the working-tree probe's exclusion: git clean already skips gitignored files, so the common path (a repo where ratchet init gitignored the runs dir) is unaffected, but the explicit exclusion also preserves the in-progress run record in a repo that tracks .ratchet/ without gitignoring the runs dir — the seeded mutant is still fully removed, only ratchet's own transient records are kept. The excluded path is derived from the same .ratchet/evals/runs constant as the probe, so the two cannot drift.
7. Return — once invariant.budget attempts have run (or ended early via either precondition), the harness returns { kind: 'completed', mutants }. mutants may be shorter than budget when one or more attempts produced an empty diff.

Injectable seams (MutationHarnessDeps)

export interface MutationHarnessDeps {
  bash?: BashRunner;
  spawner?: Spawner;
  agentName?: string;
}

Field	Default	Purpose
bash	realBashRunner (src/core/batch/engine/proof-of-work.ts)	Runs the cleanliness probe (git status --porcelain -- . ':(exclude).ratchet/evals/runs'), git add -A, git diff --cached, invariant.test, and the revert command (git reset --hard HEAD && git clean -fd -e .ratchet/evals/runs, whose clean exclusion mirrors the probe).
spawner	realSpawner (src/core/batch/engine/agent.ts)	Spawns the coding-agent subprocess built by the resolved adapter's buildRequest.
agentName	the engine's default agent	Selects which registered adapter (resolveAdapter) builds the seed request; unset resolves the configured default, exactly like JudgeDeps.agentName.

All defaults are production-real; every field is independently overridable with a fake so tests never shell out to a real git command or spawn a real coding agent.

MutationHarnessOutcome

export interface MutantOutcome {
  index: number;
  diff: string;
  outcome: 'killed' | 'survived';
  testResult: BashResult;
}

export type MutationHarnessOutcome =
  | { kind: 'unusable-working-tree'; reason: string }
  | { kind: 'oracle-not-green'; reason: string }
  | { kind: 'completed'; mutants: MutantOutcome[] };

• unusable-working-tree — the fail-closed cleanliness precondition tripped; reason names why (an uncommitted change outside .ratchet/evals/runs, not a git repository, or git unavailable). A freshly persisted run record under .ratchet/evals/runs does not trip it. No mutant was seeded.
• oracle-not-green — the green-baseline precondition tripped: invariant.test exited non-zero on the clean, unmutated tree, so classifying mutants would be vacuous (every mutant scores killed regardless of the fault). reason names the test command and its baseline exit. No mutant was seeded; the baseline run reverted itself. evaluateMutation maps this to unevaluable (never pass). A baseline oracle that throws is not represented here — it propagates as a thrown call, so the evaluator records "harness could not run" instead.
• completed — the budget-bounded loop ran to completion. mutants holds one MutantOutcome per attempt that actually seeded a fault (a non-empty diff), in attempt order; an attempt whose diff was empty contributes no entry. MutantOutcome.index is the 0-based attempt number within the budget-bounded loop (not re-indexed past skipped no-diff attempts), so a mutant's position in the loop is traceable even when earlier attempts were skipped.

MutationHarnessOutcome is intentionally not an InvariantOutcome itself — reducing it into that shape, and enforcing threshold (the floor on how many mutants must be evaluated), is mutation-evaluator-fold's job. This harness stays independently testable in isolation from that reduction.

Agent-neutrality

Every seed request is built through resolveAdapter(deps.agentName).buildRequest(...) (or the RATCHET_EVAL_AGENT_CMD test override, gated through the single shared buildAgentSpawnRequest helper) — the same adapter registry and spawn seam judge.ts's llm-judge binding uses. An active override prints the one-line ⚠ agent overridden by RATCHET_EVAL_AGENT_CMD notice, emits agentOverride: true in --json, and stamps via: "env-override" on the run record. There is no agent-specific branch anywhere in this module: runMutationHarness never checks which coding agent is configured before seeding, satisfying the multi-agent-support standard by construction.