Replay benchmarks — method and records

Can a model fix the bugs we actually had? A replay benchmark takes closed issues from our own repositories, rebuilds each repository as it was just before the fix (no git history, no network), gives an agent only the issue text, and compares the patch it produces with the fix that was merged. This package holds the code of such a study: the record types and the live cockpit. The data lives in the spaces that use them, never in this repository.

The split: code, measurements, publication

The harness that produces the measurements (agent loop, runner, judge, hidden-test grader, task selection, report) is in the repository under devtools/issue-replay/ — its README describes how to run it.

The method

• Tasks. Closed issues, stratified by the size of the real fix (easy ≤ 30 changed code lines, medium ≤ 250, hard above), with a fixed seed. Wave 1 is a stratified subset every model runs first.
• Agent. A minimal tool loop (bash, read_file, write_file, edit_file, finish) that reads the repository's AGENTS.md first. Every model gets the same tasks, tools and budget.
• Harness versions. The budget and the history kept are part of the measurement, so every record names its harnessVersion, maxSteps, maxOutputTokensPerCall, wallBudgetSeconds, contextWindow and historyCharsKept.
• Grading. A judge model compares the candidate patch with the merged fix (resolved, partial, wrong, no_change; error when the judge failed). Hidden tests — the real fix's tests re-run on the candidate — go into hiddenTestResult as they become available.
• Contamination. The harness guards the sandbox on a best-effort basis; a post-hoc scan of every executed command's output is the second line. A run whose output reached outside its snapshot (a real clone, another run's workspace or results, harness files, a read outside the snapshot) lists those steps in contamination and is judged excluded — never graded and left out of every resolve-rate denominator. The rule is strict: a directory listing counts.

How a run reaches the space and gets its verdict — no person involved

Mac (runner.py → push.py) ──signed POST──▶ {portal}/api/hooks/{space}/Ingest ──▶ {space}/Ingest/_Inbox
                                                                                   │ RunIngest hub: verify, land
                                                                                   ▼
                                   {space}/Runs/{model}__{taskId}__{harness}  (+ Patch, Summary)
                                                                                   │ same hub: judge
                                                                                   ▼
                                            judgeVerdict … judgedAt written back onto the row

1. The Mac pushes its own row. When devtools/issue-replay/runner.py finishes a run it queues one delivery (push.py): the row derived on the Mac from result.json, task.json, candidate.diff and trajectory.jsonl — timings, tokens, tool use, patch size, contamination, the host — plus the patch (cut to fit the inbox's 1 MB cap, and marked) and the agent's summary. It is signed with the Mac's OWN heartbeat key (Hosting__ModuleReportSecret in ~/.memex-local/values.local.yaml, the value control holds for the machine) as X-Hub-Signature-256, and stays in <root>/outbox/ until the portal answers "signature":"verified". On start the runner also queues every finished run the space has not accepted yet, so a Mac that gets its key later catches up by itself. No personal token, no laptop.
2. The ingest verifies and lands it. The space's Ingest node (Benchmarks/RunIngest) is the webhook target. Its hub accepts a delivery only when exactly one per-host key under secretConfigKey (default Hosting:ModuleReportSecret, i.e. Hosting:ModuleReportSecret:{host}) verifies it, that host is listed in senders, and the body and the row both name it; the id must be the row's own {model}__{taskId}__{harness}. The section's own fleet-wide value never authorises a delivery. A retry of the same run updates the same row with its measurements replaced and its verdict kept; a RE-RUN (a new finishedAt, i.e. a new patch under the same id) clears the verdict so the new patch is judged again, and a verdict computed for the old run is never written onto it. A contaminated run is excluded at landing. A signed body of the wrong shape is refused and deleted, never replayed.
3. The same hub judges it. Every finished row without a verdict gets ONE tool-free call on the declared judge model (judgeModel, else the instance's judgeTier tier) with judge.py's prompt — the "data, not instructions" framing included — at most judgeDailyCap calls per UTC day; a failed call is an error ATTEMPT, retried after 15 min, 30 min, 1 h, … up to judgeMaxAttempts, and never counted as a result; a contaminated run is excluded and an empty patch no_change without a call. The reference fix is read from GitHub through the instance's App with a token scoped to the one repository, read-only: the issue text (never stored) and baseSha...mergeSha from the task's BenchmarkIssue, lock files omitted. The verdict is written back through the row's owning hub with judgedAt.
4. The ingest stays awake. A webhook delivery is stored in {ingest}/_Inbox without activating the ingest's hub, and an idle hub retires. The package therefore anchors Benchmarks/Ingests (Benchmarks/IngestKeeper, anchors in the package root): its hub lives as long as the portal, finds every Benchmarks/RunIngest node and wakes each every 3 minutes, so a run posted at night is landed and judged within minutes.
5. The cockpit is live. It binds to the rows, so a landed run and its verdict appear without a republish.

What the instance must DECLARE for this (its deployment record, never by hand): the ingest node path as a WebhookInbox target with SecretConfigKey: Hosting:ModuleReportSecret, and one Hosting__ModuleReportSecret__{host} mount per Mac from the same vault object control mounts for it. A Mac with no heartbeat key queues its runs and sends nothing until it is paired.

The records

Benchmarks/BenchmarkRun — one per (model, task, harness version); node id {model}__{taskId}__{harness} (model with : and / replaced by _), so a re-upload updates the same node and never duplicates it. A run appears as soon as it finished, with judgeVerdict empty, and is updated when its verdict lands.

Benchmarks/BenchmarkIssue — one per task: taskId, order (published order, wave 1 first), repo, issueNumber, fixPrNumber, tier, wave, codeLinesChanged, codeFilesChanged, filesChanged, linesAdded, linesRemoved, fixHasTests, originalFixModel (from the fix commits' Co-Authored-By trailers), commits, issueToMergeHours, prToMergeHours, mergedAt, baseSha, mergeSha. No issue titles or bodies: they may name clients, and the record type has no member that could hold one (a test pins that).

Benchmarks/Cockpit — content runsPath, issuesPath, hostsPath. Area Dashboard (default): KPI tiles (resolve rate per model), the verdict mix as stacked bars, resolve rate, median wall time, output tokens per minute, steps and budget exhaustion, wall time against completion tokens, model time next to the original fix's PR-open-to-merge time, a per-model table and the model × issue matrix in task order. Area Runs: every run in one grid. Both filter by model, tier, wave and harness version and update as records arrive.

Caveats a reader must keep

• One run per model and task: a single cell is noisy; read rates over a wave.
• The judge is a model. Hidden tests are the objective check and arrive later.
• Original-fix times are calendar time, not effort, and the original authors had context the issue text does not carry.