What the retry steward will re-run, and what it will never re-run

scripts/retry-known-transients.py re-runs a red Plugin Catalog CI run once, and only when every failed job is proven to be infrastructure. It is not a flake-retrier, and nothing in it may become one: re-running a red until it passes is how a real race gets shipped.

This page is what a reader needs before adding an entry to one of its lists.

The four lists, and what a match MEANS

Precedence inside classify is INFRA → VOID → LOG → verdict. INFRA is first because a log can carry both a blip and the quota refusal that actually failed the job, and retrying into a wall the retry cannot clear is worse than declining. VOID is next because in a void run the other reds are not verdicts at all.

The fourth shape: a run that never started (#2077)

Resolve the released platform is the run's FIRST job and the SOLE source of the core commit every other job compiles, gates and tests against. When it refuses — which it does by design, raising a ResolutionError whose text explains itself — nothing in the run has been checked out, compiled or executed yet. The run did not fail. It never started.

The refusal measured on 2026-09-18 (PR #2071, run 35328406174 attempt 1, and again within the hour on PR #2043, run 35334000904) is GitHub serving a stale page 1 of core's main-cd.yml run listing — ~2,600 runs behind: its newest was main-cd #6215 while this repository's own main had already passed on core CD #8892. The resolver's own error text ends:

Re-run this job; the resolver refuses rather than re-reading, because this red is the harmless answer and a silently old platform is not.

Before #2077 the steward classified that job, and the 17 jobs that died behind it, as UNPROVEN, and declined — "18 failed job(s) not proven transient — no retry. Investigate it." The one red whose own message says re-run this job was the one the retry lane would not re-run. #2071's hand re-run then went fully green with no code change, which is the proof the refusal was transient.

🚨 A void is proven at the ROOT and nowhere else

The tempting shape — teach the steward each cascade sentence — is wrong, and the run says why. One refusal produced 17 downstream reds wearing TEN distinct failed-step names:

Three of those are sentences a real red also produces. Naming them individually would teach the steward to excuse "an artifact is missing" and "a gate did not run" — permanently, for every run. So a void is claimed only by the NAMED job, failing on its OWN step, with the NAMED refusal in its own log; everything else failed in the run is then reported as that refusal's cascade, which is a consequence, not a proof.

🚨 A void excuses only what DEPENDS on it

Raised in review on #2123, and it was a real hole: the first version of this change treated every other failed job as cascade. Parsing ci.yml's needs: graph says why that is wrong —

Any of those 7 can carry a genuine red beside the resolver's refusal, and a full re-run would hand it a second roll of the dice — the one thing this steward exists to prevent. So the steward derives the closure from the workflow itself and declines, naming the job, when a failed job is not in it. Of the 18 failed jobs in the reference run, 0 are independent, so the measured case still re-runs.

Everything unknown resolves the safe way. A job name that cannot be mapped to a ci.yml job is INDEPENDENT, so a parser gap makes the steward decline (the status quo) and can never make it over-retry. Measured: 2 of the 38 job names in the reference run are unmappable — Live runner on aks-silos and -dind, whose name: is an unexpanded ${{ vars.MW_RUNNER_HEAVY || 'aks-silos' }} — and both are independent jobs anyway, so failing closed gives them the right answer for the right reason. An unreadable ci.yml declines too.

The mapper handles the three shapes the API produces, each asserted against a real payload: an exact name:, a reusable lane <caller> / <inner>, and a matrix <name> (<values>) — with exact-name first, because RN app + web clients (typecheck + test) is a literal name containing parentheses. Today's ci.yml cannot tell the two orderings apart, so that rule is proven on a synthetic workflow built to make them disagree.

🚨 And a void run is re-run IN FULL

POST /runs/{id}/rerun, never rerun-failed-jobs. Two structural reasons:

1. The cascade contains skipped jobs, and a failed-jobs re-run does not bring them back. In the measured run Required CI inputs was skipped because platform-ref failed — and Every gate executed failed on exactly that skip ("these gates SKIPPED rather than running: preflight"). GitHub's own wording for the endpoint is "all of the jobs that failed and their dependents", which reads as though a skipped dependent would come back. It does not, and the discriminating case is this steward's own history — run 35371177496, where it POSTed rerun-failed-jobs on 2026-09-18:

   job	attempt 1	attempt 2
   Module bundles / Module bundle (batch 6/8: …)	failure	success
   Module bundles / Module bundle (batch 4/8: …)	failure	success
   Module bundles / Module tests (…)	skipped (its need failed)	still skipped

   The need that caused the skip was re-run and went green, and the skipped dependent still never ran. Across the two real rerun-failed-jobs re-runs in the steward's last 60 runs, 11 of 11 jobs skipped in attempt 1 were skipped or absent in attempt 2. So the retry would reproduce Every gate executed's red and waste the one attempt.

2. Nothing in attempt 1 is worth preserving. Every job that could have said anything about the change was waiting on a platform that never resolved.

The one-attempt guarantee is unaffected: retry-known-transients.yml fires only on run_attempt == 1, and a full re-run creates attempt 2.

What the stale-listing signature matches, and what it does not

It matches ##[error]GitHub served a STALE run listing (MeshWeaver#4433): page 1 of — the ##[error] prefix included, so it is the refusal the job raised, never prose that mentions it.

It deliberately does not match:

• The resolver's own self-test banner. Every resolver job — green ones included — prints "a run listing whose page 1 is provably STALE … is refused rather than resolved from". A pattern loose enough to match that banner would void every resolver red there is. The self-test's negative control is the real captured log with its one ##[error] line removed and the banner still standing.
• The resolver's other refusals: a set below the declared floor, a freeze naming an unsealed set, an unreadable main ceiling, an API that failed on every attempt. None is cleared by a re-run, and the last one also ends with "Re-run this job" — which is why the signature keys on the sentence, not on the remedy.
• The same sentence in any other job. The job name is matched first, anchored, and must be a job ci.yml declares.

But the same refusal can be raised elsewhere, and that is covered separately. ci.yml runs resolve-platform.py from FOUR places: platform-ref, and the Re-resolve the platform — a re-run must not test a stale set step of test-drift, rn-app and memex-template. A stale listing served to one of those fails that one job while the rest of the run is fine — so the run is not void, and the ordinary one-retry is right. The same sentence therefore also sits in LOG_SIGNATURES, job-agnostic; VOID is checked first, so the root still voids the run and a re-resolve elsewhere gets rerun-failed-jobs. Covering only the root would have been a fix landed on one of four call sites.

What an entry must carry

A RUN_VOID_SIGNATURES entry is admissible only if, and the self-test asserts each:

1. its job name is anchored (^…$) and contains no regex metacharacters — a void names ONE job, never a pattern;
2. that job is declared in ci.yml under that exact name;
3. its log pattern names the filed incident it was measured on;
4. the sentence it keys on is still one scripts/resolve-platform.py raises — a reword there would otherwise retire the signature silently, the same defect shape as a renamed verdict step.

LOG_SIGNATURES entries must name the host they blame or the incident they were filed under. A bare dial tcp …: connect: connection refused names neither, which is exactly why it also matched a suite's own Testcontainers Postgres and was removed.

How a signature is proven

Never from the issue's paraphrase. The measured refusal did not read the way #2077 quoted it: the live text came from the ceiling branch ("its newest run is main-cd #6215, but this repository's main has already passed on core CD #8892"), not the 12-hour-age branch. The signature is written against a verbatim captured log (scripts/testdata/retry-known-transients/job-105546810757.log), and the self-test drives:

• the positive case — the real log, the real job → VOID, and the run → rerun;
• the negative control on the other side of the change — the SAME log with its one ##[error] line removed → UNPROVEN, and the run → no retry, which is the pre-fix behaviour intact;
• the refusal in a different job → TRANSIENT, rerun-failed-jobs, never VOID;
• a step-less failure carrying the log → never VOID;
• INFRA alongside a void → declines, and says the void it is holding back.

Every one of those arms is mutation-tested — 21 mutations, 21 reds, with a positive control on the harness itself: removing the signature, loosening it to STALE, un-anchoring or renaming the job, dropping VOID's precedence over UNPROVEN or letting it beat INFRA, downgrading the remedy to a partial re-run, reaching VOID without a failed step, dropping the cascade label, restoring the excuse-everything cascade, treating an unmappable name as a dependent, computing the closure as "everything", trying the matrix rule before the exact name, breaking each needs: spelling, dropping the lane-prefix rule, reading the display label as a job name, and unwiring steward from the names it collects.

🚨 Two traps this change walked into, both worth remembering

1. A green self-test over a broken live path. The dependency fix was written, every arm passed — and the live steward then declined the very run the feature exists for. steward puts describe(job) ("<name> · runner …, completed …") in the tuple, while the self-test's fixture used the bare name in that slot, so the label/name distinction did not exist in the test. Caught only by running the real steward against the real run. The fixture now carries the label, and a VOID with no bare names declines loudly rather than resolving the wrong string.
2. A guard that matched its own source. The source-level check on steward's wiring first grepped the whole file for the string it asserts — and found the assertion itself, so it could never fail. It is now scoped to steward's own body, with a check that the slice was found at all.

🚨 The root cause this does NOT fix

A retry signature is the right answer to "this red is not about the change". It is not the answer to "why does GitHub serve a listing 2,600 runs behind", and it does not claim to be.

resolve-platform.py already retries a 5xx and a rate limit, bounded, inside fetch. It does not re-read a stale-but-200 listing, deliberately — the refusal says so in the code ("the resolver refuses rather than re-reading, because this red is the harmless answer and a silently old platform is not") and the measurement recorded beside it says "a re-read minutes later was correct". So a bounded re-read with a short backoff, on the same evidence, would clear this class inside one job in seconds instead of costing a whole CI cycle and a steward round — and would leave the refusal in place for a listing that is still stale after the re-read.

That is a change to a decision the resolver's author made explicitly, in the failure's own text. It belongs in its own change with its own measurement, not folded into a retry signature, and it is filed as MeshWeaver#4750 — including the two things that change has to settle: that the refusal lives byte-identically in core's .github/scripts/ copy AND in this repo's scripts/ copy (the one ci.yml actually runs), and that the ceiling branch is the one with a crisp success condition to re-read for.

Until then the steward's full re-run is the honest remedy — and it is honest precisely because it is ONE attempt: a listing still stale on the second read stays red. If #4750 lands, this signature stays useful as the backstop for exactly that case.