Review Findings Answered

The check Automatic review answered BLOCKS until the automatic review has landed on a pull request and every inline thread that review opened has a reply from a person. While that is merely in progress — threads not answered yet, a review not landed yet, inside their deadlines — it reads pending, not red (policy review-pending-while-answering; see Pending while answering), and it goes red only on a real failure. It never skips, it never passes on no evidence, and a pull request the reviewer could not review is released only by a maintainer's visible waiver. With no finding on the pull request, the only evidence of a review is a Copilot review that COVERS the head — of the head itself, or of an earlier head when only main was merged in since (policy review-on-real-change) — see Reviewed, nothing found. It is the build of option B on #4299, decided by the maintainer on 2026-09-17.

• Workflow: .github/workflows/review-answered.yml
• Predicate and self-test: .github/scripts/check-review-answered.py
• Status: 🚨 REQUIRED on main since 2026-09-17 — measured that evening on ruleset 2128472, which now lists Automatic review answered beside Consolidate test results. The Rollout section below is kept as the record of how it got there; it is no longer the current state.

🚨 What being required FEELS like, because it is not obvious from the outside. An unanswered thread leaves the pull request mergeable_state: blocked with every build green — and blocked is the same word REST returns for a pull request merely waiting its turn in the queue. Three pull requests sat green, armed and silently OUTSIDE the merge queue for ~45 minutes on the evening this landed, and nothing in the REST view distinguished that from progress. If a green, armed pull request is not merging, read this check before anything else, and read the merge queue ITSELF rather than mergeable_state — the queue is one of the two things REST cannot express:

gh api graphql -f query='{repository(owner:"Systemorph",name:"MeshWeaver"){
  mergeQueue(branch:"main"){entries(first:20){totalCount nodes{position state
  pullRequest{number}}}}}}'

A totalCount that does not contain your pull request, while other pull requests merge through it, is the reading that separates "held" from "waiting".

🚨 It is answered by replying ON the thread, and nothing else does it:

gh api "repos/Systemorph/MeshWeaver/pulls/<n>/comments/<comment-id>/replies" -f body='…'

🚨 Quote the path. Unquoted, the shell reads <n> as a redirection and the command fails before gh runs — which looks like a broken instruction rather than a quoting mistake.

🚨 Answering every thread used to be NECESSARY, NOT SUFFICIENT — the check now finishes the job itself. Branch protection reads the check's pull_request run, not the pull_request_review_comment run your reply starts, so the verdict it read stayed an older failure on the same head while the newest run said GREEN, until somebody re-ran the pull_request run by hand. The lane's refresh job now does that re-run on its own after every GREEN verdict taken on a different event. The measurement, the self-refresh, and the manual fallback for the cases it cannot cover (a fork, or a red refresh job) are below, under "The check's log says GREEN and the pull request is still BLOCKED".

Why: a review was advisory

The ruleset main pr protection (2128472) carries a copilot_code_review rule, so every pull request into main is reviewed. The control plane (PrArming) arms a pull request once its review lane is satisfied; auto-arm.yml only DISARMS on a push. Nothing joined the two, so a merge waited for the required checks and never for the review — and on the pull requests whose checks are fast, the checks usually won.

The instances on the filing (#4269, #4291, 2026-09-14) were admitted before the reviewer had finished; condition 1 below holds those. In every miss measured after it — #4310, #4343, #4347, #4366, #4371, #4383 — the review had landed; it had not been read. A check that waits only for the review to complete (option A) would have held none of those, which is why option B asks whether each finding was answered.

The rate at the time this page was written, replaying the predicate over 60 merged pull requests (2026-09-16T08:48Z → 09-17T07:41Z), each as of its own merged_at: 32 of 60 would have been red at merge, all of them for unanswered threads; 126 findings, 86 of them unanswered at merge. None was red for a missing review. That is the cost of making the check required, stated before anyone decides to.

The rule

All three hold, or the check is RED:

1. The automatic review has landed — a review by the reviewer account, at a non-PENDING state, whose body is not a refusal. What makes it a review is who posted it, not how it is worded — see "Provenance, not presentation" below for why, and what requiring a recognisable shape cost. When the pull request carries no finding at all, the review must be a completed Copilot review of the CURRENT head — see Reviewed, nothing found. One thing releases this condition — and only this one — without a review: a maintainer's waiver. (A generated-only App pull request is not owed a review; that is a provenance rule, not a claim that it was reviewed.) And the review must COVER the head (policy review-on-real-change): a Copilot review of an earlier head counts only when every commit since it is a merge of the default branch that left the pull request's own diff unchanged. A head whose own diff changed owes its own review — see A review on every real change, none for a merge of main.
2. Every thread the reviewer started has a person's reply — for every comment by the reviewer with no in_reply_to_id, at least one comment in that thread by an account of type: User (following in_reply_to_id to the root, so a reply to a reply counts).
3. The inputs were read completely — every read succeeded, and the comment listing is not shorter than the review_comments count the pull request reported before the listing began. Measured on 60 of 60 recent pull requests: listing and count agree.

A reply that says nothing still counts as answered. That limitation is known and accepted on #4299; the predicate is still strictly better than none.

Pending while answering — red only on a real failure

Policy review-pending-while-answering, with its deadlines under policy review-answer-deadline. Measured on the day it was decided: about 60% of the red pull requests were this check reading RED while Copilot's threads simply had not been answered yet. That state is not a failure. Nothing but a person's reply changes it, and a red X sends everybody to investigate it. So the required context now has three readings:

🚨 It still blocks exactly as before. A pending verdict is a check-run with status in_progress and no conclusion. Branch protection and rulesets accept only success, neutral or skipped for a required context, so in_progress is never satisfied, and auto-merge waits for it like any pending required check. The writer never posts neutral or skipped. The self-test pins that (a PENDING body is in_progress with NO conclusion), and mutating the body to neutral turns four cases red.

How a workflow can say "pending". A job's own check-run always ends with a conclusion when the job ends, and every job is capped at 45 minutes. So the verdict is written through the Checks API instead (check-review-answered.py --verdict-check-run NAME, run_verdict):

• One check-run per head, named exactly like the required context. The first evaluation of a head creates it, and every later evaluation PATCHes it, whatever event started that evaluation. This also removes the duplicate problem described under The check's log says GREEN and the pull request is still BLOCKED: there is now one verdict per head, not one per run.
• The judging job is renamed (Review answered — verdict writer), so its own check-run never carries the required name. It goes red only when the verdict is failure.
• The newest read stands. The check-run's external_id records when the writing evaluation read the pull request (mw-review-answered:v1:<read time>:<kind>). An evaluation whose read is older than the one on file writes nothing (verdict_write_plan).
• Reopening a completed verdict. When a completed verdict must become pending again (for example, a new thread on an answered head), the writer re-opens it. If GitHub refuses, the old check-run is closed as failure with title Superseded and a new pending one carries the verdict. An old success therefore never stands beside a newer pending verdict.
• GITHUB_TOKEN writes it, so the check-run belongs to the GitHub Actions App (id 15368). That is the App MeshWeaver.Plugins' classic protection pins its review context to.

What re-judges a pending verdict with no event. Copilot's own review event cannot start a run here (see below), and a deadline passes without any event at all. So the caller's schedule (every 15 minutes in core) runs --verdict-sweep. The sweep re-judges every open, non-draft, same-repository pull request whose verdict on its current head is pending or unmeasured (SWEEP_KINDS). Every other pull request costs one listing read. That is also why the writer takes no review wait and no settle wait: a pending verdict is not red, so there is nothing to wait out, and the sweep flips a clean review to green within 15 minutes.

Where nothing changed. The job keeps the required name and its old red/green behaviour in these cases:

• a fork pull request, whose token is read-only and so could write no verdict;
• a Dependabot pull request, whose token is read-only unless granted;
• a merge-queue entry, whose commit is judged once, at admission;
• every caller of the fleet lane that has not opted in.

One condition picks the shape. It is written twice in each workflow, in the job's name: and in VERDICT_CHECK_RUN, and the self-test asserts the two are identical. There is still no job-level if:: the shape is chosen by name, never by skipping a job, because a skipped required context counts as satisfied.

The refresh machinery stays coherent. An answered state still flips the context to success, directly: the reply's own run PATCHes the head's verdict to success, whichever event it was. The lane's refresh job runs only after a success verdict and prints why it does nothing on a pending one. After a success it still re-runs every older red of the job-shaped check on the head. That covers a pull request that was open across this change and still carries job-shaped reds from before it. The outage marker step and the retry listener are unchanged: an outage takes no verdict and writes nothing.

Satellites opt in, per repository. The fleet lane (node-repo-review-answered.yml) has an verdict-check-run input, empty by default, which gives exactly the old behaviour. A repository opts in with one change to its caller:

1. pass verdict-check-run: '<caller job> / Automatic review answered', exactly as its protection lists the context;
2. grant checks: write on the uses: job;
3. add a schedule: trigger (the sweep);
4. update its row in .github/lane-caller-grants.yml (core first, as pending:).

Answering automatically (policy review-thread-auto-answer). When Copilot's threads land, the control plane's babysitter pass queues ONE agent run per (pull request, head, review round). The pass is fed by the org webhook, and no model polls. The run answers the findings that are not defects ON their threads, through the App. It leaves every real defect unanswered, so this check stays pending until a fix with a test lands. The lane is off by default, and its model and effort are deployment-record config. Its manual is MeshWeaver.Plugins Hosting/PrThreadAnswerer.

🚨 That lane's answers are posted by a Bot. Under policy app-marked-answer-counts, condition 2 counts a reply from the control plane's App ONLY when it carries the lane's marker (<!-- pr-relay:answer comment=<root id> -->) naming that thread's root. The App is matched by account id or either login (systemorph-com[bot], systemorph[bot]). An unmarked App reply, a marker naming another thread, and a marked reply from any other bot still answer nothing (is_app_answer, self-tested).

🚨 Why the lane's judging job has no permissions: block any more. The writer needs checks: write. A called job's permissions: is a hard requirement on every caller. Writing checks: write there would startup-fail every caller that has not granted it, and five of six call the lane at @main. That is the #3933 outage shape. Without a block, the job holds exactly what its caller grants on the uses: job, which the roster records per caller.

Controls. The self-test's pending: cases run against a fake GitHub:

• the four the decision named: unanswered and fresh → pending; past the deadline → failure; all answered → success; unmeasurable → failure;
• the review deadline, the refusal, an unrecognised reason and an unreadable clock;
• the write plan: create, patch, and skip on a newer read;
• the re-open fallback;
• the sweep, with a draft, a fork and a completed verdict as negative controls;
• the workflow condition parity.

Watched failing under five mutations:

Not established — measure it on the first pull requests.

• Whether the rollup reads the API-written check-run. The rollup that branch protection reads left out the check-runs of pull_request_review_comment runs on #4649, and nobody measured why. The verdict check-run is created by whichever evaluation runs first on a head, usually the pull_request run of the push. Read the pull request's rollup on the first pull requests after this lands. A pending verdict must show the merge as blocked, and a success must clear it.
• Whether a ruleset blocks on an absent required context. Read the four rows below in order.
  1. Classic protection blocks on an absent context. This was measured on Plugins#1453.
  2. Whether a ruleset blocks is not measured. Reading CI Signals records it as satisfied.
  3. The writer creates the context on its first evaluation, a few seconds after the event. The window before that is shorter than the job-shaped check's queue time. Every required suite takes far longer than that window.
  4. An outage that keeps even that first write from landing leaves the context absent until the retry listener re-runs it.

Reviewed, nothing found — the only evidence that counts

Policy review-evidence-any-head answers the question #4730 left open: "0 unanswered threads" is produced by two opposite states, reviewed and nothing found and never reviewed, so what evidence separates them?

Only a completed Copilot review object on the pull request with zero findings — on ANY of its heads. In the check that is copilot_review_of_pull_request(reviews, head): a review by the Copilot account (id 175728472, type Bot) at a non-PENDING state, whose body is not a refusal, against the current head or any earlier one, and no reviewer thread anywhere on the pull request. A re-review on every push is not required: a clean review followed by a push — a fix, or a merge of main — stays reviewed.

With findings, the threads prove a review happened — but since policy review-on-real-change they no longer stand in for a review of a head whose own diff changed after them: such a head is red until its own review lands, and that review's threads are answered like any other.

Why not the current head only. For one day the gate required a Copilot review of the CURRENT head (policy review-evidence-current-head, now superseded). The rulesets then reviewed once per pull request (review_on_push: false), so every clean pull request pushed again after its review — most often by the merge of main that the merge-before-push rule requires — went RED — NO REVIEW OF THIS HEAD until a maintainer re-requested the review by hand. Policy review-evidence-any-head replaces it: a Copilot re-review is not required on every push. Never-reviewed stays red, which is the state #4730 was about.

Controls. The self-test's EVIDENCE cases pin each row above with a control beside each case. Cut copilot_review_of_pull_request back to the current head (copilot_review_on) and the earlier-head case, the refusal-then-earlier-review case and the person's-lock-PR case turn red; widen it to any reviewer and the internal-reviewer cases turn green and fail.

The reviewer, as measured

Measured 2026-09-17 over 50 merged pull requests (#4487–#4568) through the REST endpoints the check uses:

The App is being renamed systemorph-com → systemorph, in three steps — (1) every check accepts both spellings, (2) the App is renamed in the organisation's settings, (3) the old spelling is retired. A rename changes the App's slug and its bot login (systemorph[bot]); the App id (4918443) and the bot account id (328286035) do not change. Until step 3 the check treats systemorph-com[bot] and systemorph[bot] as the same reviewer (INTERNAL_REVIEWER_LOGINS) and both slugs as the same App (DEGRADATION_APP_SLUGS, always together with the id); the RENAME cases of --self-test pin both, with another App's bot and a User account of the new name as negative controls.

Historical — superseded by the next section. Under the retired policy internal-code-review two reviewers were accepted: Copilot, and the internal GLM-5.3 reviewer of MeshWeaver.Plugins' PR steward, which posts through the systemorph-com App (COMMENTED or CHANGES_REQUESTED, never APPROVED). That policy planned for Copilot to leave the set; the next section reverses it.

Copilot is the reviewer again

Policy copilot-code-review reverses internal-code-review. GitHub Copilot reviews every pull request again; the internal reviewer is no longer required and is being switched off.

• The request. Every repository's ruleset carries copilot_code_review with review_on_push: true (review_draft_pull_requests: false), so Copilot reviews every pushed head of a non-draft pull request. One review per pull request below turned that off for a while; Every push is reviewed again turns it back on.
• What branch protection requires. internal-review is not a required context in any repository. Core still requires Automatic review answered.
• This merge gate. It takes a landed Copilot review, as it always did. Every thread Copilot or the internal reviewer opened still needs a person's reply.
• The stage gate and the arm gate. They accept a landed Copilot review of the pull request: not PENDING, and a body that reads as a review, never a refusal (copilot_review_of_pull_request in check-review-answered.py — the head's own review first, otherwise one of an earlier head; see below). They check this before looking for an internal-review run, so no head waits out the stage gate's fallback for a reviewer that has been switched off.
• What does not count. A refusal on any head. The internal bot's review alone, because its verdict on a head is its internal-review check run. The self-test pins every one of these cases, and replacing copilot_review_on with one that finds nothing turns the acceptance cases red.
• One timing difference. Copilot's own pull_request_review event starts no workflow run (see The reviewer's own event cannot start a run here below). So a Copilot review releases a held stage gate on the next stage-advance.yml sweep, at most about 15 minutes later, rather than on the event itself.

One review per pull request

Policy review-once-per-pull-request narrowed copilot-code-review: a pull request was reviewed ONCE, not once per push. Its ruleset half is reversed — see Every push is reviewed again; its gate half below still holds. With review_on_push: true every push bought a full new review round, and every round's new threads held the merge again until a person answered them — so a pull request that needed three fix pushes paid for four reviews and four rounds of replies.

• The gates. All three — Automatic review answered (evaluate), the stage gate (stage_readiness) and the arm gate (arm_readiness) — count a landed Copilot review against ANY head of the pull request as "reviewed". copilot_review_of_pull_request returns the newest landed review against the current head when there is one, otherwise the newest landed review against an earlier head, and the verdict's note names which head it was. pulls/{n}/reviews lists only this pull request's reviews, so every review it returns reviewed this pull request. The stage and arm gates used to look at the head and now do not. The merge gate's condition 1 did not look at it either, except for the one day the zero-findings case was narrowed to the current head (policy review-evidence-current-head, superseded); under review-evidence-any-head it again reads every review on the pull request. When the pull request carries NO reviewer thread, Automatic review answered requires that review to be Copilot's — on any head (policy review-evidence-any-head; see Reviewed, nothing found).
• What is unchanged. Every thread a reviewer opened still needs a person's reply, whichever head it was opened on. With zero findings the review must be Copilot's (policy review-evidence-any-head). A refusal is still not a review on any head. No review at all is still red.
• The ruleset (reversed). The copilot_code_review rule ran with review_on_push: false; it runs with true again now. The script that applied false is replaced by .github/scripts/set-copilot-review-on-push.py, which can only move a ruleset TO the in-force value (see Every push is reviewed again).
• The order. The gate change merges FIRST, the ruleset flip comes second. Flipped first, every new push would carry no review of its head while the old stage and arm gates still demanded one.
• Drafts. With review_draft_pull_requests: false a draft is not reviewed while it is a draft; the one review must then come when it is marked ready. Whether review_on_push: false still requests that review on ready-for-review was not measured when the policy was set. Check it on the first pull request opened as a draft after the flip. If no review arrives, the stage gate's fallback and the review-waived label are the existing exits, and review_draft_pull_requests: true is the remedy — the one review then lands while the pull request is still a draft.
• The self-test. The ONCE cases pin it: an older-head review with every thread answered is green on all three gates, an older-head review with an unanswered thread is red, an older-head refusal is red, and no review at all is red. With copilot_review_of_pull_request cut back to the head alone, every stage- and arm-gate ONCE case that reaches the thread condition turns red.

A review on every real change, none for a merge of main

Policy review-on-real-change replaces Every push is reviewed again. With review_on_push: true, a push that only merged main (which the merge-before-push rule requires) bought a full Copilot round on code the pull request did not change: new threads, and Automatic review answered red until a person answered them. The rule now is:

• A push that only merges the default branch owes no review. "Only" is computed from the repository, never from a commit message: every commit on the pull request since the reviewed head is a merge commit, and the pull request's own diff (merge-base..head, GitHub's compare) has the same patch-id as the reviewed head's (hunk line numbers ignored, every other byte kept). That is merge_only_same_diff in check-review-answered.py — the same function the internal-review carry-over uses — so a conflict resolution that changed what the pull request does is a real change.
• A push that changes the pull request's own diff gets a review. The review-on-change lane (.github/workflows/node-repo-request-review.yml, core's caller request-review-on-change.yml) runs on every pull_request synchronize and requests ONE Copilot review with a single GraphQL requestReviews call (botIds: [<the Copilot bot's node id>], union: true; REST requested_reviewers refuses a bot). It compares the head against Copilot's newest landed review, not just the previous head, so a real change whose review never landed is not forgotten under a later merge of main; with no landed review yet it compares against the push's previous head (github.event.before). Its success signal is the mutation's own answer listing Copilot as requested. It requests nothing for a draft (the ruleset decides drafts), for an App's own pull request (Copilot does not review those), when a Copilot review is already pending (read off the REST timeline), when the head is already reviewed, or when Copilot already answered this head with a refusal (a maintainer re-requests that one) — so it is idempotent, and since a review request pushes nothing it cannot loop. A fork pull request is printed as a ::warning:: naming the manual request; every failed read or request is red, named.
• The sweep. A push that finds a review already pending requests nothing, and that pending review can then land on the PREVIOUS head — with no event to follow, because Copilot's own review event starts no run here. So the caller also runs on schedule (every 15 minutes) and the lane re-judges every open non-draft pull request (--request-review-sweep): it requests a review only where a landed review does not cover the head and nothing is pending, and never for a pull request with no landed review at all (that is the opening review's business).
• The merge gate enforces the same line. Automatic review answered takes an earlier head's Copilot review only when copilot_carry_over proves it covers the head (copilot_currency). Otherwise the verdict is RED — THIS HEAD OWES A REVIEW, waited for (up to the check's --wait-for-review window) exactly like a review that has not landed, and released by the review landing, or by a maintainer's review-waived label. Unanswered threads stay red either way. An unreadable compare is a real change: the gate fails closed.
• The ruleset. The copilot_code_review rule runs with review_on_push: false (Copilot still reviews a pull request when it opens); review_draft_pull_requests keeps its value. .github/scripts/set-copilot-review-on-push.py carries REVIEW_ON_PUSH = False and is applied per repository, only once that repository calls the review-on-change lane (--repo NAME --apply). --out DIR writes, before any PUT, <repo>.<id>.before.json — the ruleset's fields as read, the restore body — and <repo>.<id>.put.json, the body it PUTs; restoring is a PUT of the .before.json file. A repository flipped without the lane would leave every real change after the opening review unreviewed, so the script enforces both halves: --apply without --repo is a usage error (never the whole FLEET), --apply refuses a repository whose default branch carries no .github/workflows/request-review-on-change.yml, and --out refuses to overwrite a .before.json already there — a retry into the same directory after the change would otherwise record the changed value as the restore point.
• The order. The gate half and the lane land in core first (the gate change is safe under review_on_push: true, where every head is reviewed anyway); each repository then adds its thin caller; only then is that repository's ruleset flipped.
• The rollout is complete — measured 2026-10-30. A dry run of set-copilot-review-on-push.py over its whole FLEET read review_on_push: false — unchanged — on all ten rulesets (MeshWeaver 2128472, MeshWeaver.Plugins 19008635, Memex 21038115, Crm 21741681, Education 19153714, Reinsurance 20302588, SocialMedia 20302587, Manufacturing 20625978, FundReporting 23730648, Deploy 25787784), and every one of those ten repositories carries .github/workflows/request-review-on-change.yml on its default branch (contents API). Nothing is left to apply.
• The self-test. The REAL CHANGE and real-change cases pin it: a main-only merge after a review is green (and its negative control, the same merge with a changed diff, is red); a real change after a review is red until its own review lands; an unanswered thread is red over a covered head (control: answered → green); the lane requests exactly once for a real change and never for a main-only merge, a pending request or a reviewed head. Restoring the any-head reading (copilot_currency returning the latest review whatever changed) turns the REAL CHANGE NEG cases green and fails them.

Every push is reviewed again

Being replaced by A review on every real change, none for a merge of main (policy review-on-real-change), repository by repository as each adopts the review-on-change lane.

Policy copilot-code-review, as amended, puts review_on_push: true back on the copilot_code_review rule in all eight repositories' rulesets (review_draft_pull_requests stays false). Copilot re-reviews automatically after every code change.

• Every push to a non-draft pull request triggers a fresh Copilot review, so every push opens a new review round, and that round's new threads hold the merge until a person answers each one ON the thread. That is why AGENTS.md says to push only for a real defect: a nit or a question is answered on its thread, not with a commit.
• Nobody hand-requests a review. The ruleset requests one on every push, so an author never asks for a review or a re-review, and never withdraws one. The one exception is a review the reviewer REFUSED (below, A refusal is posted as a review): once the reviewer can review again, a maintainer's re-request or the review-waived label clears it.
• The gates do not change. A landed review of any head still counts as "reviewed" (review-once-per-pull-request, gate half; review-evidence-any-head). A push therefore never goes red just because its own review has not landed yet. Threads the new round opens still need replies.
• The ruleset is reconciled, not hand-edited. .github/scripts/set-copilot-review-on-push.py holds the in-force value in one constant (now REVIEW_ON_PUSH = False, policy review-on-real-change; it was True under this section) and applies it to the nine repositories in its FLEET idempotently: dry-run by default; --apply PUTs the ruleset with every other field as read, then reads it back and fails unless only that one parameter changed. Changing the policy means changing that constant in the same pull request. (Superseded: the script now holds False, and on 2026-10-30 a dry run read every ruleset in its FLEET — ten, FundReporting and Deploy included — already at false; see The rollout is complete above.)
• Drafts are not reviewed while they are drafts. The review comes on the first push after the pull request is marked ready (or when it is opened ready).

One account, two logins. A predicate keyed on either login alone sees half of the reviewer. The check matches on the account id or either login, and only for type: Bot, so a person who names their account Copilot is not the reviewer.

Exactly one review per pull request on all 50, always COMMENTED, usually on the first commit rather than the head. At the time of that measurement the ruleset carried review_on_push: false and review_draft_pull_requests: true: the reviewer reviewed once, on open, drafts included. That is no longer the configuration — see Every push is reviewed again.

A refusal is posted as a review. On #645–#654 (2026-07-25/26) the reviewer answered every pull request with a review, from the same account, whose whole body was:

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

So "a review by the reviewer exists" reads a quota outage as "reviewed". The check therefore classifies the body:

A refusal wins. Everything else the reviewer says is the review.

The generated-file exception was measured, not designed: on MeshWeaver.Plugins#4507 a lockfile-only pull request drew ### 🟢 Approval recommended followed by Copilot wasn't able to review any files in this pull request. and a one-entry 1 file not reviewed list (package-lock.json: Generated file), and the gate reported that real review as UNREVIEWABLE. A skipped file with any other reason, or a list shorter than its count, keeps the body a refusal (fails closed).

Provenance, not presentation — and what the other way cost

The body is read for one purpose: to separate a review from a refusal to review. What makes a review a review is that the reviewer account posted it at a non-PENDING state. That is is_reviewer plus the state check, and the body has no part in it.

Until 2026-09-18 landed additionally required the literal string Pull request overview, and this page said so, adding: "If the reviewer changes its format, every pull request reads red naming the new first line, and the fix is one marker in the script." The failure was therefore foreseen and accepted — sound reasoning for an advisory check, where a false red costs a reader a glance.

Two things then happened within a day of each other. The check became a required context (ruleset 2128472, 2026-09-17T21:15Z), and the reviewer dropped its overview block, posting a short verdict body instead — ### 🟢 Approval recommended, ### 🟡 Changes recommended, ### 🔵 Needs a closer look, plus a sentence or two. The marker matched nothing. Six core pull requests were genuinely reviewed and every one read "the automatic review has not landed".

Promotion changed the cost of "cannot-tell is never a pass" from a glance to a hard stop on every open pull request, with no exit but a maintainer waiver — and answering the findings could not clear it, because the unanswered-threads check is a second reason and the review-landed reason stood regardless. A whole lane was blocked while the reviewer had in fact reviewed everything.

So the marker was removed rather than updated. Chasing the format would have re-armed the same trap on the reviewer's next revision. A decorative substring is the reviewer's choice and can change without notice; the account id cannot.

🚨 The self-test could not have caught it. All three review fixtures carried the marker, so the suite proved the rule only on the side of the change where it held — a control with no case on the other side. The fixtures now include the three bodies measured on 2026-09-18, and each is verified to go red if the marker rule is restored.

When it is evaluated

There is no job-level if:, no path or branch filter and no continue-on-error: a skipped required context counts as satisfied, so every event evaluates the predicate in full. The script's self-test runs before the verdict in the workflow, and again in dotnet-test.yml's CI-shell lane so a pull request that breaks the predicate goes red on itself.

Evaluations are serialised per pull request for pull-request events and per queue entry for merge_group — the queue head ref carries the entry's sha, so a re-queue of the same pull request may evaluate alongside an older entry. That is the right granularity: what must not race is two evaluations publishing the same check-run name on the same commit, and two queue entries of one pull request are two commits with two check-runs. Nothing is cancelled mid-read, which also matters because cancelling a merge_group run ejects the entry rather than retrying it. A burst of replies arrives as one submitted review event per reply; GitHub keeps one running and one pending run per group, so a burst of fourteen collapses to a few evaluations, and the one that runs last read last.

One judgement per reply, not two

A person's reply to a finding raises two events in the same second: pull_request_review_comment: created and pull_request_review: submitted. Every review comment, a finding or a reply, belongs to a review. While this check listened to both, every reply was judged twice. Each judgement is two full evaluations in core (the answered job and the fleet lane), and each spends REST calls on the repository's shared github.token budget. On 2026-10-25 a burst of 27 replies on #7180 started about 60 runs, and the budget ran out for every pull request (MeshWeaver#7187).

Measured on core's last 500 runs of Review answered (2026-10-30 07:49–16:45Z):

So the comment trigger reported nothing the review trigger did not. MeshWeaver.Plugins#3101 found the same pairing for the stage-advance lane (60 of 60 and 52 of 52).

The rule, for core's caller, the lane's caller contract and stage-advance.yml:

• review-answered.yml listens to pull_request_review_comment: deleted only. A reply being deleted raises no review event and can make a thread unanswered again. Nothing else would re-judge a GREEN verdict, so deleted stays. Deletions do not come in bursts.
• stage-advance.yml does not listen to review comments at all. Its lane header already said so; core's own caller still listened to created.
• Nothing that branch protection reads is lost. Branch protection never read a comment-driven run's job check-run (#4649, below). The verdict writer's check-run is written by the paired review-event run. The refresh job follows a GREEN verdict on a pull_request_review run exactly as it followed one on a comment run.
• It is enforced. check-review-answered.py --self-test reads all three workflow files and refuses a created trigger (trigger_contract_failures, trigger_self_test). It reads the active YAML and a caller contract written as # comments as two separate layers, so a commented # types: [deleted] under a bare active trigger does not count. In the active YAML it reads the top-level on: block only, so an if: or concurrency expression that compares event_name to the event is never mistaken for a trigger. It reads one shape only: pull_request_review_comment: followed by types: [..] on the next line. The key may be bare, 'single-quoted' or "double-quoted", because YAML reads all three as the same key. Any other shape is red and named: a flow mapping such as {types: [created]}, on: [..] (quoted items too), a block-style types: list, or a file with no on: block. A bare trigger counts as listening to every type. When a workflows directory sits beside the script, each expected file is checked, and a missing one is red. The check is skipped only when there is no workflows directory at all, which is how the lanes run a fetched copy. Each refused shape has a negative-control fixture that must fail for exactly its own reason.

What was not measured: how many REST calls one judgement of this check makes. A satellite whose caller still lists created keeps paying double until it adopts the contract. The self-test checks core's tree only.

🚨 The reviewer's own event cannot start a run here — so the check waits, briefly

Measured on this check's own pull request, #4575 (2026-09-17T08:05Z): the reviewer's pull_request_review event does reach the repository and GitHub does create a workflow run for it — run 35197843933, conclusion action_required, zero jobs. The triggering actor is Copilot, and the repository requires approval for runs triggered by a first-time contributor (actions/permissions/fork-pr-contributor-approval → first_time_contributors). The two pull_request_review_comment runs the review's own comments raised were action_required too.

So the event that would say "the review has landed" cannot evaluate anything, and a pull request whose review raises no findings would keep the red from its opened evaluation until somebody pushed again — a permanent red over a review that did land.

The check closes that itself: --wait-for-review 15. While the only thing missing is the review, the step re-reads every 30 seconds for up to fifteen minutes and then answers RED (the job's cap is 20 minutes). Nothing else is ever waited for — an unanswered thread needs a person, an incomplete listing needs another read. On #4575 the review landed 4m38s after the pull request opened; the range on #4299's thread is 3–12 minutes.

A person's reply is not gated: it re-runs the check within seconds, which is what turns a red into a green once the findings are answered.

The cheaper mechanism is a repository setting, not a wait: if runs triggered by Copilot no longer need approval, the review's own event evaluates the check the moment it lands and the wait becomes dead weight. That is a maintainer decision about the Actions approval policy, and it is the first item in Rollout.

Every read is REST — pulls/{n}, pulls/{n}/reviews, pulls/{n}/comments, issues/{n}/events, collaborators/{login}/permission — with the job's read-only GITHUB_TOKEN. The collaborator read is taken on every run for the pull request's author (when that is a person), so a token that cannot verify a waiver is discovered on an ordinary pull request rather than when a maintainer needs the waiver.

🚨 Three ways to be red, and only one of them is something you can do anything about

The check has one red and three causes, and until 2026-09-18 it printed one sentence for all three — "the automatic review must land. It usually arrives minutes after the pull request opens…". That sentence is correct for exactly one of them.

The middle row is the one that cost something (#4730). On 2026-09-18 the reviewer refused for quota from 11:39Z, and six pull requests — every one green on Consolidate test results, every one with auto-merge armed — sat blocked for over four hours reading "it usually arrives minutes after the pull request opens". Nothing was arriving. There were no findings to answer, and no push, re-run or new commit could change the answer, because the refusal is about the reviewer and not about the pull request.

So a refusal now names itself in all three places a reader looks — the run's headline (RED — UNREVIEWABLE (the reviewer REFUSED to review this pull request)), the step summary's heading, and the To go green line, which for a refusal says explicitly that pushing, re-running and replying all leave it exactly where it is.

🚨 And the run no longer WAITS for it. waiting_would_help existed for a real case — the reviewer's own event cannot start an evaluation here, so a review that raises no findings needs a bounded wait to be seen at all — but it asked "does the reason contain has not landed?", and the refusal reason is spelled "the automatic review has not landed — the reviewer posted, but not a review: …". So --wait-for-review 15 slept a quarter of an hour printing "waiting for the automatic review" at a reviewer that had already answered: it said no. Nothing arrives in that window by construction, the run then contradicts its own summary, and it spends a runner doing it. That is the same defect as the guidance line, one layer down, and it is why the discriminator has to be the field rather than the prose.

🚨 The VERDICT did not change and is not meant to. A refusal was red before and is red now; the check still fails in the safe direction, and the remedy is still a maintainer's. refused is a field on the verdict rather than a substring of the reason text, for the reason this page's own header gives about presentation-keyed reading — and the self-test asserts what the reader is told (says / never_says, over the guidance, the summary and the run headline), not only what the verdict is, because that is precisely the half that was wrong while the verdict was right. Each half has a negative control: remove the wait's not verdict.refused and the refusal-only wait case goes red; delete the headline and the refusal case goes red.

Decided: what a structurally unavailable reviewer does to the merge gate. It holds. Policy review-evidence-any-head accepts no timeout, no degradation run and no other reviewer's word as "reviewed, nothing found"; the review-waived label stays the escape. See Reviewed, nothing found.

🚨 An API outage is not a verdict — a fourth red, and nobody's to fix

A read that GitHub refused for capacity says nothing about the pull request. Until #7187 it was reported as the ordinary red — "cannot read the review of #N, so it cannot say it was answered" — and nothing re-evaluated it: the refresh follows only a GREEN verdict, and the degradation listener only the internal reviewer's check run. Measured on 2026-10-25, when the repository's shared installation budget (5,000 REST calls an hour, shared by every workflow of the repository) ran out:

Sessions papered over it with hand re-runs and a polling script. The fix is at the root, in the one adapter every mode of check-review-answered.py goes through (the verdict, the refresh, the review-on-change request, the stage gate and the stage advance, the arm gates):

1. Classified. A failed gh api call whose text names a rate limit (403/429 "API rate limit exceeded"), a secondary limit, an HTTP 5xx or a dropped connection is an outage (classify_api_failure). A plain 403 ("Resource not accessible by integration", "cannot be retried") and a 404 are answers and stay ordinary failures.
2. Retried, by its own clock. A primary limit waits for its own reset, read off GET /rate_limit (which does not count against the limit it reports); a secondary limit backs off from 60 s, a 5xx from 10 s (outage_delay). The budget is per run, MW_API_OUTAGE_WAIT_MINUTES (5 in every lane), and every lane's timeout-minutes carries it.
3. Never swallowed into a verdict. Past the budget the adapter raises ApiOutage, which is deliberately not a ReadError: three readers catch ReadError and degrade (read_diff_carry and read_carry to "the head owes a fresh review", read_required_contexts to an empty list), so an outage in the middle of a read would otherwise have become a finding. The script exits 3 with outcome=api-outage on $GITHUB_OUTPUT and says, in the annotation title, "API outage — no verdict was taken".
4. Retried later, automatically. Each review-answered job then fails a marker step named exactly API outage: no verdict was taken (retried automatically) (OUTAGE_STEP_NAME; the core self-test asserts both workflows carry it). review-answered-retry.yml → node-repo-review-answered-retry.yml hears the run complete, re-runs it only when that step failed — a verdict red is never touched — waiting first for the primary limit to reset (up to 35 minutes), and stops, RED and named, at the third outage attempt in a row — counted from the attempts whose marker step failed, so an earlier verdict red or refresh re-run of the same run consumes none of the allowance. 🚨 The wait comes BEFORE the listener's own metered reads (only the free GET /rate_limit is read while waiting), and the decision is taken on those fresh reads and posted at once: reading first made the listener die on its first read while the limit was spent, and a decision read before a 35-minute wait could re-run a pull request that had since closed, moved its head or been re-run. The listener reads pulls/{n}, so the lane and every caller grant pull-requests: read (a called workflow cannot widen its caller's grant; the self-test asserts the three grants). 🚨 It re-runs EVERY outage-failed run of the check on the pull request's current head, from ANY event — not only the run that woke it: branch protection kept reading the FAILED check-runs of pull_request_review and pull_request_review_comment runs that met the limit (12:48–12:52Z on two plugin-repository pull requests) after newer runs had succeeded (13:53Z, 14:27Z), and the merge API answered "Required status check … is failing". So any FAILED run of the lane — a verdict red included — wakes a listener that lists the head's failed runs and applies the same decision (outage marker, streak bound, fresh read) to each; a run whose own listener died is recovered by the next failure on that head. The in-run budget, the listener's wait and the re-run's own budget together cover the hour a primary limit can last, so ONE listener pass suffices for a primary limit. Whether GitHub raises workflow_run again for a run the workflow token re-ran was not measured; nothing depends on it, and the attempt bound holds if it does. The review-on-change lane and the stage advance need no listener: their 15-minute schedule sweeps re-judge every open pull request, and the advance re-runs a stage gate that failed for any reason once stage 1 is green.

The wait no longer spends the budget it then runs out of. The review wait (above) re-ran the whole read every 30 s — 7–10 calls a poll, ~30 polls, in two jobs per event in core — up to ~600 calls of the shared hourly budget for one push whose review was slow. It now probes the only three inputs that can end it — the reviews, the head's internal-review check runs and whether the review-waived label is on (review_wait_fingerprint, three calls) — and takes the full read again only when one changed, and once more at the deadline, so the verdict is still taken on a fresh, complete read. The label is probed because a waiver's own labeled run cannot judge sooner in core: it queues in this run's non-cancelling …-gate group, so an unprobed waiver was honoured only at the final read, ~15 minutes later. How much of the 2026-10-25 exhaustion this lane caused was not measured; the arithmetic bounds its share, it does not prove it.

Every half has a negative control in the self-test, run against a fake gh: revert the classification and nine outage cases go red (the #7173 case reads rc=1, not 3); make ApiOutage a ReadError and the carry reader turns the outage into "the head owes a review"; skip an in-flight read run again and the #7198 case posts nothing; wait for a not-started read run again and the four pending/queued cases answer wait; read before the listener's capacity wait again and the spent-limit cases raise ApiOutage while the low-limit one re-runs a pull request whose head moved during the wait; bound on run_attempt again and a first outage on attempt 3 is called three in a row; re-run only the waking run again and the three retry sweep cases leave a review or review-comment outage run red; restore the full re-read and the wait case counts 30 full reads instead of 2; let the listener treat any red as an outage and it re-runs a verdict.

The waiver

A pull request the reviewer cannot review — a quota refusal, an outage, a change with no reviewable files — is released by exactly one thing: the label review-waived, applied by a maintainer.

• The label's presence is read from the pull request; who applied it is read from the REST issue events (the latest labeled/unlabeled event for that label).
• It is honoured only when that account is a person and its role_name on the repository is admin or maintain. A writer's label, a bot's label, or a label with no attributable event is refused, and the refusal names the account.
• It releases condition 1 only. Every thread the reviewer did open still needs a reply, waiver or not.
• It is never automatic, and the log names who waived and when.

The first remedy for a refusal is not the waiver: a maintainer re-requests the review, and a real review replaces the refusal.

🚨 An agent never applies the waiver. Agent sessions here run under the maintainer's own GitHub account, and the check reads an account's role, not who was at the keyboard — it cannot tell a maintainer's waiver from an agent's. The label event in the pull request's timeline is the audit.

The reviewer-unavailable degradation — the exit that needs nobody

🚨 RETIRED for the merge gate by policy review-evidence-any-head (#4730): a degradation run says the reviewer did NOT review, so it no longer releases condition 1 of Automatic review answered; the review-waived label is the escape. The stage gate still lets the suites start on one (runner spend only), and the arm gate still refuses to arm on one. The section below is kept as the record of what it did.

Why it exists (MeshWeaver.Feedback#86). With the waiver as the only exit, the gate was circular: when the internal reviewer itself stops completing rounds — on 2026-09-29 they aborted at the 30-minute cap (MeshWeaver.Plugins#2564, #2565, #2568) — every pull request is held, including the one that repairs the reviewer, until a person is available to waive. Whether a change can merge must not depend on a person's availability, and a defect in the review step must not be able to block its own fix.

The rule. Condition 1 is also released when the pull request's head commit carries a check run that satisfies ALL of:

The newest completed internal-review run from that App on the head decides, so a later real round supersedes an earlier degradation (and runs from other Apps are never looked at). --as-of ignores a run completed after the instant; the merge-queue path reads the pull request's head, not the queue's merge commit. The check-run listing is read on every evaluation, and a failed or incomplete read is RED, like every other input. The gate and the fleet lane grant checks: read for it explicitly: measured on #5920 the listing answers without it, but only because core is public — a private caller's token would refuse it and hold every pull request.

Provenance, not presentation — the same principle as for the review itself. A neutral internal-review from any other App, a run under any other name, or that title at any other conclusion is not a degradation. The self-test carries a negative control for each field, and mutating the predicate to drop any one of them turns its control red.

It is a DEFERRED review, not a skipped one. The Plugins steward (Governance/PullRequestSteward, MeshWeaver.Plugins) re-kicks a failed round once, and posts the neutral run only for an infrastructure cause — never because a change was hard to review — naming that cause in the run's summary. A pull request merged on a degradation owes a post-merge review, recorded on the item. The GREEN verdict is never silent about it: the run headline reads GREEN — REVIEWER UNAVAILABLE, review deferred (degradation, not a review), a ::warning:: annotation carries the check run's id, title and summary, and the step summary heads with reviewer unavailable: degraded, review deferred.

It releases condition 1 only, exactly as the waiver does: every thread the reviewer DID open before it went down still needs a person's reply. When a degradation and a waiver both stand, the log credits the degradation — the system released it, nobody had to.

How the degradation turns the context green without a push

A predicate is re-asked only when an event starts the gate, and none of review-answered.yml's triggers fires when a check run completes. Measured and read, in order:

• The internal reviewer's events DO start runs here. Over the last 100 runs of review-answered.yml (2026-09-30), systemorph-com[bot] triggered 11 pull_request_review and 30 pull_request_review_comment runs, none action_required — unlike Copilot's (#4575, above). So a review that does land re-evaluates the gate by itself.
• check_run cannot be added to review-answered.yml. A check_run-triggered run executes on the DEFAULT branch with GITHUB_SHA = main's last commit (GitHub's event reference), and the event fires only for a workflow file already on the default branch. Its Automatic review answered check-run would therefore land on main's commit, where the pull request's branch protection never reads it — the "green in the log, BLOCKED on the pull request" shape below — and a red one would sit on main for every unrelated check run. Check runs created by GitHub Actions do not raise the event at all, so a re-run cannot loop.
• So review-answered-on-degradation.yml listens for it and RE-RUNS the gate. On a completed check run matching the contract, it lets any gate run still in flight for that head finish (one that read before the degradation would otherwise publish a stale red), then re-runs the newest pull_request run of review-answered.yml unless it already reads success. A pull_request run is the one branch protection reads (#4649), and re-running it is the remedy this page already names as legitimate. The listener is single-shot — the degradation completes once — so each of its API calls gets three attempts before it goes RED, naming the head it could not re-evaluate. Since 2026-10-03 it re-evaluates on ANY completed internal-review check run of the internal reviewer, not only the degradation: core #6014, #6017 and #6018 were reviewed after the gate's 15-minute wait had already published red (the review queued behind the reviewer's capacity bound for over 75 minutes), and nothing re-ran the gate when the review arrived. The listener decides nothing: the re-run applies the whole predicate, provenance included, so a drift in its filter can cost or miss a re-run but never turn a gate green.
• 🚨 Not yet observed end to end. A check_run workflow cannot run before its file is on main, so the first real degradation after the merge is the first observation. Read it there: the listener's run on main's head names the gate run it re-ran, and that gate run's new attempt must read GREEN with the degradation line.

Why --wait-for-review stays 15 minutes although the internal reviewer takes 23–30: the wait exists only because Copilot's event cannot start a run. The internal reviewer's review event and its degradation check run each re-evaluate the gate, so a longer wait buys nothing they do not already deliver — and would hold two runners (answered and lane) for half an hour on every pull request opened.

The fleet lane (node-repo-review-answered.yml) fetches the same predicate, so a satellite caller that moves its pinned sha past this change judges the degradation too; the re-run listener is core's own file, and a satellite that needs the context to turn green without a push adds the same listener naming its own caller workflow.

What it does not see

• A reply that says nothing counts as an answer (accepted on #4299).
• Resolution. REST has no resolved state for a review thread, so a thread resolved without a reply stays red. Reply; resolving is optional.
• Suppressed comments. The reviewer lists some findings only inside its review body (Suppressed comments (N)). Those are not threads and are not held.
• A human reviewer's threads. Only the automatic reviewer's threads are held.
• A finding that lands after the queue entry was judged. The merge_group evaluation reads the review once per queue build. Because the reviewer posts once, on open, and the pull-request check cannot be green before that review has landed, a new thread during a queue build needs someone to re-request the review while the entry is building.
• Its own edits. Like every check in this repository it runs the pull request's copy of the script, so a pull request that changes check-review-answered.py is judged by its own change. Read that file's diff yourself.
• Pull requests into other branches. The ruleset reviews only the default branch, so a pull request into another branch reads red; the check is not required there.
• Other repositories run it through the fleet lane. Since #4776 every gated repository requires the same predicate; see "The rollout is complete" below. Memex does not.

🚨 This WAS a core-only gate, and the rest of the fleet showed what that costs

Measured 2026-09-19, from branches/main/protection and every active ruleset:

🚨 MeshWeaver.Plugins is NOT covered, against the common assumption that it is. Every satellite carries a Copilot review for default branch ruleset, so the review is requested everywhere — nothing outside core requires it to be answered.

What that produces, over the last 20 merged pull requests of each of six repositories:

In all 55 the ratio is N of N — not one finding answered on any of them, never a partial. So outside core, findings are not occasionally missed, they are structurally not read: about twice the rate this repository measured before the gate landed (32 of 60, above).

It is not cosmetic. On 2026-09-18, five satellite pull requests merged with 13 unanswered findings between them. Assessed on the code: 11 of the 13 were real, and only two could be declined — one whose premise git itself rules out, one whose failure branch is unreachable. Those 11 reduce to 6 distinct defects, because three were raised twice (independently, on two repositories' copies of one file) and three were three sites of one root. Three of the six are in the platform's own canonical gen-manifests.py, replicated byte-identically into four repositories — including a --resolve that reported ✓ … the merge can be committed whenever git could not answer. Answered and fixed in #4775 after the merges; the remaining vintages in #4777.

Porting it is a workflow_call lane, never six copies — the predicate is ~900 lines with its own self-test, the settle wait and the event-class concurrency split derived from #4649. Six hand-copies is how gen-manifests.py reached five vintages (#1426). 🚨 And the rollout order is not optional: five of the six use CLASSIC protection, where an absent required context blocks every pull request in the repository forever (measured on Plugins#1453), so the lane lands observe-only, is watched publishing its context on live pull requests in that repo, and only then is the context added. Proposed with the full measurement in #4776.

The fleet lane — step 1 of 3 is landed

.github/workflows/node-repo-review-answered.yml is that lane. It fetches check-review-answered.py from this repository at its scripts-ref input, asserts the file's identity, runs the self-test on every call and never falls back to a local copy; it takes no checkout of the caller at all. The caller contract — the lane pinned to a 40-char core sha and that same sha passed as scripts-ref, in a separate review-answered.yml with the three pull-request triggers, the event-class concurrency block, and no job-level if:, path filter or merge_group: trigger — is written in the lane's header, because every one of those is either a skip-trapdoor or the #4649 eviction.

It is not dead code waiting for an adopter: core's own review-answered.yml calls it as a second job, lane, on every event, with scripts-ref: ${{ github.sha }} so both jobs judge with the same copy of the predicate. That job publishes lane / Automatic review answered, which is not required — the ruleset requires the first job, Automatic review answered, by name. The two must agree on every pull request; a disagreement is a defect in the lane.

Historical — the rollout below is complete; see "The rollout is complete". The plan, as it was staged:

MeshWeaver.Plugins is the first adopter (MeshWeaver.Plugins#2727, at the maintainer's direction). Its staged pipeline (node-repo-stage-gate.yml) holds the heavy legs until the review has landed and been answered, but only when the stage gate runs with review-before-suites. On Plugins it runs with that off: the suites start in parallel with the review, and the gate's own log says "merging and arming still require the review landed and every thread answered". Arming does: the steward's PrArming checks it. Before step 3 a merge did not, because nothing in Plugins' classic protection read the answer; it does now.

Measured over the 25 most recent merges on 2026-10-08, two merged with Copilot findings unanswered:

• #3115: one thread on head 0c60ab5bbc, merged by hand 54 minutes after the review;
• #2962: four threads.

For Plugins, steps 2 and 3 of the table above went like this (done):

• Step 2 has two parts, in this order:
  • its roster row lands here as pending:;
  • then its review-answered.yml caller lands observe-only. The caller floats @main with scripts-ref: main, because Plugins' check-shared-lanes.py refuses a pinned lane. That repo's own rule overrides the lane header's "pin a sha".
• Step 3: the context review-answered / Automatic review answered is added to Plugins' classic protection, and only after it has been seen published on live pull requests there.

The rollout is complete: every gated repository requires the answer

Measured 2026-10-09, after the rollout (#4776). For each repository, the caller's own pull request published review-answered / Automatic review answered as success before the context was added to protection. So no repository was left with a required context that never reported.

Two things the rollout found, worth knowing before the next adopter:

• A satellite that carries scripts/check-shared-lanes.py classifies an unknown lane as source-building (Reinsurance, Crm, SocialMedia and Manufacturing do; Education carries no such script). It then demands platform-ref, so the caller's own Nothing pins the platform / policy gate goes red. In those repositories the caller PR must also add node-repo-review-answered.yml to SCRIPT_ONLY_LANES, with self-test cases for both directions.
• There is no legacy context to carry, so no shim job is needed. The shim-job route in Renaming A Required Check exists for a rename. Here the context is new, and the only rule is ordering: the context is published first, then made required. Under classic protection, an open pull request opened before its repository's caller landed reads "Expected" until its next push or review event runs the workflow. That pull request is blocked, not unguarded.

Not covered: Memex (ruleset 21038115), which calls only auto-arm.yml.

The same gap, measured wider — 240 merged pull requests

The 120-PR sample above was extended on 2026-09-19 to the 30 most recently updated merged pull requests in each of eight repositories (adding Memex, and deepening the six): 240 merged, 325 findings, 182 never answered, across 93 pull requests.

Core is 0 of 59, which is the check working — and it is also the control that makes every other row readable, because the same instrument finds both states. A zero here is a measurement, not a broken query. Memex is the worst of the eight and was not in the earlier sample.

A further 45 unanswered findings sit on 24 open DRAFTS (Plugins 34, of which #1910 alone is 13 of 13; Memex 11). Those describe code that never shipped and are deliberately left; several drafts are abandoned.

Treating the backlog: bound it, and state the bound

There are ~5,700 merged pull requests fleet-wide, so every sweep is partial. Report the denominator you actually swept and what is left, or the next session cannot tell a treated repository from an untreated one. Note that sort=updated is not sort=created: an older pull request that received a comment recently enters the window, which is why the windows differ in span per repo.

gh api "repos/Systemorph/<repo>/pulls?state=closed&per_page=100&sort=updated&direction=desc" \
  --jq '.[]|select(.merged_at!=null)|.number'
gh api "repos/Systemorph/<repo>/pulls/<n>/comments?per_page=100" \
  --jq '{findings:[.[]|select(.user.login=="Copilot" and .in_reply_to_id==null)]|length,
         replies:[.[]|select(.in_reply_to_id!=null)]|length}'

A finding counts as answered only when some comment's in_reply_to_id is that root's id. A PR-level comment answers nothing, and neither does resolving the thread.

Reply on every thread whatever the verdict. Four verdicts, and the declines need their reason on the record more than the acceptances do: real (fix it), real but cosmetic, obsolete (verify against the current file and quote the evidence), wrong (say so, with the measurement). A merged finding nobody answered and nobody declined is indistinguishable from one nobody read.

🚨 -F, never -f — the reply that posts its own filename

gh api -X POST repos/Systemorph/<repo>/pulls/<n>/comments/<id>/replies -F body=@reply.md

-f body=@reply.md sends the literal seven-to-twenty-character string @reply.md as the comment body, and the POST returns a normal comment id with a 201. So it reads as a successful reply; the thread's root now has a comment whose in_reply_to_id points at it; and every detector built on in_reply_to_id — including the sweep query above, and check-review-answered.py's own predicate — counts the finding as answered. The finding is untreated and nothing says so.

Measured on 2026-09-19: 100 posts across four sessions went out this way, each a @-prefixed filename or absolute path — Memex 41, Crm 24, Reinsurance 20, Manufacturing 15. Three of the four sessions believed they had replied and reported thread counts to prove it; the proof was the very field that cannot distinguish the two. The fourth caught it only by reading one reply back.

🚨 And the first audit of it was itself understated, for a structural reason worth keeping. A thread-centric sweep — enumerate the review threads on the swept pull requests, check their replies — saw 74 of the 100. The flag is a property of the call site, not of the thread, so it also hits PR-level issue comments (issues/comments, a different endpoint) and replies on the sweep's own pull requests, neither of which a thread-centric audit reaches. Re-audit by call site: every comment authored today on both endpoints whose body matches ^@. That found the remaining 26.

So the verification is read the body back and check its length, never the reply's existence:

gh api "repos/Systemorph/<repo>/pulls/<n>/comments?per_page=100" \
  --jq '.[]|select(.in_reply_to_id!=null)|"\(.id) \(.body|length)"'

A length near 20 is the bug. Repair with PATCH /repos/{o}/{r}/pulls/comments/{reply_id}, not a second reply — re-posting leaves the stub standing beside the real answer, and the thread then reads as two answers, one of them noise.

🚨 TWO flag mistakes produce a stub, and they leave DIFFERENT first characters. A sweep keyed on one of them misses the other, and the second is the worse instance because it looks less obviously wrong:

-F reads a file only with the @ prefix, so dropping it silently turns the path into the value. Measured 2026-09-19: the first mechanism produced most of the fleet's stubs, and the second produced a further batch that an ^@ sweep did not see. So do not key the sweep on a flag or on @ — key it on the shape of the body: a reply that is a single token with no whitespace is a path, whatever put it there. Re-audited on that key, 0 of 533 comments authored during the sweep were stubs; the earlier ^@-keyed pass could not have said so.

🚨 A stub also inflates every "how much is left" count, so a denominator taken before the repair is understated by its own stub total. A stubbed thread has a reply, so the sweep query above calls the finding treated — which means any backlog figure published while stubs are outstanding is a floor, and not by a knowable margin. Re-measure after repairing, and when reporting a backlog say whether the count was taken before or after. Two of this sweep's per-repo denominators were re-taken for exactly this reason.

The general rule, of which this is one instance: any gh api write whose payload came from a file is verified by reading the field back. -f and -F differ silently in both directions — -f treats @path as a literal string, and -F type-coerces a value that merely looks numeric or boolean — so the flag is the wrong thing to reason about. The response is not evidence either: it carries an id and a 201 whatever went in. Read the stored field and compare it with the source.

And the comparison is not a length check. @r_115_4028258629.md is 20 characters, which is short, not obviously wrong, and a genuinely terse reply would fail the same test. Assert a content signature you know is in the file — the verdict string the reply opens with — or byte-compare against the draft, allowing for the trailing newline GitHub appends.

This is the sweep's own instance of the defect class it exists to find: an answer that reads like a pass. The question to ask of any reply mechanism is the one that applies to a gate — if this had failed, would the output differ? Here it would not have.

🚨 A control can be the thing that cannot fail

The cases above are all checks that could not fail. The worse shape is a control that cannot fail, because a control is what you appeal to when you have stopped trusting the checks.

Measured 2026-09-19 on MeshWeaver.Education#335. A finding said a liveness census could not parse the heartbeat it exists to read, so wedged was false by construction and the instrument reported a clean pool over a real wedge. It was written up, agreed with, and given a control: fed the tick lines the script's own header quotes as the cause of a past incident, the census went from 0 of 5 wedged to 3 of 5, naming the window — a swing so clean it read as proof.

The parser was correct. The producer — ProcessLiveness.Describe() — renders every delta parenthesised and gap always as actual/threshold; its unit test asserts the rendered line, and a platform doc shows the same. The forms the control fed it (poolCompleted=+11343, a bare gap=10.00s) appear in no real log. The control had been built from an abbreviated transcription in a comment, so it proved something about the transcription and nothing about the parser.

Three things generalise:

• An example that merely APPEARS in the code under test is not a fixture. The header in question was not prose describing a format — it was a quoted log excerpt, which is exactly why it persuaded. A fixture is something the producer wrote (a format string, a golden file) or something a test asserts. Nothing else earns the name, however much it looks like captured output.
• A control that swings dramatically is not thereby a good control. A fabricated input makes a correct parser look broken, and the bigger the swing the more it reads as confirmation. Size of effect is not evidence of validity.
• The falsifying artefact was in the same repository the whole time — a sentence in the harness's own README stating that the real captured log does produce the window — and was read afterwards, while looking for somewhere to file the finding. Reach for the artefact that would falsify you before the one that would confirm you.

The real defect was the misleading transcription, which had by then generated the same false finding twice: once by the reviewer, once by the reader who agreed with it.

The two-part key: a read-back needs a baseline

This page's own sweep reported "182 of 182 findings answered, proven by read-back". The key it used was a substantive reply exists on this thread — and that half alone is unfalsifiable, because it passes on a reply an earlier session left. What makes the number mean anything is the other half: the same 182 were measured unanswered at sweep start.

So the baseline measurement is not a caveat on the result, it is half the instrument. State both halves or neither. The same applies to any "N of N done" claim built by re-reading a store you also wrote to.

🚨 A watcher's polarity decides whether a failed read waits or FIRES

The twin of "a broken query reports not-yet forever", and worse, because this one triggers an action. Measured 2026-09-19 at 09:08:35Z on MeshWeaver.Plugins: a blob watcher armed as

until [ "$(gh api "$PATH" --jq '.sha' 2>/dev/null || true)" != "$BASELINE" ]; do sleep 60; done
echo "ACT NOW — the file moved"

fired a false ACT NOW. GitHub answered the secondary-limit 403, --jq '.sha' returned the error body, and != against the baseline was true. Nothing had moved. || true plus != on an unvalidated value compose into any failed read is a positive result — and the announcement named a remedy (merge and push), which had it been trusted would have merged a stale canonical and burned a CI run during the very limit that caused it.

The polarity is the whole difference, and it is easy to get right by accident and wrong by accident:

So: write the condition so a failed read is FALSE, never TRUE — wait for the value you expect, never for difference from a value you remember. Where difference is genuinely what you need, shape-guard first (a sha must be 40 hex characters; a size must be digits) and make "could not read" its own printed outcome that backs off, rather than a silent third state folded into one of the other two.

The question to ask before arming any watcher is one line: what does this print on a 403? Ask it of the whole expression, including the || true you added so a transient failure would not kill the loop — that fallback is usually where the defect enters.

The safe form for posting

Build the whole request body as JSON with jq, and hand that one file to --input:

printf '%s' "$(jq -Rs '{body:.}' < reply.md)" > payload.json
gh api --method POST "repos/{owner}/{repo}/pulls/{n}/comments/{id}/replies" --input payload.json

🚨 The safety is jq's, not --input's. --input takes one file that is the complete request body — it has no per-field behaviour to confer any protection (Copilot review on #4788, which caught this page attributing it to the wrong half of the pipeline). What makes the recipe safe is that jq does the quoting and escaping, so no value is ever interpreted as a flag argument.

The one session of four that posted this way was the only one with no stub. It sidesteps the -f/-F question entirely rather than requiring anyone to remember which flag reads a file — which matters more than it sounds, because there are two ways to get that wrong, not one (below).

🚨 One defect, five copies — fix the canonical, then RE-COPY IMMEDIATELY

Findings cluster hard on vendored files, because the reviewer reads each repository's copy independently. Alongside the gen-manifests.py collapse above, the 2026-09-19 sweep found 25 of the 182 against five satellites' copies of scripts/resolve-platform.py, collapsing to five distinct defects in core's canonical — one of them reported three times over (fixed in #4779; the one deferred as needing a design decision is #4780). Patching a vendored copy alone is how this fleet reached five vintages of one script (#1426).

🚨 A canonical fix REDS EVERY SATELLITE THE MOMENT IT MERGES, so the re-copy is part of the same piece of work — not a follow-up. node-repo-validate.yml declares platform-ref with default: main, and scripts-ref falls back to it. A satellite whose validate: job passes no inputs — which is every one of them — therefore has the canonical fetched at core main, live, and check-resolver-copy.py has been hard-red since RED_FROM 2026-09-15. Measured 2026-09-19: core #4773 merged at 08:02:23Z and by 08:16Z every satellite's validate / Validate node repos — a required context in all of them — was failing with

scripts/resolve-platform.py has DRIFTED from the platform's canonical: 32 code line(s) differ
(76 raw), RED since 2026-09-15T00:00:00Z

Do not reason about this from a platform-ref: literal in a satellite's ci.yml. Those literals pin other jobs (compile-check, tag-modules, the pack lanes); the validate: job passes nothing and takes the default. Reading the wrong job's input produces the confident and wrong conclusion that a canonical fix is invisible to the satellites — it is the opposite, and the guard's own docstring says so ("a canonical fetched at @main is live on merge for every caller", MeshWeaver#4027). Read the run: the job log prints SCRIPTS_REF: main.

So the shape of the work is: fix the canonical, merge it, and re-copy into every satellite in the same sitting —

gh api repos/Systemorph/MeshWeaver/contents/.github/scripts/resolve-platform.py --jq .content \
  | base64 -d > scripts/resolve-platform.py

— verifying the guard then reports CODE-IDENTICAL (exit 0). A re-run does not help an open pull request, because the guard reads the branch's copy: that branch needs git merge origin/main after the re-copy lands on the satellite's main.

🚨 And re-check the canonical immediately before the merge, not only before the push. Measured 2026-09-19: the canonical moved twice in one morning (201,067 → 228,485), so a re-copy that was byte-identical when it was pushed was stale before anyone merged it — three satellites re-copied the first canonical, landed it, and were behind again within the hour. Education's session checked before merging rather than after and found its own main still red on the guard at the intermediate size, which is the check that saved a pointless merge. The window between waves can be shorter than the time a pull request takes to go green.

🚨 And while the copy is behind, the rest of the lane is silently ungated

This is the more expensive half, and it is a skip-trapdoor made by step ordering rather than by an if:. The drift check sits mid-job, so its failure skipped 16 subsequent steps of the same job (measured on MeshWeaver.SocialMedia#210, job 105867277548) — among them:

• Every PR-reachable secret in this repo is asserted by a preflight
• Every manifest.lock is current (and carries a version)
• Every module's version matches its content
• No mapping in this repo's workflows writes a key twice
• No pin comment names a commit this repo no longer pins

Each reported skipped, which under both protection mechanisms counts as satisfied. So for as long as a satellite's vendored resolver is behind, every pull request in it is unchecked by all of those, and the only visible symptom is one red about an unrelated file. Filed as #4784; the durable fix is if: ${{ !cancelled() }} on each independent guard, or one job per guard family, so that a single red reports rather than masks.

Controls

The predicate, replayed through the real REST adapter with --as-of on pull requests whose outcome is known:

The self-test (--self-test) states, for each of its fixture cases, the exact reasons the case must be red for, so a fixture cannot pass by being red for a second, unintended reason. Its negative controls were watched failing: twelve mutations of the predicate — a bot's reply counted as an answer, a refusal counted as a review, an unrecognised body counted as a review, any bot taken for the reviewer, the completeness check removed, a waiver releasing threads, a writer or a bot allowed to waive, --as-of ignored for comments, only direct replies followed, a PENDING review counted, a short sha accepted as a queue ref — and each one turned the self-test red.

To replay a pull request yourself:

python3 .github/scripts/check-review-answered.py --repo Systemorph/MeshWeaver --pr 4310 --as-of 2026-09-14T14:04:21Z

Observed end to end on #4575

The check's own pull request, in order, with the run that did the work:

The review that counts was submitted on the previous head (c496a0bffb) and the check is green on the new one (bf37e92a09): condition 1 asks whether the review landed, never whether it landed on the head — the any-head gate (policy review-once-per-pull-request, gate half). At the time of this record the reviewer also reviewed only once per pull request; it now reviews every push again (Every push is reviewed again), and the gate still counts any head's review. And the 15-minute wait was exercised against #645 — three polls, then RED naming the quota refusal.

Rollout — done; kept as the record

🚨 This section is history, not instructions. The context was added to ruleset 2128472 beside Consolidate test results on 2026-09-17, and the check is required today (see Status above). Read the steps below as what was confirmed before the switch was thrown — do not re-run them as a plan, and do not read "the check lands non-required" as the current state.

Automatic review answered

What was confirmed before adding it:

1. Decide the approval policy for Copilot-triggered runs. Today they are action_required with zero jobs (measured, above), so the bounded wait is what makes the check self-sufficient. If the policy is relaxed, drop --wait-for-review to 0 and the job's cap to 5 minutes.
2. Watch it on live pull requests — red on open, green once the review lands on a pull request with no findings (through the wait), and green after the replies on one with findings.
3. ✅ Teach the merge-queue steward this check — done 2026-09-17. It read only the failed merge_group run of MeshWeaver Build and Test under the pull request's queue prefix, so an ejection caused by this check reached it with both outcomes wrong: no such run → rejected as unclassifiable, naming the wrong workflow; or an older failed build of the same pull request → it classifies a failure that is not why the entry was removed, and if that stale failure is a catalogued flake it re-queues a pull request whose findings are unanswered. merge-queue-steward.py now reads the other merge_group workflows first and rejects with kind=gate, naming the workflow — keyed on "not the test workflow" rather than on this check by name, so a gate added later is covered the day it runs rather than the day somebody remembers that file.
4. Decide the cost. 32 of the last 60 merges would have waited for replies.

The timing, re-measured 2026-09-17 (39 merged pull requests)

The original framing — "checks outran the reviewer" — is no longer the live mechanism, and the numbers say so:

The required gate finished before the reviewer submitted on 0 of 37. The denominator is 37 of the 39 because two merged pull requests carry no automatic review at all — #4584 and #4582, both with zero reviews of any kind, checked rather than inferred from the gap in the counts. With no review there is no submission time to compare against, so they are excluded from the comparison rather than counted as a win for either side. They are also exactly the pull requests this check would hold: no review landed, condition 1 unmet — so the two numbers disagreeing is itself a measurement, not an inconsistency.

The reviewer is reliably first. So what merges past a finding today is not a race — it is that the review lands, and nothing requires it to be read. That is precisely what this check asserts, and it is why the remaining step is the ruleset edit rather than any further engineering.

What an author does

Reply to every thread the reviewer opened — "fixed in <sha>", or why not — through the pull request page or REST:

gh api "repos/Systemorph/MeshWeaver/pulls/<PR>/comments?per_page=100"
gh api -X POST "repos/Systemorph/MeshWeaver/pulls/<PR>/comments/<comment id>/replies" -f body='Fixed in <sha>: …'

Never hand-request the review to turn the check green, and never apply review-waived yourself.

🚨 Posting a reply from a file needs -F and @ — miss either and you post the PATH

Reading a body from a file takes both the typed flag and the @ prefix: -F body=@reply.md. Two independent near-misses each post the path itself as the whole reply, and they are distinguishable by the body's first character:

Each row posts its own value verbatim, which is what makes the near-miss reproducible as written. 🚨 In the wild it does not look that obvious, because the path arrives in a variable — the shape that hid the missing @ from its author:

reply() { gh api graphql -f query='…' -f id="$1" -F body="$2"; }   # ← "$2" needs @, and has none
reply "PRRT_kwDOTQ2qnM6XczkO" /Users/roland/.mwgate/reply1.md

-f sends its value literally and never interprets @; -F interprets @value as a file but passes a bare value through unchanged. Both apply to gh api graphql variables as well as REST fields. Either way the call SUCCEEDS, in_reply_to_id is set, the thread renders as answered, this check goes GREEN, and the finding is never read by anybody. It is strictly worse than not replying at all, because it consumes the one signal — "this thread is unanswered" — that would otherwise bring somebody back to it. Worse still when the same loop goes on to resolve the thread, as MeshWeaver.Plugins#370's did: the finding then looks deliberately closed rather than merely answered.

Measured 2026-09-19: 100 stub posts across four sessions. On MeshWeaver.Plugins#370 four Copilot findings sat that way for six weeks, one of them a privilege-escalation shape in an admin-invokable action. The answers had been written — the files were still on disk — and only the posting failed.

🚨 Never key detection on the stub's shape. The body is whatever string the caller passed, and three shapes have already been observed: a bare absolute path, @ followed by an absolute path, and a single x. Two predicates that do not depend on shape, used together over every reply:

# SUSPECTS, not proof — a body with no whitespace at all
jq -r '.[] | select(.in_reply_to_id != null and (.body | test("\\s") | not)) | "\(.id) \(.body)"' comments.json
# every short reply — the backstop for a stub that DOES contain whitespace
jq -r '.[] | select(.in_reply_to_id != null and (.body|length) < 200) | "\(.id) \(.body)"' comments.json

🚨 Neither predicate is a gate, and the first is a high-signal SUSPECT list rather than proof. It has false positives — a bare commit SHA, or a lone #1234, is a legitimate whitespace-free reply — and false negatives, because a placeholder containing a space passes it, which is why the second predicate is the backstop and not a refinement. Both are scoped to in_reply_to_id != null: a top-level review comment is not a reply, and counting one as a stub is how a census overstates. Read every hit. The discriminator is what the body NAMES: a real answer names a commit or a finding, a stub names a file. The only thing here safe to automate is the repair verification below, where a byte-compare against the posted file is exact.

Repair with PATCH, never a second POST — a new reply leaves the stub standing beside it:

gh api -X PATCH repos/<owner>/<repo>/pulls/comments/<stub id> -F body=@reply.md   # -F, never -f

🚨 Verify by BYTE-COMPARING the fetched body against the file, not by the exit status, not by the reply existing, and not by its length. "A comment exists whose in_reply_to_id is the thread root" passes on the very stub it is meant to catch, and length cannot separate two paths of similar length. Fetch pulls/comments/<id> afterwards and assert equality with the file's contents.

Two traps when sweeping a repo for stubs, both of which produce a confident undercount:

• Page to exhaustion, and get the denominator from the Link header (rel="last"), not from a fixed page budget. A 25-page sweep of pulls/comments?per_page=100 caps at 2500 comments; MeshWeaver.Plugins has 46 pages, so such a sweep silently misses the oldest ~45% — which is exactly why #370's August replies were invisible to it while a newer pair was not.
• A secondary-limit 403 lands INSIDE --slurp output as a JSON object among the arrays. Its final element is then an error, not a short last page, so "the last page held 3 rows, therefore pagination was exhausted" is a false pass. Type-check every page (select(type=="array")) and count what you actually read. And note that /rate_limit answering 5000/5000 remaining while calls are refused is the secondary limit — the primary quota is not the signal.

🚨 The check's log says GREEN and the pull request is still BLOCKED

The check now refreshes itself. After a GREEN verdict on any event other than pull_request (and never for a merge_group entry, which is its own commit), the refresh job of node-repo-review-answered.yml (lane / Refresh the verdict branch protection reads in core) re-runs the newest pull_request run of the same workflow for the current head — the WHOLE run (POST …/runs/<id>/rerun), never only its failed jobs (check-review-answered.py --refresh-read-run). The whole run is the right call because what the refresh must produce is a fresh pull_request evaluation of live state, and every job of the lane reads live state. It is also the only call GitHub accepts for the commonest candidate: a pull_request run that a newer arrival REPLACED in the concurrency group while it was pending, so it was cancelled with ZERO jobs and published no check-run at all. For that run rerun-failed-jobs answers 403 This workflow run cannot be retried and rerun is accepted (measured on #6315, run 37807892371); the refresh used rerun-failed-jobs until then and went red on exactly that. The stage advance below keeps rerun-failed-jobs, because its held run is a full test run whose green jobs must not be repeated, and it never revives a cancelled run. The decision is the pure function refresh_action, covered by the script's --self-test. It re-runs that run only if it has completed, is not success, and started before this GREEN verdict was taken. A run that is already green, or started after the verdict, is left alone, and the job prints why. 🚨 A run that is still in flight is WAITED FOR, never skipped (#7187): a run reads in_progress until its LAST job ends, and the job whose check-run branch protection reads may already have finished — red, on a read taken before this verdict's state — while the lane job of the same run is still going (on #7198, run 40194864793's gate job completed at 13:00:58Z and the run was still in_progress when the refresh looked at 13:01:31Z). The old rule, "it reads live state itself", left exactly that red standing. The refresh now polls the run (every 30 s, up to 30 minutes — the job's cap is 45) and judges what it left. A run that has NOT STARTED (requested, pending, queued, waiting) is NOT waited for: it read nothing yet, so its verdict reads live state after this one — and in core it is very often pending in this very run's …-gate concurrency group, where a wait would hold it back for the whole 30 minutes and then fail. Before it POSTs a re-run it reads the pull request AGAIN and judges once more: a push or a closure during the wait makes the candidate an obsolete evaluation, and re-running it would queue it in the …-gate group, where it can displace the current head's pending run. That makes it idempotent per (head sha, answered state), with no loop and no polling: the re-run is itself a pull_request run, which never refreshes, and a re-run made with the workflow token raises no new event. Covered the same way: a pull request that went RED because answers were missing goes green when a later reply's run is GREEN, because that run refreshes too. refresh is the only job in the lane that writes, and it holds actions: write alone. A caller adopting the lane grants it on its uses: job and records the grant in .github/lane-caller-grants.yml in the same change.

🚨 Every older red on the head, not only the pull_request one (the duplicate sweep)

What the paragraph above did not cover. Every run of this workflow publishes its OWN check-run named … Automatic review answered on the head, one per event, and GitHub cannot be told to merge them. The refresh above looked at ONE of them, and only after a GREEN verdict taken on a review or comment event. Measured on MeshWeaver.Plugins#4276 (head 264517aeaf):

• At 14:48Z three runs, 40342994248 (pull_request_review_comment), 40343008573 (pull_request_review) and 40343010678 (pull_request_review_comment), each went red. Each red was true when it was taken: the head owed its own Copilot review, which had been refused for quota.
• Copilot's review landed at 18:28Z, and its event started no run.
• At 18:36Z a person re-ran the pull_request run 40342980035, and it went GREEN. Its refresh did nothing, because a pull_request verdict never refreshed, and no refresh had ever looked at a run of another event.
• A REST merge then answered 405 Required status check "review-answered / Automatic review answered" is failing, although the newest check-run of that name read success. The pull request stayed blocked until the three older runs were re-run by hand. Core #7250 and #7252 merged the same way, and core #7204 carries 12 check-runs of the two names on one head: 8 failures and 4 successes.

So which duplicate branch protection reads is not assumed any more. #4649 measured one selection and #4276 another. After every GREEN verdict, from any event (pull_request included, never merge_group), the refresh job also lists EVERY run of this workflow on the head in one call. It reads each run's jobs only when the run could hold a red: completed runs are read once per attempt, and a run that is green or has not started is never read. It then re-runs the WHOLE run when all of these hold:

• its review check-run (a job named Automatic review answered, or any <caller job> / Automatic review answered) is red;
• the run is completed;
• the run STARTED before this verdict.

The decision is the pure function duplicate_red_action, and run_refresh calls it beside refresh_action off the same listing. Its self-test replays the #4276 head end to end, with negative controls: a duplicate already green, a run red only in its refresh job, a red started after the verdict, a red on an older head, a zero-job cancelled run, a fork, a closed pull request, and a truncated listing, which is RED and never judged. A review job still in flight is waited for. A review job never waits on a refresh, so that wait cannot cycle. A red inside a run that is still finishing, because its own refresh is running, is waited for only when that run is OLDER than this one. Waiting only on older runs is what keeps two refreshes from waiting on each other. 🚨 This is ONE policy, in_flight_policy, and BOTH decision paths use it: the newest pull_request run as well as the sweep. With two separate policies, an older comment run waited on a newer pull_request run whose refresh was waiting on it, and both ran into the 30-minute limit (#7257 review).

🚨 One re-run per concurrency group per refresh, and never into an occupied slot (plan_reruns). The caller's group (…-gate, or …-feedback for review-comment runs) keeps ONE pending run, and a re-run enters it like a new run. A second re-run POSTed into the same group would replace the first before it evaluates anything. The replaced attempt is cancelled with zero jobs, its red stands, and the refresh would still report success.

So the reds are re-run as a CHAIN. The chosen re-run judges live state, and when it is GREEN its own refresh re-runs the next red, which also started before that verdict. A group that already holds a pending run gets no re-run: that run is the next link. A re-run that stays red stops the chain there, and that red is true. The self-test replays such a chain to convergence.

Two more shapes are covered (#7257 review):

• A re-run replaced while pending. Its latest attempt is cancelled with ZERO jobs, while the EARLIER attempt's red check-run still stands. So the refresh judges the newest attempt that did create jobs.
• A run whose review jobs DISAGREE. This happens only in core, which runs both answered and lane: say the bare job is red and the lane job is green. The green job's refresh cannot re-run its own run, and an older refresh does not wait on it. A newer run whose review jobs are ALL red IS waited for, because its refresh is skipped and it waits on nothing. The retry listener re-runs the disagreeing run when it completes (self_contradicting), under the same three-attempts-in-a-row bound as an API outage.

A re-run softens nothing: it applies the whole predicate to live state, so a thread that is really unanswered is red there too.

Why not ONE stable check-run per head, updated by every event. That would also clear the red history on the pull request page, which a re-run does not do: each re-run replaces its run's check-run, so the page ends green, but the older attempts stay in the run history. It is not the fix in this change, for four reasons:

1. A workflow job ALWAYS publishes a check-run under its own name, one per run, and nothing in a workflow suppresses it. So a single verdict needs the job RENAMED, so that its check-runs stop carrying the required name, and the verdict written through the API under that name.
2. That moves the producer of a required context in every repository. Core's ruleset requires the bare Automatic review answered. MeshWeaver.Plugins' classic protection requires review-answered / Automatic review answered pinned to app_id 15368 (both measured 2026-10-26). Under that pin a commit STATUS can never satisfy the context, and a check-run counts only if the GitHub Actions app wrote it. Changing a required context is a protection edit, which is the maintainer's call, and it needs the rename procedure in Renaming a Required Check.
3. A fork pull request's token is read-only, so it could write no verdict at all. Under classic protection the context would then block forever. Today the gate judges forks in full.
4. Not established: whether the workflow token may PATCH a check-run that another run's job created, and whether a check-run created through the API from a pull_request_review_comment run lands in a check suite the rollup reads. #4649 showed that the comment runs' own check-runs were absent from the rollup, and nobody has measured why.

The re-run sweep needs no protection edit, and every caller gets it when its pin moves. The stable verdict stays recorded here as the shape to measure next.

The staged pipeline's hold is re-evaluated on the same reply events by the existing listener, stage-advance.yml → node-repo-stage-advance.yml. Its on-answer job fires on pull_request_review_comment: created and pull_request_review: submitted and re-runs the failed jobs of the head's held dotnet-test.yml run once stage 1 is green.

Manual fallback. The self-refresh cannot help in two cases. A fork pull request's token cannot re-run a workflow, so the job warns and names this remedy. And if the refresh job itself is red, it names what it could not read. In both cases, re-run EVERY run of the check on the head that is not success, whatever its event (the duplicate sweep above says why):

# every run of this check on the head that still carries a red (any event)
gh api "repos/Systemorph/MeshWeaver/actions/workflows/review-answered.yml/runs?head_sha=<HEAD SHA>&per_page=100" \
  --jq '.workflow_runs[] | select(.status=="completed" and .conclusion!="success") | "\(.id) \(.event) \(.conclusion)"'
gh api -X POST "repos/Systemorph/MeshWeaver/actions/runs/<ID>/rerun"   # once per id listed

An empty commit does the same thing by a longer road — it gives the check a head sha it has never judged — but 🚨 --allow-empty does not mean "empty": it means "allow a commit that would be empty", and anything already staged rides along. Under pressure, with a half-staged tree, that pushes an unreviewed change while you are trying to repair a check. Refuse a dirty index first:

git diff --cached --quiet || { echo "REFUSING: the index is not empty — this would commit staged work"; exit 1; }
git commit --allow-empty -m "chore: one clean head for the review gate"
git push

Prefer the re-run above: it changes no history and cannot carry anything with it.

Measured on #4652 (2026-09-17), the first time the remedy was used: BLOCKED / rollup FAILURE → CLEAN / rollup SUCCESS / isInMergeQueue: true, within a minute of the re-run. Re-arming auto-merge first did not clear it — the rollup was not stale in the usual sense, it was citing a real, still-current failure taken in a suite nobody reads. Measured again on #4662 (2026-09-18): two pull_request runs re-run, both green, blocked → clean and into the queue.

🚨 This re-run is legitimate, and it is not "re-run and see". The check reads LIVE state, and that state genuinely changed when the replies landed: the first evaluation was correct when it ran, and so is the second. This is the one case where re-running a red is the right response rather than a way of hiding one — and it is safe to write down because if a thread is genuinely unanswered the re-run is red too. (Every other red in this repository still means investigate, not re-run.)

What is actually happening (measured on #4649, head 94fc73b4fdaa, 2026-09-17). Answering a thread fires both a pull_request_review and a pull_request_review_comment event. Fifteen runs of this workflow resulted; eight published a check-run; and the statusCheckRollup that branch protection consumes carried exactly three:

So the mechanism is not a latched rollup and not "the newest run loses":

1. A pull_request_review_comment run's verdict is invisible to branch protection. Both successes were from that event, so they were never candidates — by construction, not by timing.
2. The read-eligible runs that arrived after the replies were EVICTED. pull_request_review runs at 19:48:19Z and 19:48:28Z were cancelled in the one pending slot they shared with the comment-driven runs, so they published nothing and the last read verdict on that sha stayed an early failure.
3. Hence the symptom: the newest check-run reads success, the check's own log says "GREEN — 5 of 5 answered by a person", and the rollup correctly reports a real failure in a suite nobody was looking at. Re-arming auto-merge does nothing, because the rollup is not stale.
4. It is invisible in REST, because mergeable_state: blocked is also what a PR waiting its turn reports. The rollup is the only place the difference shows.

What was done about it. Two changes, and one option that is not ours to take:

• The concurrency group is split by event class (gate for pull_request / pull_request_review / merge_group, feedback for pull_request_review_comment), so the survivor of the gate lane is always a run whose verdict is read. That removes cause 2 deterministically.
• --settle-replies 60 makes that surviving run judge a state that has stopped moving, so it publishes the settled verdict rather than a snapshot taken mid-answer. Every gap measured inside an answering burst was 1–11 s (#4649, #4656, #4646); 60 s is ~5×, and the one 954 s gap in that sample was a separate later round that rightly gets its own evaluation. It softens no verdict — an unanswered thread that stays unanswered is still red when the wait ends — and it waits only while unanswered threads are the whole complaint, never for a missing review or an incomplete listing.
• Not taken, because it needs the maintainer: reporting the verdict as a commit status against the head sha (POST /repos/{o}/{r}/statuses/{sha}) instead of as a job's check-run. That is the only shape correct by construction — one context, last write wins, whatever event produced it — but it is a required-context rename plus a ruleset edit on 2128472, i.e. a change to how everything merges. Until that is decided, the two changes above plus the re-run remedy are what stands.

🚨 And a reply must be ON THE THREAD. A pull-request-level comment does not answer anything: the check counts replies whose in_reply_to_id is the reviewer's root comment, and its log names every thread it still considers unanswered. If the log disagrees with what you think you answered, read that list before reaching for the re-run.

The arm gate — auto-merge is armed only on a reviewed, answered head

Measured 2026-10-03/04 on MeshWeaver.Plugins: auto-arm.yml armed auto-merge on every open, push and undraft, BEFORE the internal review had run on the new head. On Plugins the review is comment-only and nothing required waits for it, so #2549 (6 of 6 findings unanswered), #2643/#2647 (reviewer unavailable) and Memex#641 merged unreviewed or unanswered, and agents disarmed by hand after every push (#2791 twice in an hour).

Arming is a decision, so it moved to the control plane (policy review-then-suites). The control instance's PR steward (MeshWeaver.Plugins PrArming, App systemorph-com) already reads every fleet pull request's review state and answered-findings verdict; when it is enabled, it is the one place that arms a reviewed pull request. While it is switched off (Hosting:PrBabysitter:Enabled, currently off fleet-wide), nothing arms a reviewed pull request, and a person merges it once it is green and answered. No workflow arms auto-merge except for generated-only App pull requests, which owe no review (see The one arm a workflow takes below) — ArmedMergeMustTriggerMainsPushLanesGuard.NoWorkflowArmsAutoMerge fails the build on any other.

What the control plane arms on

It arms a pull request only when all of these hold, read on its current head:

1. it is open, not a draft, and its head branch is in the same repository (a fork never arms itself);
2. its base is the repository's default branch, read from repos/{o}/{r} and never assumed. An unprotected base has an empty required set, so an arm there is an immediate merge (MeshWeaver.Plugins#1685 merged 61 seconds after it opened, before its own CI started);
3. the head has its own review: a Copilot review submitted against the current head (policy copilot-code-review; commit_id equal to the head, not PENDING, not a refusal — copilot_review_on, see Copilot is the reviewer again above), or, while one is still posted, a completed internal-review check run of the systemorph-com App (id 4918443) on the head whose newest run is not the neutral Reviewer unavailable … degradation. The degradation releases the merge gate's review condition (a down reviewer must not hold every pull request), but it is not a review — arming on it would land an unreviewed change nobody decided to merge, so a person merges it by hand;
4. every thread the automatic reviewer opened has a reply from a person, following in_reply_to_id to the root, read from a provably complete listing — the same reviewer_threads / listing_incomplete predicate this gate uses, so the two can never disagree;
5. every required status check of the base is success on the head (required_checks_green, policy review-then-suites — see Staged Pull Request Pipeline).

The reference implementation of 1 and 3–5 is check-review-answered.py --arm-gate (arm_readiness), and its --self-test cases are the test vectors the control plane's port must pass. The arm is the GraphQL enablePullRequestAutoMerge mutation (or enqueuePullRequest on a base with a merge queue) carrying expectedHeadOid = the head it judged, so an arm that races a push is refused by GitHub instead of landing on the new, unreviewed head. It is attempted once per head and read back (autoMergeRequest / mergeQueueEntry); a head a person disarmed is not re-armed.

🚨 The port has to keep up with the reference. PrArming is a one-for-one port of arm_readiness, so a condition widened here is not live until the port carries it. Copilot's acceptance in condition 3 is exactly such a widening (the register lists the port as owed under copilot-code-review): until PrArming accepts a Copilot review of the head, a head reviewed only by Copilot is never armed by the control plane and needs a person's arm.

What stays in the workflow: the disarm

A push invalidates the review, and GitHub keeps auto-merge armed across a push by anyone with write access — so a pull request armed on a reviewed head would merge the next, unreviewed head the moment the required contexts pass. auto-arm.yml (still the fleet's one lane, still called by every repository through workflow_call, its name kept so no caller changes) therefore now does exactly one thing: on synchronize, if the pull request is armed, it takes the arm off (gh pr merge --disable-auto) and reads the result back. It reacts to the event itself, so a fast CI cannot outrun it, and it tolerates no failed step: a disarm that silently did not happen is the hole reopened. It uses the meshweaver-cloud token it always used (contents: write, pull-requests: write) — no checks permission, and no change to any caller's grants.

The one arm a workflow takes: generated-only App pull requests

The control plane's PR steward is model-driven and is switched off fleet-wide (Hosting:PrBabysitter:Enabled) because it spent the model budget. It was also the only thing that armed the generated pull requests main's own jobs open on MeshWeaver.Plugins — settle-locks (every manifest.lock) and stamp-floors (mesh-floor.lock plus each root index.json's minMeshVersion). Without it, Plugins #3236 sat green and unarmed for about 10.5 hours, and no Plugins merge published for about 13.5 hours until a person merged it.

Such a pull request owes no review: the generated_only rule (author and every commit the meshweaver-cloud App, every changed file a lock or a floor-only index.json line) already releases the stage gate and the required verdict. Arming it therefore decides nothing a review would have decided, so a deterministic executor arms it: auto-arm.yml's second job, arm-generated.

• It runs after the disarm (needs: arm) on every forwarded event: open, reopen, undraft and push. On a push the arm comes off first, then goes back on only if the new head is still generated-only.
• The decision is check-review-answered.py --arm-generated (arm_generated_verdict), fetched from the platform's default branch, never from the pull request's tree. It is true only when the pull request is open, not a draft, a branch of the same repository, onto the default branch, and generated_only holds over file and commit listings read in full, with the head unchanged across the reads. Any doubt answers false, and a failed read is a red run that armed nothing.
• The arm is gh pr merge --auto --merge --match-head-commit <the head it read>, made with the App token (never GITHUB_TOKEN, see #2916) and read back. The required checks still gate the merge.
• A person's pull request, another bot's pull request, and a mixed one are never armed here. The self-test cases under arm: (generated → arm; mixed or human → never; unreadable or short file list → never; moved head → never) are the test vectors, and ArmedMergeMustTriggerMainsPushLanesGuard.NoWorkflowArmsAutoMerge exempts this step only while it is gated on that verdict and pinned to the head.

Related

• The Merge Queue — the queue this check runs in, and the steward named in the rollout
• Reading CI Signals — why a skipped required context counts as satisfied