A Failure Report Answers Its Own Instruction

The rule: if a report ends by telling the reader to go and find something, the reporting code must first check whether it is already holding it — and if a report carries a field for outstanding work, every producer of that report fills it, because an unfilled one does not abstain, it asserts the opposite. A fact about a PAST event is read from what was RECORDED then, never re-derived now from a structure that has moved since.

All three shapes were measured on live portals in September 2026, within a week, in subsystems that share no code. They are the same defect.

Related: Reading a Disposal Stall Verdict — what a snapshot field actually measures; Bounds Must Be Ordered — why the innermost, most specific answer has to win the race; Controls That Cannot Fail — one word covering several states.

Shape 1 — the report tells you to find what it already knows

MessageService.Dispose() discards deliveries a hub accepted and parked behind its initialization gates, and files each one at Error (event 7301). The line ended:

[DISPOSE-DISCARD] Hub RiskTransfer is disposing with SubscribeRequest (id=…, from cache/…)
still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the
message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. Accepted work
must be drained before a hub goes down; find why this hub disposed before its deferred work
could run.

Find why this hub disposed. The hub knows. MessageHub records disposeRequestedBy, disposeReason and cascadeOwner one frame earlier, in HandleDispose — added precisely so #3510's "[QUIESCE-START] on a root should name who asked" could be satisfied. But [QUIESCE-START] is Information, and the red-log pipeline files Errors. So the one line that becomes an issue was the one line without the attribution.

Measured on Admin/_LogIncident/d2249f800ffc2577: 364 occurrences, 2026-09-08 → 2026-09-14, 13 pods. Every captured sample names the message, its sender and the gates it sat behind. Not one says which teardown threw it away — so the reader cannot tell an operator recycle from a NodeType rebind from an owner's cascade, which are three different investigations.

The fix is not a new measurement. It is passing a value that already exists across ten lines of the same method:

… RunLevel=ShutDown. This teardown was requested by a cascade from its owner mesh; why: the
owner's own teardown — … . Accepted work must be drained before a hub goes down — that
attribution is who to ask why this hub went down with work still parked behind its gates.

The second reader is the one who cannot see the log at all

The stranded sender is in another process as often as not. It gets a NACK, and that NACK said only that the hub had gone away. The same clause therefore goes into the NACK text — the same reasoning that removed the generic disposal sentence from the sibling path in Retiring an Activation.

Attribution must VARY, or it is decoration

DisposalAttribution renders four distinguishable answers, and the absence of a routed request is itself one of them:

Three of those distinctions are load-bearing and none of them can be inferred from the others:

• The cascade form is not the direct-dispose form. A cascaded child IS disposed by a direct Dispose() call from HostedHubsCollection, so before cascadeOwner existed every child in a wave reported a direct Dispose() — indistinguishable from host teardown, and reading the child told you nothing about the root. That was #3510's wedge one level down. The cascade carries the originating cause, not merely the immediate parent, so a leaf names the event that started it however deep the tree is.
• The last row rules the message path out, which is exactly what the production incident needed and could not get.
• A blank reason is an unstated one. Reason is free text from the poster; null was once the only value treated as "not stated", so an empty string rendered a literal why: with nothing after it — the defect again in a new costume. It is normalised at the single capture point, so every reader inherits it rather than each having to remember.

One cause, claimed once

Two writers can record the cause — the DisposeRequest handler on the action block, and the owning collection's cascade note on whatever thread is disposing the owner — so "first cause wins" has to be a claim, not a pair of reads. It is one Interlocked.CompareExchange, and the handler additionally declines to claim at all when the teardown has already begun: Dispose() sets its shutting-down flag and only then posts its first ShutdownRequest, so a routed request arriving in that window still sees RunLevel=Started, is admitted, and its turn runs after the teardown started. Without the check it would overwrite a true a direct Dispose() reading with its own.

A report that names the WRONG cause is worse than one that names none — it is this same defect, pointing somewhere else.

Corollary, and it is the reason this section names a poster: the rendering is only as good as what posters supply. MeshOperations.Recycle — the operations / MCP recycle verb, the one teardown that comes from OUTSIDE the framework's own lifecycle and therefore the one whose cause a log cannot reconstruct — was the single production poster of a DisposeRequest with no Reason. It now states one.

Shape 2 — a field that renders "not measured" the same as "none"

The recursive delete reports every stage failure through one line:

[DeleteNode] timeout path=… stage=<stage> partial-deleted=<n> unanswered=<set>

unanswered= exists so an operator learns WHICH node stopped the operation. The pre-flight stage fills it, and its occurrences read like this:

stage=pre-validate-descendants partial-deleted=0
unanswered=sglauser/AgenticBusiness/01-MeetYourCoworker/AskAdvisor, …

The commit stage never set it. So a commit-stage timeout rendered:

[DeleteNode] timeout path=Hosting/TriageStatus stage=commit partial-deleted=3 unanswered=-
System.TimeoutException: [DeleteNode:commit] the bottom-up delete of 'Hosting/TriageStatus' made
no progress for 30s — 3 path(s) removed from storage so far

(memex-cloud, 2026-09-14T08:45:38Z.)

- is the same rendering the line uses for "there is nothing outstanding". The field did not abstain — it asserted the opposite of the truth: three paths were removed, the drain was stuck on the rest of the subtree, and the report said it owed nothing. And the sentence beside it offers only a count, which identifies what SUCCEEDED and therefore identifies nothing about the failure.

Both halves of the answer are live in that closure: the plan (collected.ToDelete) and the progress (SnapshotProgress()). The difference is the answer, and it costs one set operation on a path that has already failed:

[DeleteNode:commit] the bottom-up delete of 'Hosting/TriageStatus' made no progress for 30s —
3 of 9 planned path(s) removed from storage so far; still owed by the plan:
Hosting/TriageStatus, Hosting/TriageStatus/…

Two details that are not incidental:

• "still owed by the PLAN", not "remaining". A drain can remove MORE paths than it planned — a child created between the plan snapshot and the drain — which is the whole subject of #3392. The set is what the plan still owes; it is never a claim about what else storage may hold.
• The count stays. Reporting real progress rather than 0 is the half of #1198 that landed earlier. Names are ADDED; a rewrite that swapped one fact for another would read as a fix and be a regression.

Shape 3 — a report that re-derives a fact from a structure that has since moved

This is the shape the [] in the very first log excerpt on this page came from, and it is worth separating because the other two are about a fact that was never reached, while this one is about a fact that was reached and then overwritten by the passage of time.

A hub holds its closed initialization gates in one dictionary, and OpenGate removes a gate from it. So gates does not record what held a delivery; it answers "which gates are shut right now". Every report about a PARKED delivery that read that dictionary at report time was therefore describing the hub at report time and not the delivery at park time — and the two disagree in exactly the case a reader most needs the answer:

The empty case is the damaging one, because an empty list does not read as "not measured". It reads as a measurement: nothing was holding it. That is what 364 production Errors said, for six days across 13 pods, while alleging in the same sentence that the delivery was "still deferred behind its initialization gates".

The fix is a field, not a measurement: the deferral tracker records the gate set at the moment it parks the delivery, under the same lock that made the deferral decision, and every report reads it from there:

// Called under gateStateLock at the deferral decision. Teardown opens the gates before
// draining these trackers, so reading gates.Keys during Dispose loses the cause (#3712).
var gatesAtDeferral = string.Join(",", gates.Keys.OrderBy(x => x, StringComparer.Ordinal));

Two facts, separately labelled — not one replacing the other

The live read is not wrong; it is a different fact, and the pair is the diagnosis:

• gates still shut ⇒ the gate is stuck. Look at the dependency that never initialised.
• all of them since opened ⇒ the hub did initialise and the delivery's turn still never ran. Look at what is holding the turn loop. That is a different investigation, and before this change it was indistinguishable from the first — it rendered as the same empty list.

So the recorded set is the subject of the sentence and the live read is stated beside it, each labelled as what it is. Where a hub-level line already exists (the startup-timeout Error names the hub's own still-shut gates) it stays exactly as it was, and the per-delivery answer is ADDED — a rewrite that swapped one fact for the other would read as a fix.

Why the last of the three readers went unfixed for a week

The discard at disposal was fixed the day after it was filed. The other two readers of the same drain — the startup-timeout answer and the 30-second per-message deferral timeout — kept the live read, and the deferral one could not be reached by any test at all, because its budget was a hard-coded static readonly TimeSpan. A path no test can reach is a path whose wording nobody checks, which is why that bound is now per-hub configuration (WithDeferralTimeout) with the default untouched, and why a regression now parks a hub's turn loop across the gate open on purpose:

Hub late-gate/1 deferred GatedRequest (id=…) for >4s; initialization gates closed at deferral:
[gate-the-test-opens] — every gate it was parked behind has SINCE OPENED, so this hub did
initialise and the delivery's turn still never ran — look at what is holding the turn loop,
not at the gates.

Read that report while the loop is still parked — it is logged one line before it is posted, which is the only way to see it, since the DeliveryFailure it produces cannot be routed until the loop is released. That asymmetry is itself worth knowing: a hub can tell the log something it cannot yet tell the caller.

What this rule is not

It is not a licence to downgrade or silence a report. Both changes here leave every level, every event id and every existing fact exactly where they were; they add the fact the reader was told to go and find. The two nearby changes that DID reclassify a line (#4178, a hub discarding its OWN deferred delivery; #3647, a queued turn the pump is about to take) each moved a level only after establishing, by measurement, that the line asserted stranded work with no waiter — and each kept the line, at Debug, naming the same facts. A classification that stops reporting is a silenced fault, not a classification.

Checklist when you write or review a failure report

1. Does it end with an instruction (find why…, check which…)? If so, is the answer reachable from where the line is written? If it is, put it in the line.
2. Does it carry a field for outstanding / unanswered / pending work? Then every producer of that line fills it, and "unset" must not render as "empty".
3. Does the report identify what FAILED, or only count what SUCCEEDED?
4. Does the value VARY with the condition? Write the negative case down — the absence of a cause is itself an answer and has to be spelled as one.
5. Does the party who cannot read this log — a caller in another process — get the same fact in its NACK or response?
6. Is any fact in it about something that happened EARLIER? Then it is read from what was recorded then, not re-derived now — and ask what the re-derived value renders as when the structure has emptied, because an empty collection reads as a measurement, never as an abstention.
7. Can a test reach this line at all? A report behind a hard-coded bound is a report whose wording nobody ever checks — three readers shared one drain here, and the two nobody could reach are the two that stayed wrong.