The Subsystem in the Title Is Not the Cause

An auto-filed incident is titled after the logger that emitted the line, and the ticket body's "probable cause" is written from that line alone. Both are honest and neither is a diagnosis: a log site sees the condition it noticed, not the condition that caused it. So a cluster of tickets that share a subsystem name may share nothing else.

Reading a Routing Saturation Report is the extreme case — 78 tickets on one gauge, one genuine routing defect. This page is the same discipline applied to a small cluster, and it records what changed when each ticket's evidence was re-read rather than its title.

Ask, per ticket: which way does the arrow run?

Three questions, in order, before believing the subsystem in the title:

  1. Is this subsystem the SUBJECT or the INSTRUMENT? A router that reports a delivery failure may be reporting a target that died, a process that is exiting, or a silo that has lost its CPU. Only the first is a routing fault.
  2. Is the failure REAL, or real-but-mis-reported? A DeliveryFailure surfaced to a sender is as often a classification question as a delivery one: the condition happened, and what is wrong is the verdict attached to it. That is a fix, not a silencing — but it is a different fix from making the condition stop.
  3. Does the root have a record of its own? If not, folding the ticket deletes the only record there is. A ticket whose cause is genuinely elsewhere may be closed pointing at that elsewhere; a ticket whose cause is nowhere must stay open, with its remaining work corrected.

The cluster, per ticket

ticket subsystem in the title where the arrow actually starts
pod-hub DeliveryFailure (#2299) routing routing's own — one of the few. A misrouted delivery gets the right refusal; the verdict attached to it was terminal because the classifier's discriminator was never passed on that leg. See A Departed Silo Is Not A Delivery Defect
cancelled pod-hub deliveries (#5034) message delivery the router's own process going away, not the sender's operation — see the correction below
stream-routed enqueue timeout (#2322) memory streams a silo-local stall: the queue grain's turn could not run, so the publish timed out. A co-located caller timed out on the same activation, which rules a network partition out. The stream leg was the instrument
pulling agents lose queue grains (#4802) memory streams deployment churn meeting a RAM-resident queue grain. Structural, recorded, and its remaining work is not what the ticket says — see below
placement retries (#4890) silo churn deployment churn, absorbed by Orleans' own placement retry. Every occurrence was retried, none lost, and no application frame appears in any sample
router-addressed traffic (#1140) messaging the detector's own keying plus node CRUD whose target fell back to the router — never the stated "portal hubs not assigning their own address". See Router Traffic Detection

Two of the six have a cause outside the subsystem they name; one has a cause inside it whose remedy lives in another repository; one is genuinely its subsystem's own.

Correction 1: a pod-hub address suffix is a PROCESS id, not an operation id

A ticket reported two directed deliveries failing 2 ms apart, to import/SpkmR6TjbUq2wZ0QDz3d5w and cache/SpkmR6TjbUq2wZ0QDz3d5w, and reasoned from the shared suffix that both legs belonged to one operation — "a per-operation cancellation token shared by the import activity" — because the sender was an import activity.

The suffix is the mesh id: cache/{meshId}, import/{meshId}, portal/nodeops-{meshId}, portal/reads-{meshId} are the pod hubs of one process. The positive control is in a neighbouring incident, where cache/TmrXhZkCpE6r94nulQBzpQ and portal/nodeops-TmrXhZkCpE6r94nulQBzpQ appear minutes apart on the same pod — the same suffix under two different address types, which an operation id cannot be.

So the two failures are two pod hubs of one target process, cancelled at the same instant. That moves the question from the sender to the router: the routing pool's teardown cancels every accepted leaf, and an activation being deactivated has its outstanding requests cancelled too. Neither is established — the pod was not observed stopping in that window, and this is recorded as the corrected question rather than an answer. What is established is that the ticket's stated mechanism cannot be read off its own evidence.

🚨 A shared identifier is evidence of a shared SOMETHING, and which something is a fact about the naming scheme, not about the incident. Reading it as the sender's operation put the whole investigation on the wrong side of the call.

Correction 2: two "memory streams" tickets, two different live owners

Two tickets carry the memory-streams label and look like one root — a RAM-resident MemoryStreamQueueGrain that dies with its silo. Measured against the code they are not:

The ticket's own stated remedy — "delete the provider registration, it is the last row" — is therefore not available yet, and a session that acted on it would have deleted a live transport. Same label, same grain type, different live traffic, different remaining work.

The rule about folding

Consolidating a symptom ticket into the ticket for its root is how a cluster shrinks honestly. It has one precondition: the root must have a record. A closed issue is a record; a doc page and a pinned test are records. Nothing is not.

Applied here: the churn-absorbed placement ticket can close pointing at the departed-silo root, which has an open issue, a closed sibling, a predicate, a test that pins the production strings verbatim and a doc page. The dequeue ticket cannot close, because the slice it is evidence for has no other record of the fact that the provider is still live in the fleet — so it stays open with its remaining work corrected instead.