Roll churn is recorded, not ticketed

Systemorph/MeshWeaver#4802 (canonical) and #2906 (the decision it implements). In one day, 2026-09-23 on memex-cloud, the same Orleans log site filed #5597, #5598, #5602, #5606, #5607 and #5619 — on top of the dozens already folded into #4802.

The site

Orleans' memory stream keeps each queue in a MemoryStreamQueueGrain that lives in the RAM of one silo. When that silo leaves, every PersistentStreamPullingAgent reading the queue logs fail: —

under Orleans.Providers.MemoryAdapterReceiver…Memory.memory-N or Orleans.Streams.Memory — and then recovers on its own once the queue re-activates on a surviving silo. Orleans owns the line and its level; there is no MeshWeaver frame on it. The real remedy is to retire AddMemoryStreams, and that is blocked on the stream's two remaining production producers (see #4802's 2026-09-22 comment), so until then every roll ticketed a fault nobody can act on.

The rule — and what it refuses to classify

A burst is roll churn only when both hold (RollChurnClassifier, in MeshWeaver.Observability.Contract):

  1. it is that site: a memory-stream reader category, a SiloUnavailableException or TimeoutException, and a call to a memorystreamqueue/ grain; and
  2. a pod of the same workload (the pod name without its ReplicaSet hash and id) logged Application is shutting down... within five minutes BEFORE the burst.

The window is derived, not tuned: the host's ShutdownTimeout (90 s, the silo holds its queue grains until it has stopped), a Dequeue in flight to it timing out (30 s), membership declaring an unclean stop dead (ProbeTimeout 15 s × NumMissedProbesLimit 5), plus the agent's retry of that read.

🚨 The same failure with no preceding exit keeps filing. That is a silo that stalled, or was killed without shutting down — an OOM kill logs no shutdown line — which is exactly the case #2906's analysis said must stay visible. Measured on memex-cloud on 2026-09-23 (Logs actions Ops/logs-memexcloud-20260924-memstream-rollchurn-shutdown and …-started):

failure nearest preceding shutdown line verdict
17:57:41Z / 17:58:22Z, Dequeue timeout …-4d84f9-ncwwf, 202 s / 243 s before roll churn
20:13:53Z, SiloUnavailableException …-776847-vq68t, 26 s before roll churn
18:11:08Z / 18:12:09Z, timeout + SiloUnavailableException none for 17 minutes still files

The 18:11Z pair is the instructive one: pod …-4d84f9-n7g6b logged a fresh Application started at 18:13:12Z with no shutdown line before it — it was killed, not stopped. The classifier leaves those lines on their ordinary fingerprint, where they are the only red evidence of that kill.

Recorded, never dropped — and never able to hide a steady-state occurrence

A classified burst is not discarded. It goes to its own incident, Admin/_LogIncident/roll-churn-{siteFold} — one per log site — which the portal records as Suppressed from its first sighting, with classification: RollChurn and the evidence that classified the latest burst ("<pod> logged 'Application is shutting down...' at <t>, N s before this read failed"). It is counted and sampled like any incident, and it asks for nothing, so no triage round and no issue.

Because it is a different node, a later occurrence of the same site outside a roll cannot fold into the suppressed incident and disappear: it keeps the site's ordinary identity and is triaged as before. That is why the classification changes the fingerprint rather than the status of the existing incident — suppressing the ordinary incident would have silenced both.

The watcher also logs one Information line per classified report, naming the fingerprint and the evidence, so the watcher's own log shows what it did not triage.

Where the exit comes from

The watcher's query is deliberately unfiltered, so the host's own Microsoft.Hosting.Lifetime line (Information in the portal's appsettings.json) is in the same stream as the failure. A pass reads one page per poll interval and the failure trails its exit by up to minutes, so the watcher carries the exits it has seen forward, per namespace, for one window. They are held in memory by design: a watcher restart forgets them, which can only make a churn burst file as an ordinary fault — never hide one.

The contract is additive in both directions: an older watcher sends no classification and everything files as before; an older portal ignores the field, and the roll-churn-… incident is triaged as an ordinary one.