A pass must say that it ran

A liveness alarm that cannot name which failure it is looking at is one instrument short, not one threshold short. Before this change the fleet watch's silence had three possible causes — a pass running long, a pass hung mid-flight, and a schedule that had stopped entirely — and the evidence for all three was the same thing: Ops/Status/* readings ageing out.

What was measured, and by whom

Two earlier sessions established the shape of the problem on the control instance, read-only; this page builds on their measurements rather than restating them as new.

The two promises, and why collapsing them was self-referential

A Hosting/DeploymentStatus reading declares staleAfter = sampledAt + 3 × interval (FleetWatchCadence). That is a promise about observation, and its tolerance is sized for READER noise — three missed passes, so a healthy fleet never flickers. It is stamped with the row's own clock, i.e. at whatever instant inside the pass that row happened to land.

The heartbeat held the sweep's LIVENESS against that same instant. The pass budget is derived from the same tolerance (PassBudget = Tolerance − interval), and the budget between the two is exactly zero:

pass N starts at S, its last successful row is observed at T_last, the pass ends at E
reading deadline   = T_last + 3 × interval          = T_last + 15 min
pass N+1 opens at  = E + interval                   (the sweep is self-clocking)
its first write at = E + interval + t1              (t1 = one row's observe + write)
legal ceiling      = E ≤ S + PassBudget             = S + 10 min

slack = 10 min − (E − T_last) − t1

A pass whose rows are all fast (T_last = S + 30 s) and whose Apply tail legally runs to E = S + 10 min re-arms at S + 15 min + t1 against a deadline of S + 15 min 30 s. It is breached as soon as t1 > 30 s, which the 60-second ObservationTimeout floor alone makes possible. A pass that is alive and inside every one of its budgets could trip OVERDUE. The alarm's own sentence then said "the fleet watch is OVERDUE … the sweep is not being observed" about a sweep that was running.

🚨 This is a real defect and it is NOT the cause of the hours-scale silences above — it is worth seconds to a minute or two, not nine hours. It is fixed here because the two promises are different promises, not because it explains the ticket.

The other direction of the same collapse: a pass that RAN and wrote NOTHING ages every reading out too, so alive and blind reported identically to never ran — and the sweep's own Unobserved findings, which do carry that condition to the bell, were the thing nobody looked at because the heartbeat had already fired on the same evidence.

What the pass records now

One node per sweeper — {space}/Watch/fleet-watch, a Hosting/FleetWatchPass — rewritten twice a pass. The pass HISTORY is that node's own version history: one node per pass would add 288 nodes a day to a space whose whole point is that an observer must not become the load it observes.

Open, written after the roster frame and BEFORE the first row:

Close, written after Apply — including on the abort path:

🚨 Neither deadline is padded, and neither may omit a stage. The pass's work is cut at mustFinishBy and the record is written after that cut, so the OPEN record's deadline is mustFinishBy + WriteTimeout: a deadline excluding the write it is carried by would fire on every pass that spent its whole budget — an alarm going off because its own evidence was still in flight. The CLOSE record's is the same argument over a longer chain, and getting it wrong reproduced the very defect this record exists to remove:

nextRecordDueBy = finishedAt
                + WriteTimeout      the close write — finishedAt is taken BEFORE it
                + interval          the wait, which starts only once that write completes
                + OpenLatency       the successor's roster prologue (one QueryTimeout per roster
                                    home, read serially) plus its own open write

A deadline naming only interval + WriteTimeout is shorter than the schedule it describes: at today's budgets a perfectly healthy successor lands up to 90 s late against it. The open record cannot be moved ahead of the roster prologue instead, because a pass cannot state its DENOMINATOR before it has read the roster — and k of n with an unknown n is the thing that made slow and blind indistinguishable in the first place.

🚨 k of n counts what reached the MESH, not what the pass decided. Three early returns bypass the status write entirely — a dormant record, a record with no namespace, the catch-all — and a row that overran its budget never reaches it. So a pass can complete, report swept, and advance no sampledAt at all; that is the case the numerator exists to make visible.

🚨 An INACTIVE pass promises nothing, and its steady state records nothing. Every installation that is not the fleet control instance completes a pass with nothing to sweep; a record every five minutes there would be load an observer must not become. "Nothing here has swept" is a different sentence from "the sweep stopped", and the absence of a record is how it is said.

But the FIRST inactive pass after anything else writes ONE record that SUPERSEDES — closed, outcome: inactive, and no nextRecordDueBy. Without it the last ACTIVE pass's promise stands for ever, so a portal whose deployment records have all been removed would be reported as a stopped schedule while its sweep completes inactive passes normally: a permanent red, which is the noise-gets-muted path this whole subsystem exists to avoid. "Was the previous pass inactive" is read off the sweep's own liveness — never from a point read of a path that may not exist.

The three verdicts that stop looking alike

FleetWatchPassReading.Of(record, now) is pure over a record and a clock, and the heartbeat mirrors the same rule over the node's wire fields:

hung and stopped are different defects with different fixes, and before this record they were one silence. The heartbeat holds the pass record's deadline in preference to the readings' whenever a pass record exists; where none does — an older Hosting package, or a portal where the sweep never armed — every verdict is exactly what it was.

🚨 On the pass path the NEWEST record SUPERSEDES, which is the opposite of the rule the observation chain follows, and deliberately so. Track keeps the newest reading that made a promise (MeshWeaver#4611) because MANY status rows share one sampledAt, so which one was "newest" came down to listing order and a promise-less row could disarm the dead-man's switch by accident. The pass record is ONE path with a monotonic recordedAt: "the newest record on it" is unambiguous, and superseding is exactly what the writer means when a pass with nothing to sweep publishes no further promise. Both rules are asserted, each against the case the other would get wrong.

🚨 And a missed PASS deadline does not make the readings history. The pass deadline is deliberately SHORTER than a reading's freshness window and the pass write is deliberately non-fatal, so that verdict can fire while every status node is fresh. The overdue sentence therefore says what its own evidence establishes and points at each reading's staleAfter for the question it does not answer; the observation branch keeps the HISTORY claim, where it is what the evidence says.

And the record is a RETROSPECTIVE discriminator

pass counts from 1 at each ARMING and armedAt names that arming. So a silence that ends at pass 1 with a new armedAt is a schedule that was GONE and a fresh activation that re-armed it; one that ends at pass 137 on the same armedAt is a schedule that never stopped, and the silence happened INSIDE a pass. Neither was answerable from outside the process before, and the version history of the pass node now answers it after the fact — which is exactly what the two sessions above had to reconstruct from status-node intervals.

Two more things this change fixes, both in the schedule itself

The report's publication used to sit OUTSIDE the pass's fault boundary. The chain was Defer(SweepOnce).Timeout(…).Catch(→ a Report).DefaultIfEmpty(…).Do(stamp + log).SelectMany(Timer), and .Repeat() re-subscribes on COMPLETION — never on error. Stamp publishes into a BehaviorSubject, whose OnNext rethrows whatever a subscriber throws, so one faulting reader of the documented liveness surface would have terminated the sweep for the life of that hub, leaving a single log line behind. Nothing subscribes today, which is why this is a latent fault and not the observed silence — and it is one subscriber away from being it. The publication is now inside the boundary: a reader's fault is REPORTED at Error naming what happened, the pass it was reporting stands, and the schedule continues.

The terminal line was a WARNING. Reaching it means the schedule has stopped for the life of the hub, so that one line is the whole of what a person gets. It is now an Error, on the same argument the heartbeat's own overdue line is levelled on: an Error line is what the fleet's log watcher turns into a LogIncident and a ticket from outside the process, and a warning is read by nobody. The cost is one line per hub, at most once ever. It names the pass record's path, so the reader's next step is a get.

The passes that swept NOTHING — an empty roster read as nobody (MeshWeaver#5011, #5259)

The first reading the pass record made possible named a cause neither hypothesis above had. On memex.meshweaver.cloud, 2026-09-22, Ops/Watch/fleet-watch versions 63 and 66 are fresh activations whose pass 1 concluded inactive — 0 of 0 row(s) in about 100 ms, and version 67 is pass 2 of the version-66 activation, opening over the 4 Hosting/Deployment records that exist. Same hub, same queries, minutes apart: the roster's answer depended on the pass, not on the store.

The mechanism — established by construction and by precedent, NOT by reading the identity those two passes actually carried, which nothing records — is the one MeshWeaver#4958 measured on SelfUpdateRouting, on the same hub. The roster and issue reads built MeshQueryRequest.FromQuery(query) with NO identity, sequenced through ToObservable().Concat(), on Hosting/PlatformBuilds — whose own ambient identity is by design nobody. MeshService.Query stamps the viewer at call time off the delivering thread, so a pass saw the fleet only when that thread happened to carry the system context left behind by an earlier AsSystem write. When it carried nobody, row-level security answered both homes with complete, EMPTY frames — and an empty roster is exactly how this sweep says "this portal is not the control instance": the pass wrote no reading, opened no record and promised nothing, and every Ops/Status reading aged into history behind a schedule that was, by every in-process measure, running.

That also reads the 37.5 h silence of 2026-09-19 08:47Z → 09-20 22:22Z (the build reading's version history on the same portal) without any hung pass: before the pass record existed, an activation whose chain carried nobody could run inactive passes indefinitely, because an inactive pass performed no AsSystem write that could have changed what the next pass's thread carried. That is an inference consistent with the evidence, not a measurement — nothing of that activation survives to read.

The fix puts the identity ON the request — FleetWatch.IndexRequest(query) = FromQuery(query).AsSystem().RequireViewer() — so it survives every scheduler and pool hop, and a request that still reached a provider unresolved throws, which the sweep reports as a BLIND roster (Unobserved) rather than as an inactive one. The same identity-less shape was swept on every other index read armed on the two anchored inbox hubs: the _Inbox drains of PlatformBuildInboxWatcher and ModuleInventoryInboxWatcher, OperationalSpaceProvisioning's grant and subtree snapshots (read as nobody, every grant looks absent), and TriageIntake.Snapshot. The test pins the construction — TheIndexIsAskedAsSystemByValue_WhateverIsAmbient resolves each request with nothing and with a stranger ambient, and its control arm resolves the shipped shape as the anonymous visitor.

What this does NOT establish

Stated plainly, because a control's denominator is part of its answer.

• The identity fix does not stop the hub RE-ACTIVATING. The same version history shows fresh arming instants several times an hour, clustering on writes to the Hosting package record (an install or update recycles the package's hubs). With this fix each fresh activation sweeps on its first pass instead of gambling on its thread; a window in which NO activation exists — nothing re-reaching Hosting/PlatformBuilds after a recycle — is a different gap, and whether #5259's 45 minutes (11:57Z → 12:42Z, before the pass record shipped) was that or an inactive activation is not established.

• The hours-scale silences are NOT explained. The deadline anchor is worth seconds; the observed gaps are 9 h and 12 h. The candidate root causes remain (a) the sweep's subscription terminating — now reported at Error and no longer reachable through the publication — and (b) the hub the sweep is armed on going away, since the sweep's lifetime IS that hub's (WithInitialization + RegisterForDisposal) and nothing re-arms it until something next reaches that address. The loop itself, re-read for this change, is otherwise sound: every observation failure becomes a reading rather than a fault, and every cycle writes.

• The discriminating measurement was NOT run. It is a Logs Hosting/InstanceAction on the control instance grepping for [FleetWatch] the sweep schedule STOPPED on versus [FleetWatch] pass FAILED on across the START of a gap — a WRITE on a live portal, and this change was made read-only. It remains the outstanding step, and from now on the pass record answers most of it without a log query.

• The ticket's own opening evidence may not be about the sweep at all. The OVERDUE line it was filed on came from the heartbeat's newest-reading number, and that number was measured to be a day and a half wrong while 440 newer versions of the same node existed — a per-path GetMeshNodeStream that stops delivering freezes the guard's input. It errs toward alarm, which is the safe direction, but it is a SECOND defect and this change does not fix it. The pass record narrows the blast radius (a frozen status stream no longer decides the liveness verdict) without repairing the stream.

• A pass record that never lands is silent about itself. The write is budgeted and never faults the pass, so a record that could not be written leaves the previous record's deadline to be reported as broken — correct, but the sentence then hedges between "the pass hung" and "its record never arrived", and only the log line distinguishes them.

• The heartbeat's STATE stays Beating for a sweep that is alive and blind. The pass census is in the sentence (0 of 5 row(s) written) and the Critical Unobserved findings are on the bell, but the state does not degrade on it. What does degrade /health is the READINGS' census (MeshWeaver#5011, StatusFreshness): a sweep that runs and writes nothing lets every Ops/Status reading age past its own staleAfter, and IsProblemAt names each one — so the check reads Degraded on the evidence that the fleet is unobserved, beside a state that says the sweep is running. Two findings, kept apart on purpose.

• A portal that stops being the control instance still shows an observation-chain red. The superseding record above disarms the PASS chain, but the stale Ops/Status/* readings keep their own promises, so the observation chain reports OVERDUE about readings that are genuinely history. That behaviour predates this change and is not addressed here; what this removes is the pass chain adding a stronger and less defensible claim on top of it.

How this was verified

🚨 The publication case FAILED on its first run, for a real reason: a BehaviorSubject replays its current value to a new subscriber INSIDE Subscribe, so a thrower attached without Skip(1) faults the subscription itself and the case measured its own setup. That is recorded in the test, because the next person to write one will hit it too.

🚨 The mutation is what makes the new cases non-vacuous: the negative control passes with or without the change (as it must — a portal publishing no pass record has to behave exactly as before), so its green means something only because the other three are red in the same run.

Related

• FleetWatch — the sweep itself, its budgets and its denominator.
• StatusFreshness — the OTHER promise: what a reading declares about its own shelf life, and the reader that judges it.
• Controls That Cannot Fail — the family this belongs to.