A frozen reading must read as frozen
The defect is not a wrong answer. It is an OLD answer wearing the present tense. MeshWeaver#4471: the control instance's fleet watch wrote its last observation at 2026-09-16T00:19:40Z and nothing said so for 5.7 hours.
Ops/Status/memex-cloud — DegradedandOps/Status/build — Degradedrendered on the Fleet Console, on the status board and to every agent thatgets the node in exactly the words a reading taken a minute earlier would use. Six hours of no observations were the same colour as six hours of no problems.
Two halves. The cause of that particular freeze — an assembly that failed to LOAD, after which the
hub silently bound the mesh default configuration and every initialiser on its chain, the fleet watch
included, simply never ran — is
MeshWeaver#4536 and its write-up
Doc/Architecture/AnUnloadableBuildIsNeverASilentDefault. This page is the other half, and it is
the durable one: whatever stops the watcher next time, a frozen reading must not be able to read as a
current one, and the silence must reach somebody without a human going to look.
1. A reading declares its own shelf life
Hosting/DeploymentStatus gained two fields, written by the sweep at the moment it writes the
observation (ObservationQueries.StatusContent):
| field | what it says |
|---|---|
refreshEverySeconds |
the cadence the WRITER promises to refresh this node at — the sweep's own interval |
staleAfter |
the instant past which this reading must not be presented as current |
Both are absent on a reading written by a one-off sampler — a hand-run Sample action, the
refresh-status script — because those promise nothing, and "refreshed every five minutes" and
"nobody promised to refresh this" are different statements about the same JSON.
🚨 staleAfter is stored rather than only derived because the reader that matters most runs none
of our code: an agent doing a raw get @Ops/Status/memex-cloud sees the deadline beside the reading
and can judge it with nothing but a clock. Before this it saw sampledAt and had no yardstick at all.
2. ONE cadence, owned by the sweep that makes the promise
FleetWatchCadence (in Hosting/Issue/Source, beside FleetWatch) holds the whole of it:
Default |
5 minutes — the sweep's cadence, and FleetWatch.DefaultInterval now is this value |
Minimum |
1 minute — the floor a misconfigured interval is clamped to |
IntervalKey |
Hosting:FleetWatch:Interval, parsed in one place |
StaleAfterIntervals |
3 — how many missed passes a reader tolerates before it stops calling the reading current |
🚨 The Fleet Console used to carry a second number, and that number WAS the defect. Its
StaleAfter was two hours: twenty-four missed passes on a five-minute cadence, a window inside which
a dead watcher's final verdict renders as the state of the fleet — which is what it did, for 7 h
43 m. It is now derived from the cadence above, and no reader in this package carries a staleness
constant of its own.
Three, not one, for the interval count: a pass can be slow and a write can lose a race with a roll, so a reader that shouted at every late sample would be the noise-that-gets-muted path this whole subsystem exists to avoid. Fifteen minutes is short enough that a dead watcher is caught inside one coffee break.
Precedence when a reader judges one reading (FleetWatchCadence.Of): the node's own
staleAfter wins — it is what the writer promised — then its refreshEverySeconds, then this
portal's configured cadence. A reading that declared nothing is still judged; "nothing promised" must
never read as "fresh forever". The verdict says which of the two it used (Freshness.Declared), so a
reader knows when it fell back.
3. Every reader-facing surface asks the age FIRST
| surface | fresh | stale |
|---|---|---|
| Fleet Console — health column | ⚠️ Degraded |
⏳ Stale · last read Degraded |
| Fleet Console — sampled column | 3 min ago |
⏳ 7 h 42 min ago (09-16 02:19) |
| Fleet Console — attention column | degraded |
Stale — last sampled 7 h 42 min ago; the fleet watch writes every 5 min. Last read Degraded |
| Fleet Console — summary line | 4 instance(s) · none needing attention · 0 never sampled. |
…· **3** with a stale reading · … |
| DeploymentStatus — Content | memex-cloud — Degraded |
memex-cloud — Stale, with the banner leading the page |
| DeploymentStatus — Board | the recorded verdict | the stale cell + the age and the instant |
| Deployment record page — status strip | **Degraded** — 3/3 ready · … |
**⏳ Stale** — 3/3 ready · …, then the sentence |
an agent's get |
sampledAt, staleAfter |
the same two fields; the deadline is in the past |
Two rules hold across all of them:
- The age LEADS, and the reading is carried along rather than discarded. "The watcher last read Degraded" is useful; "Degraded" as a statement about now is false. This is the ordering the console already adopted for its attention column (MeshWeaver.Plugins#1962) — every other cell now agrees with it.
- A current degradation is reported exactly as before. Each case has a negative control: a fresh
Degradedreading still renders⚠️ Degradedwith no age qualifier, so this change cannot become a way of hiding live degradations.
🌍 The words are the viewer's
The freshness vocabulary — the verdict words, the ages, the cadence, the sentence — resolves off
AccessContext.Locale, the same doctrine as InstanceActionTexts one folder over: module-owned text
follows the VIEWER. Veraltet — zuletzt erfasst vor 7 Std. 42 Min.; die Flottenüberwachung schreibt alle 5 Min. The glyphs (⏳ ⚠️ 🔴 ✅) and the numbers carry the state language-neutrally.
The REST of these operator pages is still English, as FleetWatch.md records; that debt is untouched
here and is not made worse — nothing new was added in one language only.
4. Something notices, and it is not on the chain it watches
FleetWatchHeartbeat (in src/MeshWeaver.SelfUpdate.Aks — the Hosting package's compiled half)
is a hosted service that holds the deadline the sweep declared. Each status write re-arms it; the
timer firing IS the finding.
Why there. The sweep is armed inside Hosting/PlatformBuildInbox's configuration lambda —
in-mesh C#, compiled at runtime. On 2026-09-16 that build failed to load, the hub bound the mesh
default configuration, and every initialiser on that chain vanished with no exception reaching
anybody. A guard hosted inside the thing that fails is unreachable exactly when it matters. The
independence of this one is measured, not argued: through those twenty hours the sibling hosted
service in the same assembly — InboxHubAnchor — kept running and kept the dead hub activated.
What it holds. One live listing of {space}/Status for existence (never a point read of a node
that may not exist — that opens the storm breaker), then each node's own stream for content. The
NEWEST reading that declares a deadline is the heartbeat; one timer is armed at that instant.
🚨 It carries no cadence of its own. The deadline it waits on is the writer's own declaration, read off the node. A second staleness constant out here would be the same defect this page removes.
🚨 It is a dead-man's switch, not a watchdog. It never restarts anything, never resubscribes and never retries. Reporting is the whole job.
🚨 It arms the deadline BEFORE it reports "beating", and the timer is absolute. Written the
obvious way — Observable.Timer(…).StartWith(beating) — the verdict is published first and the
deadline armed after, because StartWith is a concat; a reader acting on that verdict is being told
the watch is held to a deadline that does not exist yet. The live case below caught exactly that (it
advanced its clock the instant it saw "beating", and the timer then armed relative to a clock that
had already moved past it). An absolute due time closes the other half: a subscription the scheduler
delays must not push the deadline out by however long it was delayed.
The four things it can say, and none of them is silence:
| state | when | on /health |
|---|---|---|
Beating |
a reading arrived inside its window | Healthy, printed |
Overdue |
the promised reading did not arrive | Degraded, and an Error log line |
NoPromise |
no reading here declares a cadence (every portal that is not the control instance) | Healthy, printed |
Blind |
the detector itself could not read | Degraded — "I could not check" is never a pass |
Who hears it. Two publications of one verdict:
- An
Errorlog line when the deadline passes. The fleet's log watcher —mw-log-watcher, a separate Deployment in themonitoringnamespace, reading Loki — folds everyfail:line into aLogIncident, which the triage agent turns into a tracked GitHub issue. That path is the fleet's existing incident route and it runs outside this process entirely, which is the property that matters: the portal cannot both be the thing that died and the thing that reports it. - The
fleet_watchhealth check, taggedcensusso it prints its reading whether or not there is a problem. A liveness check that is silent while all is well is indistinguishable from one that was never registered — the same ambiguity, one level up.
It is Degraded, never Unhealthy, and carries no probe tag: a monitoring outage must not restart a pod or take it out of rotation.
Why the heartbeat is the WATCHER's, not each deployment's
The verdict is reached on the newest reading across the whole operational space. Judging per
deployment would put a permanent red on any instance the sweep deliberately sets aside — a record that
is Planned, Provisioning or Suspended is not observed at all, so its status node stays frozen by
design. Measured on memex.systemorph.com, 2026-09-17: Ops/Status/partnerre has not moved since
2026-09-15T22:07Z on a record that says Planned, while the sweep writes the other four every five
minutes. A permanent red on that row is exactly the noise MeshWeaver.Plugins#1843 removed from the
bell.
(The READER still dates that row, which is right: partnerre's page says the reading is 31 hours old, because it is.)
5. What this does NOT see
Stated plainly, because a control's denominator is part of its answer.
- A sweep that writes NONSENSE on time. The heartbeat asks whether a reading arrived, not whether
it is true. A pass that observes nothing but writes a status node still reads as beating — that
case is the sweep's own
Unobservedfinding, and it is a different instrument. /healthis not polled after startup. The chart'sstartupProbeis the only automatic reader of/health; after that it answers when an operator, an agent or another portal's sweep asks. So the health entry is the PULL surface, and the log line is the half that reaches somebody unprompted.- The log path has its own dependencies. If Loki is not ingesting, or the watcher Deployment is
down, the
Errorline lands nowhere. That is a smaller failure domain than the portal's own hub chain, which is the point, but it is not zero. - Per-replica anything. The heartbeat is one verdict per process; each replica holds its own and
reports it on its own
/health. Repeated/healthcalls sample different replicas (OperatingFromThePortal.md), so one call answers about one replica you did not choose. - A reading nobody promised. A portal whose observations are all hand-run settles on
NoPromise, which is honest and is not a pass: there is no schedule there to hold anyone to.
6. How this was verified
| run | result |
|---|---|
compile-check.py against the sealed set 3.0.0-ci.8767 |
98 clean · 0 known-debt · 0 NEW break(s) — every NodeType that reads the contract compiles with the shared cadence source in its set |
dotnet build src/MeshWeaver.SelfUpdate.Aks -c Release -warnaserror |
0 Warning(s), 0 Error(s) |
MeshWeaver.SelfUpdate.Aks.Test — FleetWatchHeartbeatTest |
the deadline is driven by a HistoricalScheduler, never a sleep: overdue on silence, beating again on the next write, the newest reading deciding, an already-past deadline firing at once, the three content shapes read the same, and the module's registration (hosted service + the census-tagged check, with no probe tag) |
FleetWatchHeartbeatLiveTest — the same rule on a real mesh |
the listing, the per-node stream and the stored JSON, end to end. It is what caught the arm-after-report ordering above: the pure cases could not see it, because they hand the rule its readings |
the in-mesh Tests areas (Hosting/DeploymentStatus, Hosting/FleetConsole, Hosting/Issue, Hosting/InstanceAction) |
the measured 2026-09-16 freeze replayed on every surface, each with the fresh-reading negative control beside it |
Related
- FleetWatch.md — the sweep that makes the promise.
Doc/Architecture/AnUnloadableBuildIsNeverASilentDefault— why this particular watcher stopped.Doc/Architecture/ControlsThatCannotFail— the family this belongs to.- OperatingFromThePortal.md — what
/healthanswers, and which replica answered it.