A Sample that drops checks silently

Sample — the Hosting/InstanceAction an operator issues instead of kubectl get pods plus a curl — reads every replica's own /health and stores it on Ops/Status/<id> as replicas[].healthDetail. It is the per-replica instrument the platform's own guidance points at: "the per-replica form with no guesswork is a Sample InstanceAction".

It could not carry its own payload, and it did not say so.

The mechanism

ObservationQueries.HealthDetail flattened the body's newlines into spaces and cut the result once:

var one = body!.Replace("\r", " ").Replace("\n", " ").Trim();
return one.Length <= max ? one : one[..max] + "…";     // max = 2000

A portal's /health body is one line per printing check:

Degraded
timing: 308ms total over 16 check(s), slowest first — …
content-types: Degraded — 9 node type(s) whose content this replica cannot type — …
bake-report: Degraded — compiling sweep: framework=se055234 total=366 baked=242 … LIVE RECORD CENSUS: …
source-discovery: Healthy — 3 discovery query/queries measured on this replica …
publication-seal: Degraded — framework identity se055234… holds 8 publication(s) …
fleet_watch: Healthy — the fleet watch last wrote pearl at …
pending_module_activation: Degraded — 3 module(s) are landed but not yet loaded in this process …

So flattening destroys the only structure the body has, and one cut over the concatenation spends the whole budget on whatever happens to come first. What is lost is not a tail of one sentence: it is every check after the cut, entirely, with nothing naming them.

🚨 And the trace it leaves cannot be read. A trailing … says "something was cut" and says nothing about what — so a check that fell past the cut and a check the replica never reported are the same reading. That is the failure mode of a gate that cannot fail, one layer down: the instrument answers, confidently, about a population it does not name.

The measurement

Both portals, read-only, 2026-09-22 — by calling the public /health directly (the untruncated form) and applying the stored reading's own arithmetic:

memex.meshweaver.cloud memex.systemorph.com
body 8,872 bytes, 16 checks, 7 printing 6,672 bytes, 17 checks, 8 printing
flattened length 8,763 6,574
survive the 2000 cut timing, content-types, bake-report (cut mid-sentence) the same three
lost whole source-discovery, publication-seal, fleet_watch, pending_module_activation source-discovery, publication-seal, fleet_watch, required_modules, bundle_adoption

Every name in bold is a deployment or self-update verdict, and each was Degraded at the moment it was dropped:

🚨 And bake-report survived only as far as its first sentence. Its LIVE RECORD CENSUS — added precisely because the other bake readings are boot-time and this one is not — fell past the cut. Of the three signals the platform names as independently per-replica (content-types, bake-report's Baked/BytesMissing split, and the LIVE RECORD CENSUS), Sample could carry one, and only because content-types happened to be line 3.

Why the bound was wrong rather than merely small

The 2000 was chosen when Sample was introduced, and every check an operator now reads a replica for was added afterwards — bake-report and its two censuses, source-discovery, publication-seal, required_modules, bundle_adoption. It was never sized against a payload; the payload grew into it, silently, one health check at a time.

That makes the budget a sizing decision, and it is deliberately not the fix. A bigger number would be the same defect with a later onset — the next check added lands past the new cut and is dropped just as quietly.

The fix: whole checks, bounded separately, and the omission NAMED

HealthDetail now reads the lines as the entries they are:

  1. A leading bare verdict line is dropped — ReplicaStatus.Health already carries the verdict, and every byte of the budget belongs to a check.

  2. Each check is bounded on its own (HealthEntryMax), so one verbose check cannot evict the checks after it. A check cut to its own cap still ends in …, so a truncated sentence is never read as a complete one.

  3. Whole checks are kept, rendered in the portal's own order, which is what makes two replicas' readings comparable line for line.

  4. 🚨 Whatever did not fit is listed BY CHECK NAME, with the count and the budget, and the sentence says which fact it is about:

    ⚠ 4 of 7 check(s) OMITTED from this reading (budget 12000), not absent from the replica: source-discovery, publication-seal, fleet_watch, pending_module_activation; of 3 carrying a finding, 3 kept

Point 4 is the correctness property and it holds at any budget. The budget is now a tuning value; the instrument is honest whatever it is set to.

And the bound is spent on the FINDINGS first

Naming an omitted check says which; it cannot say why. The same defect was measured from the other end on nodetype_bake — the check that actually refuses a replica's readiness, and on a live body the one that came last, after timing + content-types + bake-report had spent the whole bound (MeshWeaver#5011). Under a pure portal-order rule a budget overrun would still drop the one entry that carries the reason, and name it. So the two halves of this fix compose:

  1. Selection is findings first — the entries whose verdict is Degraded or Unhealthy, in the portal's order, then the rest in the portal's order. A Healthy entry never displaces a finding. Once one entry is refused every later one in that order is too, so the kept set is a prefix of a principled order and never "whichever happened to be short enough". What is kept is then rendered in the portal's order (point 3), so the selection rule changes which lines a squeezed reading carries and never the order two replicas' lines are compared in.
  2. A finding longer than the room left is CUT to fit, never dropped — a cut finding still names the check and the start of its reason, which is the part that routes it. Its … says it was cut, and the note counts it (1 cut to fit).
  3. 🚨 The note is paid for OUT of the bound, not added on top of it. The bound is the caller's contract; a note appended afterwards would put the stored string over it, which is how a bound quietly stops being one. The room reserved is the note's worst case — every check omitted and named — so the names always fit at any bound that can carry a reading at all; a bound too small even for that gets the counts, and one too small for the counts gets a trailing …. At the measured default nothing is ever squeezed; these rules are what the reading does when a future check outgrows it.
  4. A body that arrives with no newlines — a proxy's rewrite, an older stored reading — is split on the name: Verdict boundary instead (ObservationQueries.HealthEntries, anchored on the verdict word because a real detail is prose full of colons). Prose with no entries at all keeps its head, which for an exception message or an HTML error page is its most informative part.

Measured on both live bodies at the old bound of 2000 (memex.meshweaver.cloud, 7 printing checks): content-types whole and bake-report cut to fit are carried — the two findings that come first — and timing, source-discovery, publication-seal, fleet_watch, pending_module_activation are named as omitted, in a reading of 1,972 characters. Under a portal-order rule the same bound carried timing and content-types and named the rest.

What this does and does not close

The tickets this reading was blocking

See also