A log query that cannot say it failed

The Logs instance action is the only sanctioned route to log-line evidence. An operator or an agent asks for a window, the control instance reads Loki from inside the cluster, and the lines land as Hosting/LogEntry nodes under {space}/Logs. When that works there is nothing to discuss. This page is about what a reader could conclude when it did not.

The defect, stated as a fact about the record

logQl and entryCount were both stamped by the LANDING phase, on its success arm alone. So an action that read its window and then could not write it — a write to a quiescing hub, a CreateOrUpdateNodeRequest that spent its 60 s budget — left both fields null. That is byte-for-byte the reading of an action that never got as far as querying:

A reader with no answer on the action node does the natural thing and opens {space}/Logs. 🚨 That folder is the output of one PAST query, never a feed. A LogEntry id is deterministic in (deployment, timestamp, pod) — deliberately, so two overlapping windows update rather than duplicate a line — which means an overlapping query also OVERWRITES each row's selector with its own. The folder therefore cannot say which run landed what it holds, and a reader who has just issued a query sees rows a different run wrote. Four failed log queries on 2026-09-21 read as four empty windows through exactly that path.

What changed

Each fact is stamped by the phase that knows it.

• "Query the logs" stamps logQl, the new linesRead, and truncated the moment the backend answers — before a single line is landed. Its log line says so in as many words: it stands whether or not the landing succeeds.
• "Record entries" stamps entryCount (landed) and the new entriesFailed, and it stamps them before it fails. Each write reports its own outcome, so a landing that partly succeeded is recorded as partial, naming the count and the first fault, instead of blaming the whole window for one line. The phase still errors afterwards: fail-fast is intact, the remaining phases are never subscribed, and the action still goes Failed.
• Every landed row names the LAST run that landed it — actionPath and landedAt on LogEntryContent. 🚨 Never "the run that landed it": the same overwrite that replaces a row's selector replaces these too, and ingest-logs upserts the same ids carrying neither, so an overlapping ingest clears them. They are strictly better than a row that says nothing about its provenance, and they are not identity — the authoritative record of what a run asked and landed is the run's own action node. Two follow-ups, named rather than implied: ingest-logs should stamp its own attribution, and an action-scoped view is what would make the fields usable in a UI. Both are optional parameters, because an in-mesh signature change is invisible to dotnet build.
• A new run clears the previous run's answer in the same patch that records its ask. InstanceActionContent.Start preserves the log fields, so a re-run over a node that already carried entryCount: 190 rendered 190 landed of 10 read — two runs' numbers in one sentence, which is the same stale-answer failure one level up.
• Three terminal states per write, not two. A row write that completes EMPTY — the owner produced neither a node nor a fault — used to contribute no entry at all, so the landed count silently under-reported and entryCount + entriesFailed stopped adding up to linesRead with nothing saying why (MeshWeaver#2454's third state). It is now a named fault. And a COOPERATIVE cancellation is re-thrown rather than counted as a failed row: a drained pool is not a landing that partly failed.
• The aggregation is a function the tests execute (Land). Every other case in this change asserts a string or a predicate; the part that decides the counts a reader acts on is covered by cases that feed it acknowledged, faulted, empty-completing and cancelled writes.
• The record page renders landed of read, with an explicit warning for a window that was read and not landed, and another for a partial landing.

The three readings are now three readings:

A task was canceled was a third unreadable answer

A wide window over a high-volume namespace returned A task was canceled at ~45 s, while the identical query over a lower-volume namespace completed in ~22 s. That sentence is what HttpClient.Timeout raises, and it names no target, no budget and no remedy — and it is indistinguishable from a host shutting down.

🚨 Measured on the instrument itself, 2026-09-21T21:48:21Z. A Logs action against namespace memex — the LOW-volume one, the control that was supposed to complete in ~22 s — started its read at 21:47:36Z and failed at 21:48:21Z, exactly 45 s later, recording:

error: "A task was canceled."
logQl: null   linesRead: null   entryCount: null

That is the whole of what the run said: not the selector, not the window, not the limit, and nothing a reader could narrow. So the ask is now stamped BEFORE the request leaves — it is a pure function of the request, known before anything is sent — because a read that never answers is exactly the case no later stamp can cover, and it is the case where the ask matters most. The fault's own message cannot carry it either: the URL is redacted to host + path (Plugins#1734), which is where the query string lives.

🚨 And narrowing IS an effective remedy — measured as a one-variable control, because advice nobody tested is the same defect as a message nobody can act on. Same namespace, same selector, same limit; only the window changed:

So the knob the message names works. It is still offered as the cheapest next STEP and not as a diagnosis: one pair is consistent with scan volume and does not isolate it from backend load at the two instants, which is exactly why the wording establishes the budget and no cause. The second row is also the pre-fix SUCCESS path, and it shows the gap is only ever on the failure path — logQl and entryCount are both recorded here.

🚨 The cure is naming, not a bigger bound — and the naming must not overstate itself. A spent budget now names the target, the budget it spent, and the knobs that narrow the question, and it deliberately names NO cause: query volume, backend load, connectivity and a response that started and stalled all spend a budget the same way, and the measurement above — the low-volume namespace timing out — is itself why "the backend is scanning too much" cannot be asserted from a timeout. Narrowing sinceMinutes or pod is the cheapest next step, not a diagnosis. Raising the budget is neither: a question the backend cannot answer inside its budget is a question asked too widely, and a bigger bound hides that from the only person who can narrow it.

🚨 Only the timeout impostor is converted. A COOPERATIVE cancellation propagates untouched: every classifier downstream reads the type and the cause to tell routine teardown from a failure, and wrapping a drained pool in a fault is the same defect as reporting a terminal over work that never happened, in the opposite direction. CancellationClassifier is the one implementation of that distinction, and ObservationQueries.BudgetSpent is the predicate the catch filter uses — exposed so both sides of it are asserted without a network.

What this does NOT fix

🚨 An action that never lands at all is a different defect, and it is in the shared create chain, not here. Measured on the control instance 2026-09-21 at 20:15:49Z: creating the Logs action node answered No response received … within 00:01:00 for CreateNodeRequest, with the delivery trail

AWAITING → POSTED → RECEIVED → ENQUEUED → QUEUED depth=1 → ROUTED onTarget=True
→ HANDLER_ENTER(+2ms) → CREATE_CHAIN_SUBSCRIBED(+2ms) → HANDLER_EXIT state=Processed(+2ms)

and then nothing. A search and a point get 78 s later both said it was absent, so it genuinely did not land.

🚨 Do not read that as a reproduction, and do not read it as new. That exact trail — HANDLER_ENTER → CREATE_CHAIN_SUBSCRIBED → HANDLER_EXIT Processed, nothing after, Not found afterwards — is an ALREADY-FILED shape with a measured, CONTENT-DEPENDENT trigger: a Feedback/Feedback create whose message carries an @path reference hung 5 of 5 times on this same instance earlier the same day, and the sixth attempt, identical but for the @s removed, was created at once (rbuergi/Feedback/20260921-feedback-create-hangs-on-an-at-reference-in-the-message). That filing states the discriminator this page must respect: 1 of N failing is noise; N of N with one text is signal. The action above carried no @-reference, and a near-identical create on the same path succeeded immediately ~90 minutes later — so it is 1 of 2, which is noise by that rule, and the honest reading is one occurrence of a known shape, not a second defect. Anyone matching a Feedback-create hang to this should check the @-reference trigger FIRST, because that one is measured and this one is not.

What the single occurrence does still establish, because each reading kills a hypothesis:

• The cold-observable explanation is refuted for this path. CREATE_CHAIN_SUBSCRIBED exists precisely to prove the chain was subscribed, and it was. All three terminal arms are present in that handler, the completed-empty backstop included, so what is left is a hang inside one of its upstream leaves — the existence read, the partition bootstrap, a validator, the NodeType probe, or the save — and the trail carries no stage between subscription and terminal to name which.
• The MeshWeaver.Mesh.RequireSubscribe channel cannot answer that question at all. RequireSubscribeObservable wraps only MeshNodeStreamHandle.Update/Overwrite/AdoptPersisted. The Hosting/LogEntry write goes through IMeshService.CreateOrUpdateNode, a bare Observable.Defer(...).CarryAccessContext(...). A silent channel there is a zero with no denominator, never evidence that a write was subscribed.

The reading was taken while the control instance was mid-roll, which was not visible from the instrument, and it did not recur once the roll converged — so it is evidence about the SHAPE of the failure and about which explanations are excluded, never about its cause.

A red that asserted something true — the no-skip-trapdoor rule, on live bytes

This page's own pull request produced the cleanest live vindication of the no-skip-trapdoor rule the author has seen, and it belongs beside the rule rather than in a chat log.

A runner was lost mid-job: Module bundles / Warm the shared build environment (once) began Publish the module-pack tool and never finished it, was killed ~10 minutes into a 45-minute cap with every later step and every Post-step at conclusion: null, uploaded no log at all (/logs answers BlobNotFound), and carried its reason in one place only — the check-run annotation, verbatim: "The self-hosted runner lost communication with the server." Two required contexts went red behind it, and both said something true: Module bundles / All selected bundles built failed on "Exactly the selected bundles were built", because nothing was built; and The Tests-area gate's inputs are present failed on "Every need this gate builds on must have succeeded" — a preflight asserting its inputs instead of asking whether they happen to be there.

🚨 Had either been written to skip on a missing input, this runner death would have painted the wall GREEN, and a required context would have read satisfied over a job that produced nothing. That is the rule's benefit measured rather than argued, and it is the exact mirror of the defect this page is about: there, an instrument that could not say it had failed; here, two gates that refused to inherit another job's silence. The same run also shows where such a death is meant to be decided — the annotation, not the log, because the log does not exist — which is what lets one automatic retry be armed for it and a second, by hand, be refused.

Also left alone, deliberately: instance actions stuck in Running (five at the time of writing, two for days). That may be the same never-terminates shape or bookkeeping nothing closes; there is no evidence yet which, and guessing would put a wrong cause in front of whoever picks it up.