Declared Is Not Landed

An install record is a statement about the source. It names the files a bundle shipped and the hash of each. Nothing in it observes the mesh, and nothing in the installer's own counters does either — InstallResult(Total, Written) counts what the installer decided to write.

That gap has been closed one exit at a time, and this page is about the exit that was still open.

The three exits an install can take

CatalogLayoutAreas.InstallOrUpdate chooses among three, on two questions: does the record's ModuleVersion equal the candidate's, and does the record carry a file map?

The incremental exit's fetch set was a comparison of two declarations — the candidate's manifest.lock against the record the installer itself wrote. A node lost after a previous install has an unchanged file hash, so it never entered delta.AddedOrChangedFiles, was never fetched, and never reached DecideAndWrite — the one place that would have restored it. The presence-awareness sat one layer below a set the absent file could not enter.

🚨 And it is the exit an update always takes

This is what turns a one-off loss into a permanent one. #3485's heal lives on the module-hash-EQUAL exit. An update never takes that exit by definition, and every new publication moves the hash again. So a package that loses a node stays short across arbitrarily many updates, each one reporting success, and the only escape is a boot on which the source's hash happens to equal the record's and the install lane happens to run.

The measured case

Plugins/Hosting on memex.meshweaver.cloud, read 2026-09-14T06:0xZ. All reads through get / search; nothing was mutated.

The morning: the files were there and they compiled. The release node Hosting/TriageStatus/Release/20260913081235-bylQeIj_ — created on that portal at 2026-09-13T08:12:35.446Z, status: Succeeded — names its inputs:

sourceVersions:
  Hosting/Deployment/Source/TriageIntake                 639248839464110360   (2026-09-13T08:12:26.411Z)
  Hosting/TriageStatus/Source/TriageStatusContent        639248839461605390   (…08:12:26.160Z)
  Hosting/TriageStatus/Source/TriageStatusLayoutAreas    639248839464108660   (…08:12:26.410Z)
testVersions:
  Hosting/TriageStatus/Test/TriageStatusTests            …
  Hosting/TriageStatus/Test/TriageStatusTestsArea        …

A release records the source nodes a successful compile actually read. So those five nodes existed on this portal, nine seconds before the compile that consumed them.

The evening: the update ran, and wrote only what had changed. The record was stamped at 2026-09-13T22:03:03.221Z — installedNodeCount: 224, version 1.19.16. Two nodes carry write timestamps from that install:

Everything else in Hosting/Deployment/Source still carries its older timestamp — 08:09:41, 09-12, 09-09, 09-08, 08-28, 08-19. The update wrote exactly the two files whose content had moved.

And eleven declared files had no node at all. Anchored reads, each with a same-anchor positive control:

All eleven are named in the record's own installedFiles map. The record asserts them; the mesh does not hold them; the update that stamped the record did not fetch a single one.

🚨 The sweep hazard: 8 of 19 understates it, and the 9th reads green

Hosting/TriageItem is not in the Error set. Its own node says:

compilationStatus:        Ok
lastCompileSucceededAt:   2026-09-13T08:12:35.5789283Z
lastCompiledVersion:      6
requestedReleaseAt:       2026-09-13T21:56:05.9665591Z
lastReleaseRequestHandledAt: 2026-09-13T21:56:05.9665591Z     ← handled without a recompile
lastCompileStartedAt:     2026-09-13T08:12:32.047279Z          ← still the morning's
compiledSources:          6 entries — TriageActions, TriageItemContent, TriageItemLayoutAreas,
                          Hosting/Deployment/Source/TriageIntake, and two Test nodes
currentSourceVersions:    {}
currentSourceFingerprint: e3b0c44298fc1c14                     ← SHA-256 of the EMPTY STRING

Every one of those six compiledSources is now absent from the mesh, and the type serves a cached assembly at compilationStatus: Ok.

So content.compilationStatus:Error is a sweep for types that have attempted a compile and failed. A type whose sources all vanished but which has not been re-driven since is in the GREEN population. The honest statement of this portal's damage is not "8 of 19 are broken" but "8 of 19 report Error, and at least one more has lost every source it compiled from while reporting Ok".

This joins the list of reasons a compilationStatus sweep's number is not a census — alongside the RLS filtering and the two-executor split already recorded in NodeType Compilation. The cross-check that does see it is the record-vs-mesh comparison on this page: compiledSources (or the release node's sourceVersions) against what the mesh holds.

The fix

IncrementalUpdate now asks the mesh before it fetches.

1. Derive the candidate manifest's declared node paths — newManifest.Files.Keys through PackageInstaller.NodePathForFile, the installer's own file→node rule and not a second implementation of part of it.
2. InstallCompleteness.ObservePresent — one batched IStorageAdapter.ReadMany over that bounded, known set. Never a query (a stale negative would re-fetch a file that is present) and never N point reads of possibly-absent paths (which is what opens a storm breaker on the owning hub).
3. InstallCompleteness.FilesToRestore — pure, so every arm is pinnable offline — returns the declared files the delta is not already fetching whose node is absent.
4. The fetch asks for delta.AddedOrChangedFiles ∪ restore. DecideAndWrite then writes them, because current is null.

It costs one batched read per incremental update — the same read Install Completeness's post-install verification already pays on this exit, moved to where it can still change the outcome instead of only reporting it.

🚨 Three answers, never two

ObservePresent returns null when the mesh was not read — no storage adapter, or a batched read that faulted — and never an empty set. An empty set would say "the mesh holds none of them", which on a Postgres read that faulted (a satellite table that does not exist answers 42P01, not "absent") would make an install re-fetch every file it ships, every time. FilesToRestore maps that null to an empty restore set, and the caller says so at Warning: an unobserved mesh is not a clean one, and the line is spelled so it cannot be read as a pass.

The other silent hole in the same method

IPackageSource.FetchPackageFiles(package, gitRef, paths) filters — locally in the interface default, server-side in RegistryPackageSource — and its contract is explicit that "paths absent from the package simply don't appear in the result". A requested path the source does not serve (a serving-side casing difference, a stale registry cache, a bundle that disagrees with its own lock) therefore vanished with no exception and no log line, after which the record was stamped with the full declared map anyway. The returned set is now compared against the requested one, the shortfall is named at Error, and the incremental update fails into a full install rather than stamping that record. (A first reading — "report only; the install that did run is a fact" — did not survive the next step: InstallNodeRepoDelta stamps the new map as the baseline unconditionally, so a file that never travelled while its OLD node survived would read as complete and diff clean on every later update.)

🚨 …and the lock's module sources were exactly such paths (MeshWeaver#4429)

A mixed package's manifest.lock declares its module's src/<Module>/… sources — and the in-tree siblings that ride its bundle — in the SAME files map as its node files, so that a source-only commit moves the module version (Plugins#878). No content source serves them: NodeRepoPackageSource keeps only {Id}/…, the registry's /api/plugins/files answers from it, and the sources travel compiled, in the module bundle. The delta nevertheless asked for every changed key. Before the shortfall guard a changed source therefore quietly failed to arrive — harmless, because the bundle carried it; after it, every such update answered "the source returned 0 of the 1 file(s) asked for" at Error and fell back to a full install that recompiles every type in the package. Measured on memex.systemorph.com at 2026-09-15T14:12Z for Edu (src/MeshWeaver.Courses/CourseAssetService.cs), on two pods. Edu/manifest.lock carries 273 src/… entries beside 117 Edu/… ones, 234 of them the AI engine's, so an engine commit reached Edu the same way.

The delta's fetch set now drops them with PackageInstaller.IsModuleSourcePath — the predicate the node mapping, the delta prune and the completeness sweep already share (#4101) — while the record still stamps them, so the next diff is clean. The guard is unchanged: whatever the fetch still asks for is a file the source should have served. Pinned by InstallCompletenessTest.AModuleSourceChange_UpdatesIncrementally_WithoutAskingTheContentSourceForIt, which drives the production NodeRepoPackageSource through a source-only update and a content-plus-source one.

The rule this generalises to

A decision about what the mesh needs may not be taken from a record the writer wrote. Any comparison whose two sides are both declarations — a lock against a record, a hash against a hash, a count against a count — can only tell you what the source did. If the decision is "does this have to be written", exactly one of the two sides must be a read of the mesh.

The same sentence is why #3485 exists, why the post-install verification exists, and why this page does. Each of the three closed it on one exit; none of them closed it as a principle, which is how the third exit stayed open through two fixes aimed at the same defect.

What this does NOT do

• It does not explain why the nodes went missing — that was found the same day, and it is the other half of the fix. The loss was Hosting/_GitSync re-importing the partition at the commit SEALED for the running framework (627fb3cd, a 2026-09-12 tree) at 21:56:05Z, on the boot after the default install had put a 2026-09-13 tree there from Plugins@main: the bake declined the Hosting bundles on their source fingerprint, SealedSyncReconcile read that — correctly — as "the live sources have drifted from the commit they claim", and the re-import pruned the eleven nodes only the newer tree has. Two unattended writers of one partition, two commit policies. The restore on this page alone would have made that an every-boot flap; the boot install now lands on the sealed commit too, so the writers agree. Timeline, evidence and the rule: The Sync-Ref Contract → "The lane the closure missed".
• It does not re-arm a PARKED NodeType. A type that failed a compile serves its cache and does not re-drive itself when sources arrive; that is MeshWeaver#4208's subject. Restoring the content and recycling the type are two steps, and on an image without #4208 they stay two steps.

See also

• Install Completeness — the five verdicts, only one of which is a pass, and the post-install read-back that reports the outcome on every writing exit
• NodeType Compilation — what a compile reads, what a release records, and the other reasons a compilationStatus sweep is not a census
• CQRS and Content Access — why the presence read here is a batched store read and never a query