The package feed and its health

The store's catalog is a live feed, not a stored list. Every configured package source — a git repo of package manifests, another MeshWeaver instance's registry, or a node-repo checkout MOUNTED on this host's disk — is polled every 5 minutes, and the union of what they answer is what /Store shows and what @Store/data/StorePackage returns.

The degradation is deliberate

A poll pass can fail: GitHub can time out, a registry can answer 503. When that happens the feed keeps the last good snapshot for that source. That is correct — a store must not empty itself because one source blinked, and a source that recovers is picked up on the next pass with no intervention.

…but it used to be invisible, and that was the defect

Before #1180 the feed could not tell three different things apart, because all three produced the same "nothing to report" marker:

So on the public instance every git poll answered Octokit.AuthorizationException: Bad credentials, for the entire readable window, while @Store/data/StorePackage kept returning 132 packages and the store looked completely healthy. The catalog was frozen at whenever the credential last worked; a package published after that simply did not appear, and the only trace anywhere was a warn line every five minutes.

What the feed reports now

Each source carries a StoreFeedStatus row — published as a second virtual collection beside the packages, from the same poll passes (never a second poll loop):

The signal is PERSISTENCE, never one failure

A source becomes degraded after 3 consecutive failed passes (15 minutes). One failed poll is normal, and serving the last good snapshot is the right answer to it — a signal that fires on one bad minute is a signal people learn to ignore. The fault kind changes the wording and the log level; it never changes the threshold — with ONE exception, the Mount fault below, which is classified by its own exception type rather than guessed from text.

A permanent fault SAYS retrying will not help

Bad credentials, a 401, a non-rate-limited 403 and a 404 are permanent: every further pass fails identically, and the verdict says so in those words. Telling somebody to "retry shortly" there sends them round a loop that cannot close.

🚨 A rate-limited 403 is transient and is checked first. GitHub answers 403 both for "this identity may not read that" and for "you have asked too often" — and reading the second as a dead credential would send an operator to replace a credential that is perfectly fine, which is the one false positive that destroys trust in a signal like this.

A broken MOUNT is degraded on its first pass (Plugins#1563)

A mount whose path is missing or unreadable on this host is the Mount fault: permanent, and degraded on the first pass, because a path that is not on this disk is operator configuration that no later pass will find. It is deliberately not Missing — that remedy is about a remote source's URL or the polling identity's access, while this one is about the mount or the configured PluginCatalog:Sources:N:RepoPath. 🚨 The listing PROBES the path and fails: core's local directory reader answers a missing root with an empty snapshot and no exception, and the health fold reads "no exception" as success, so leaning on it would have turned a broken mount into a row reading ✅ live with zero packages — worse than no row at all.

Frozen is not the same as never loaded

If a pod restarted after the fault began, the source has no last-good list at all: its part of the catalog is missing, not stale. The verdict distinguishes the two, because "you are seeing an old list" and "you are seeing none of this source's packages" call for different reactions.

Where to read it

• /Store/Catalog/FeedStatus — a table of every source, its state, its last successful poll (rendered in your time zone) and its verdict. Global admins only: it names repositories, registry URLs and raw failure text.
• The store's own page — when a source is degraded, an admin opening /Store gets a banner saying the catalog is stale, on the page that is serving the stale list. It says WHY per kind and claims no history a source does not have: a polled source is named as having failed for at least DegradedAfter passes, while a broken mount — degraded on its first failed check — is named as unreadable on this host. The banner and the no-source page are chrome, in the viewer's language (StoreTexts, English and German).
• @Store/data/StoreFeedStatus — the same rows, for a health check.
• The portal log — a failing pass below the threshold stays a warn; a degraded source logs an error carrying the verdict, every pass, until it clears. Recovery is logged too.

🚨 Zero rows is three different answers, and only one of them is a wait

A StoreFeedStatus row exists per SOURCE, so an installation that reads no source produces no row — ever. The FeedStatus page used to render that as "No package source has reported yet", which is a waiting sentence for a permanent state: on a mount-only install it is indistinguishable from a first poll still in flight, for ever, on the one configuration that can never recover on its own — and it sat under the same page whose non-empty form shouts that the store is serving a stale catalog. The page therefore reads the effective source list (StoreManifestSource.Sources, the same list the feed polls) as a separate input from the rows, and says which of three things is true:

🚨 The count is what will be POLLED, never what is DECLARED. Both readers are optional and independent — a consumer instance registers no IGitHubRepoClient and reads from a registry, the registry instance reads git — and FetchFeeds drops a source whose reader is absent, so that source produces no SourceFeed and therefore no status row ever. Counting declarations would put a consumer instance with only git sources back on the waiting sentence permanently, which is the same defect one configuration over. StoreManifestSource.Readable is the ONE predicate, used by the feed and by the page (through its hub half, ReadableSources), so "what this deployment declares" and "what gets polled" cannot drift into two answers.

A mount-only install no longer lands here: since Plugins#1563 a configured mount IS a readable source (the next section), so it has a row of its own — healthy, or Mount-faulted when its path is broken.

Which sources are live — the supply-chain rule

StoreManifestSource.Resolve decides the EFFECTIVE source list from four inputs — the Store node, the deployment's configuration, its local mounts, and the sources the instance's own administrators added ON the instance — and it is a statement about the supply chain rather than about the Store node's content:

1. the node's declared packageSources, blanks dropped;
2. plus a registry source for every registry the deployment configures (PluginCatalog:Registries / RegistryUrl) that the node does not already name;
3. plus a git source for every remote git repository the deployment configures (PluginCatalog:Sources:N:RepoPath + :Ref — what a Deployment record's pluginRepos[] entry with isRegistrySource: true renders) that the node does not already name, however the URL is spelled (case, a trailing slash, .git); the node's own entry wins when both exist;
4. minus every git source — declared or configured alike — that has no credential here, when the installation has no deployment GitHub App (GitHub:App, RegistryPackages.HasPlatformGitIdentity) and another supply (a registry or a local mount). 🚨 The instance's OWN App does not lift this rule: it reads only the repositories its administrator installed it on, so handing it the platform's declared repositories would poll them into a permanent 404 and a stale banner (HasGitIdentity counts it; this rule asks the narrower question): the fetch would go out anonymous against a private repo and can only 404. 🚨 A repository whose configured source states its own token (PluginCatalog:Sources:N:TokenKey — what a record's pluginRepos[].secretName renders, the value delivered from the vault) is kept: its fetch is not anonymous and needs no App. Core's GitSourceCredentials reads it with that token and nothing else — on the feed's poll (StoreManifestSource.SourceTokens) and on the install's import (GitSync's ResolveAuth) alike. This is the client-instance shape: no platform App by design, its own private repository mounted beside the registry. A source naming an InstallationOwner instead stays dropped — that token is minted by the App, and GitSourceCredentials refuses it on an instance without one. A TokenKey whose value did not arrive keeps the source, and its status row names the refusal and the key. The same holds for a configured source declaring PluginCatalog:Sources:N:Identity = instance (core GitSourceCredential.InstanceApp): it is read with the instance's OWN App, never the deployment's, so it is kept — also while that App is not connected, when its fetch reports gitsync.source.instanceAppNotConnected on the status row instead of the source vanishing;
5. plus a mount source for every LOCAL path the deployment configures (PluginCatalog:Sources:N:RepoPath naming a path, as core's PackageSources.IsLocalCheckout decides), always — it is read off this host's disk and needs no identity;
6. plus a git source for every package source the instance's global administrators added on the instance (Administration › Package sources, Admin/PackageSource-*, StoreManifestSource.InstanceSources) that neither the node nor the deployment already names — always, whatever rule 4 says. An administrator added it on purpose; it is read with the instance's own GitHub App while one is connected (core GitSourceCredentials), else with the deployment's App or anonymously; and a fetch that cannot authenticate says so on its status row. It is live: an added or removed source reaches the feed and the page with no restart. 🚨 These come last, after the registries and the mounts: the list order is the precedence of a package id two sources both supply (the feed's union and the Store page's id dedup keep the first), and the declared supply wins (core InstanceRepositoryAccess, policy declared-not-derived).

🚨 Rule 6's sources are NOT in the Store's public data. The Store's virtual collections (@Store/data/StorePackage, @Store/data/StoreFeedStatus) are readable by every reader of the Store — Public and Anonymous included (CatalogGate.RootGrants) — and a virtual collection has no per-reader filter. An instance source is an administrator's private inventory (its repository URL, its packages), so StoreManifestSource.Sources — what those collections are built from — is rules 1–5 only, and rule 6's tail (StoreManifestSource.InstanceOnlySources) is polled by a separate, hub-scoped InstanceSourceFeed that is NOT a collection. Only layout areas read it, and each one composes it behind the viewer's live global-admin flag (InstanceSourceFeed.AdminOnly): an administrator's Store shows those packages' cards (after the public ones, so the declared supply still wins a duplicate id — and only once the public feed's FIRST PASS has answered and the page holds its union, StoreFeedProgress + InstanceSourceFeed.AfterPublicAnswered, because the public collection's empty seed is not an answer and an instance repo that answers first must not win the id) and Provision resolves them; the feed status page and the store's banner show their health rows. An anonymous visitor or a signed-in non-administrator gets nothing of them — not a card, not a row, not a URL. The packages stay out of the public catalog by design: they are this instance's choice, not a published offer. The admin gate answers for the VIEWER captured on the render turn (CatalogViewer: IsGlobalAdmin(userId), its home, its language, and every per-user query created under its context) — the page's composition re-runs on shared notifications that arrive with no context or under whoever published the change, and an ambient read there would answer for nobody, or for the publisher. The instance source list is an input of the cards' Provision resolution, never a trigger that rebuilds the page.

The instance's own list has a read state, and the health shows it. Core's InstancePackageSourceCatalog starts UNREAD and reports a query fault as Faulted with the last entries retained — never as an error. StoreManifestSource.InstanceSources carries both flags, and while the list is not read yet or its query is down, the instance feed's health (administrators only, above) carries one extra row (kind instance, StoreManifestSource.InstanceCatalogStatus): still being read, could not be read — none of them is polled yet, or could not be refreshed — the sources last read are still polled. It is a warning, never a degraded row (the query re-opens by itself), and it clears on the first complete listing. A fault-only notification restarts no poll loop: the polled list changes only when the sources do. Those rows arrive at once — the feed is seeded with each source's initial row (StoreManifestSource.Seeded), so a first poll pass still in flight never holds them back; the package cards take complete passes only. Each source's poll FOLD (last good packages and running health) is kept on the service across the gaps between administrators' visits (SourceFeedMemory), so a reconnect during an outage still serves the last good list and continues the failure count. The status page's headline says UNVERIFIED while any source has never completed a successful poll on this hub (after DEGRADED, before "polling normally"). Administration › Package sources shows the same three states above its grid and keeps the last-read rows.

Nothing decides "no App" before the App's stores have listed. A cold replica's catalogs publish their entries before they are loaded, so every identity question waits for readiness: the feed's default token goes through core's GetInstallationTokenOrAnonymous (it waits on WhenReady, so the first poll of a private repository is never anonymous by accident), and the Package sources page renders the App's status once InstanceGitHubApp.WhenReadable has answered OR a store reports its query down (core's metadata catalog signals no loaded on an early fault), and then says "cannot be read right now" — never "unreadable — delete it", never "not connected"; the Install button reads the same state. The Add form checks every input against core's rules before its first write, so a refused form changes nothing.

Nothing untrusted is ever Markdown. MeshWeaver Markdown runs --execute fences on first render, so the Package sources page renders every value it did not write itself — the GitHub callback's reason (anyone can send an administrator that link), a refusal echoing the typed input, a repository, the App's slug and owner — as literal text (PackageSourcesAdmin.TextLine, a Label), with the trusted way back to the tab as a separate view. The feed status page is Markdown, so every verdict cell is made inert (StoreFeedStatusArea.Cell: HTML encoded, link/emphasis/code openers escaped, line breaks flattened).

The mount reader (Plugins#1563)

Before rule 5 a mount could only subtract (rule 4): a mounted package that was not installed yet was offered by neither arm of the landing page, and a broken mount had no status row. The mount is now the feed's third reader, MountPackages (Store/Publishing):

• It lists EVERY package root in the mounted tree — each top-level {package}/index.json of an accepted root type, through the same NodeRepoPackageSource listing the registry serves. Design choice: the store offers what is available; what a deployment installs by default is the platform's separate decision (InstallByDefault), and folding it in would hide exactly the not-yet-installed packages this exists to offer.
• It reads only the listing's inputs — each package folder's index.json and manifest.lock — never the whole working tree, which on a developer checkout carries build output and is re-read on every poll.
• Its checksum is a CONTENT key (MountPackages.ContentKey, a hash over every card and install field of what it listed), never an mtime: a touched or re-checked-out tree with no content change never re-emits the catalog, and a real edit to a card does.
• A mount card provisions from the mount: its install request carries MountPath, and the Install from local mount phase reads the package folder off the disk and hands it to the platform's PackageInstaller, exactly as the registry phase does. A compiled module is not landed from a working tree (it carries no sealed bundle); the content and the install record are.

Configuration and the instance's own records survive; node content does not: an unclaimed Store node is re-imported from the shipped Store/index.json on the next Store update, so an edit there is undone. Rules 2 and 3 are what let a consumer instance work with no per-instance edit — and rule 3 is how a private node repo, mounted on ONE instance through that instance's record, gets its Store card there without ever being listed on the public registry (whose registration seed grants every source it configures).

Rule 6 is the same outcome without the record — and the one to prefer (policy control-knows-basic-config-only: "control should only know about the basic config of the instance, nothing more." — "even plugins list can be kept locally."). The source lives on the instance's Admin partition, which no Store update and no roll touches, and its global administrators add, remove and choose what installs by default themselves. The identity that reads a private repository is the instance's own GitHub App, created from the same page through GitHub's manifest flow and stored encrypted on the instance — "should probably set up github app for it." Mechanism and security review: core Doc/Architecture/InstanceRepositoryAccess. The feed logs the effective list once per change: [StoreManifestSource] N package source(s): … (git identity, configured registries, configured git sources, local mount).

What this does NOT do

It does not fix a broken credential, and it must not try to. Rotating the GitHub App installation token an instance polls with is an operator action on a running system.