Canary Per Major Line

Maintainer, 2026-11-22: "Let's factor out a dedicated instance per major version where we test changes, not in prod. Prod should be stable." And: servers must not restart for module updates; only boot-time infrastructure may require one.

Status: design of record, implemented in slices. Each slice below says whether it is landed or owed. The vocabulary is Release Channels: latest is derived, stable is promoted. This page is the who and the when of that promotion for deployments.

Why — what production did on 2026-11-22

Every portal followed the newest build of its line. Platform: every working record declared updatePolicy: Continuous on its line (4.*, 3.*). Modules: the registry's install-defaults pass and each install's Auto policy, plus autoRecycleOnStaleBuild. Nothing distinguished a portal people work in from one that exists to try things.

Measured over about 19 hours:

So production was the first place a build ran under real load. That is the property this page removes.

The rule

Every live major line N has exactly one CANARY deployment. The canary follows latest for the platform image and for every module. Every other deployment of line N follows stable for both. stable for line N advances only after the canary of line N ran it for the soak window with a green verdict.

Four properties follow from the rule and are pinned by tests:

  1. An unpromoted build never reaches a follower. A follower's roll is capped at the line's promoted build. If nothing is promoted, it holds.
  2. A red canary holds promotion. The cause is filed into bug triage, and stable stays where it was. A broken promoter degrades to "stable stops advancing", never to "nothing resolves".
  3. Lines are independent. The v3 canary promotes only v3, and the v4 canary only v4. A declaration for one line never reads or writes another.
  4. Promotion is monotonic. stable never moves backwards. A follower is never downgraded by it.

The pieces

 core CD ── arm ──► {N}-latest ──► canary of N (Continuous N.*, modules Auto)
                                        │  runs build T, soaks
                                        ▼
                     control: FleetWatch (its own pass) WRITES Ops/Status/<canary>, Ops/Issue/*
                                        ┆  data only: the promoter READS these samples
                                        ┆  on its own schedule; it is not a step of FleetWatch
                     LinePromotion (own self-clocking pass, every 5 min, per declared line)
                        verdict: Promoted T | Held (reasons) | CouldNotEvaluate (reasons)
                        writes Ops/Promotion/v{N}            (the pointer; one serialised writer, never backwards)
                               Ops/Promotion/v{N}/<T>        (the record, keyed by set, never mutated)
                                        │
                                        ▼
       follower announces T' ≥ T ──► SelfUpdateRouting caps the Roll at T, admits it unattended
       follower restart-pending  ──► held; its modules land live or ride the next promoted Roll

1. The line declaration — DECLARED, not derived

A line, its canary and its followers are committed content: one Hosting/ReleaseLine node per line beside the deployment records (Deployments/Lines/v4, …). This follows policy declared-not-derived: an absent contributor is never a removal, and a line nobody declared has no canary and promotes nothing.

Field Meaning
line The major, e.g. "4". Must equal the canary's updatePattern line (4.*).
canary The ONE deployment id that follows latest for this line.
followers The deployment ids that follow stable. Never the canary, never a record of another line.
soakMinutes How long the canary must run a build before it promotes. Default 240 (4 h).
issueBudget How many non-holding issues of the canary may be open in the soak window. Default 0.
enabled false keeps the declaration but promotes nothing, and followers then hold.

The deployment records themselves keep updatePolicy: Continuous and their line pattern, so the Memex no-pins gate is untouched. A follower is not pinned. The declaration decides WHEN it takes a build, never WHICH build exists (the same contract as rollGate). The resolved tag is recorded on the control side (Ops/Promotion) and never written back onto a record.

2. The canary

The canary is a working portal, not a test fixture. It exercises what production exercises:

Choice for the current major: the staff development instance. It is already Continuous on 4.*, runs three replicas, installs every package, and carries no fleet role and no client data. Reusing it costs nothing. Its user traffic is light: one user was indexed when this was measured on 2026-11-22. So its verdict rests on what every boot exercises, as FleetWatch samples it: the roll, the NodeType bake, module adoption and the health checks. It does not rest on load. A synthetic smoke over the canary's own Tests areas is OWED, so that the "real use" bullet above is met by more than a handful of people. A new idle instance would cost a namespace, a database and secrets and would measure no more.

Choice for the previous major: a dedicated small canary instance is OWED. Only one customer portal remains on the previous major, and no non-production instance runs that line. Until the canary is provisioned, the previous line is not declared at all, so that portal keeps following latest. Declaring it a follower of a line that has no canary would freeze it with no way to receive a fix. Provisioning goes through a governed Provision Hosting/InstanceAction. The smallest envelope still keeps three replicas, because the records repository's two-legs rule holds every serving instance to a floor of three.

3. Promotion — computed once, on the control instance

LinePromotion runs on the control instance's resident platform-build inbox hub. It is a self-clocking pass at FleetWatch's own cadence (5 min), the cadence that writes the Ops/Status samples it reads. Each pass evaluates every declared line. One activation per address and a schedule that starts the next pass only after the last one finished make it a single, serialised writer.

The platform evidence. Candidate T is promoted when every item below holds. The readings are the same ones RollGates already trusts, and each unanswered reading is a reason to WAIT, named:

  1. The canary's Ops/Status/<canary> sample is current (not stale, scraped), converged, on one generation, and its running tag T is on line N.
  2. The generation has run for at least soakMinutes: the oldest pod start on T. A same-image restart makes the pods younger, so a module wave on the canary restarts the soak. That is the safe direction, and it is how a module wave is soaked as well.
  3. No StuckRoll and no Critical Ops/Issue of the canary is open, or was resolved inside the soak window.
  4. At most issueBudget other open issues of the canary. These come from the FleetWatch kinds: RestartSpike, ReplicaShortfall, the declared health-check alarms and the roll-log signatures.
  5. T is strictly newer than the current stable of the line (BuildOrder), and T is orderable against it. Two tags of unknown order never promote.

The verdict is a verdict, not a tally (Release Channels):

Verdict Means Effect
Promoted T qualified, the pointer moved Ops/Promotion/v{N} names T; Ops/Promotion/v{N}/<T> records the evidence
Held Evaluated, and T did not qualify (soaking, red, not newer) Pointer unchanged; reasons recorded on the pointer node
CouldNotEvaluate A reading did not answer (no sample, query failed, page full) Pointer unchanged; RED, reasons recorded

A red canary files into triage. A Held verdict caused by a holding issue does not open a second issue for the same fact, because the canary's own Ops/Issue is already the triage item and FleetWatch notifies it. A CouldNotEvaluate that persists beyond three passes means the promoter is blind. It is raised as a Hosting/Issue of kind PromotionBlind (Critical), because "I did not check" must be as loud as a fault.

Monotonic by the writer. The pointer write is a compare-and-swap on the node's version and refuses any tag that is not strictly newer than the one it read. The per-set record is keyed by the tag, so a retry rewrites the same entry. Entries are never mutated after creation.

4. Followers take stable — as one wave

On the control instance, SelfUpdateRouting routes every self-update announcement. For a record that a line declaration lists as a follower:

At most one restart per promoted set follows: a follower restarts when, and only when, stable advances past what it runs. MinRollInterval stays as the floor.

5. Modules follow stable — OWED, and where it goes

The control-side hold in §4 stops module waves from restarting a follower. It does not yet stop a follower from installing a module generation that the canary has not soaked. That gap matters only for modules that swap live, and it closes on the instance side:

6. The control instance

The control instance is not a follower in the first slice. Core policies platform-deploy-control-first and control-always-latest make it the first deployment of every accepted platform build. CD's control-acceptance proves it before anything else sees the build. Making it follow the canary would invert those two policies, so that is a maintainer decision, recorded as owed. Its module-wave restarts are still in scope for §5, under the same instance-side channel.

What this is NOT

Slices and state

# Slice Where State
0 This page core Doc/Architecture/CanaryPerMajorLine landed with this page
1 Hosting/ReleaseLine declaration type + LinePromotion promoter + follower routing + tests MeshWeaver.Plugins Hosting/ Systemorph/MeshWeaver.Plugins#5782
2 Deployments/Lines/v4 (canary: the staff development instance; followers: the working portals) (the previous line is declared with its canary) the deployment-records repository, mesh/Deployments/Lines/ after slice 1 is live on the control instance
3 The previous major's canary: record + governed Provision records repository + control instance owed: cost and sign-in decision
4 Instance-side module channel (§5) core MeshWeaver.PluginCatalog + registry owed
5 The control instance follows the canary (§6) core CD policies owed: maintainer decision

See also