Which portal an incident fired on
Systemorph/MeshWeaver#7334.
An incident is one node per fingerprint (Admin/_LogIncident/{fingerprint}), and the fingerprint
deliberately leaves the namespace and the pod out — the same defect on two portals is one ticket.
So a single incident folds bursts from every deployment the fault fires on.
What was measured
Read on the control instance: one pod name was attributed to memex-cloud on one incident and to
memex on two others. The incident carried
content.namespace— ONE value, set from the first burst and never moved, andcontent.pods[]— every pod the fingerprint was ever seen on, in EVERY namespace.
Both namespaces run a deployment named memex-portal-deployment, so only the pair namespace + pod
identifies a replica, and the incident had thrown one half of that pair away. Deciding whether an
occurrence was before or after a fix rolled needs exactly that half: rolls are staggered, so the
same wall-clock instant is post-roll on one portal and pre-roll on the other.
What changed
The watcher queries Loki one namespace at a time, so every report names exactly ONE namespace and
all of its pods belong to it. The ingest now keeps that pairing in content.namespaces[], one row
per namespace:
| field | meaning |
|---|---|
namespace |
the Kubernetes namespace — one deployment |
pods |
the pods of THIS namespace the fingerprint fired on |
firstSeen / lastSeen |
the first and last occurrence in THIS namespace |
occurrences |
the lines folded in from THIS namespace |
A first sighting starts the ledger; a recurrence adds to its namespace's row, or adds a row; the corpus migration unites the ledgers of a legacy incident and its successor. A retried delivery is recognised before the fold, so it is not counted twice in its row either.
The filed issue prints one row per namespace instead of the old Namespace + Pods pair, and the
recurrence comment says how often and how recently each namespace fired.
Reading an incident
- "Did it fire on portal A after A's roll?" — read A's row: its
lastSeenagainst A's roll time. The incident-widelastSeenanswers only "did it fire anywhere". content.namespaceis still first-seen. It is kept so nothing that reads it breaks; it says where the incident was OPENED, never where it fires.- An incident older than the ledger has history the ledger never saw. Its pods that no row claims
are printed apart as namespace not recorded; resolve those by hand (a governed
Logsaction per namespace) before naming a portal for them.