The Tests-area verdict must carry a machine-readable pass count

Audit finding, 2026-09-17.

What the gate actually reads

The catalog gate renders each NodeType's Tests layout area on a real mesh and classifies the frame in core's tools/MeshWeaver.PluginTester/AreaProbe.cs (ClassifyTestsFrame), in this order:

1. a framework error control ⇒ Failed;
2. any string carrying ❌ ⇒ Failed, with the failing rows as the detail;
3. the FIRST string matching (\d+)\s*/\s*(\d+)\s+passed ⇒ Passed N/M passed when the two numbers agree, Failed only N/M passed when they do not;
4. otherwise, any string carrying ✅ ⇒ Passed with the detail all rendered cases green.

Step 4 is a deliberate fallback, and it is the hole: it is a PASS THAT CARRIES NO COUNT. The gate log then reads

    ok  Chess/History: compile=Ok render=ok tests=ok
        Tests host: Chess/History/GateProbe — the probe instance the gate created for this check
        all rendered cases green

which is the same line a suite of one trivial case would print. Nothing in CI could tell the two apart, so a suite that silently shrank — a renamed runner, a case list that stopped being iterated, a Test/ folder that lost its cases — would keep the wall of ticks green.

The ten suites, and why each one missed

Measured from the gate logs of two real main runs, not from a grep: MeshWeaver.Plugins run 35199866577 (gate shards 1–4, commit d98fc2ac) and MeshWeaver.Education run 35205347144 (gate shard 1/1, commit f1d37ae8). Exactly ten tests=ok lines in the family carried all rendered cases green; every other suite already matched.

That is ~120 executed cases whose count CI was throwing away: 72 in Chess, 34 in RemoteControl/Screen, 10 in Edu/LearningJourney and 58 across the five German suites.

Two things the table settles that a reading of the regex alone does not: bold markers do not matter (the match is a search, not an anchor — **17/17 passed.** matches), and the German suites' sibling AgenticPrimerDe/DiceGame was never affected because it says passed in English.

The rule

Every Tests area renders a bare N/M passed token, with nothing between the number and the word, and the count is DERIVED from the cases that ran. Localized prose may sit either side of it — the German suites keep their German sentence and carry (N/M passed) beside it — because the token is for the gate and the sentence is for a reader.

The alternative, widening AreaProbe.PassSummary to tolerate prose, was rejected: it cannot be made to work for a language that does not use the word passed, it would have to guess how many words may sit in the gap, and the probe lives in the platform IMAGE — a change there reaches these gates only when the image is rebuilt and adopted, while a change here is proven on the PR that makes it.

A hard-coded count is the same defect wearing different clothes. Edu/LearningJourney printed 6/6 from a literal while executing a straight-line list of assertions; it is now a named case list whose length IS the denominator, so dropping a case moves the number.

What stops it coming back

scripts/check-test-suites.py — the Tests-area ratchet that already turns tests=skipped into a PR failure — gains a second assertion: every tests=ok line must carry a count, with --self-test cases in both directions. A suite that renders a countless green verdict is then a new hole, exactly like a type with no Tests area at all, and it fails the PR that introduces it rather than passing silently for weeks.

🚨 The sequence is not optional, here or anywhere it is repeated. The suites are fixed and a REAL green gate run is read back first — every one of the ten reporting N/M passed in the gate log — and only then does the assertion land, per repo. Armed over unfixed suites it reds the whole fleet on a defect that was never in anybody's diff.