Reading .NET-written text from Python

Finding (2026-09-18, issue #222). Every one of this repo's 56 committed scope proxies carried a UTF-8 byte-order mark in the middle of the file, and every reflection getter it emitted was null-unsafe. Neither is visible in an editor, in git diff, in code review, or in the generator's own --check. This page records the mechanism, the instrument, and what the same investigation found underneath.

1. encoding="utf-8" does not consume a BOM — it decodes it

scripts/gen-scope-proxies.py runs a .NET program (ScopeCodeGenerator) and reads back what it wrote. .NET's Encoding.UTF8 always emits a byte-order mark. Python's plain utf-8 codec does not consume that BOM — it decodes it to a literal U+FEFF character. Only utf-8-sig consumes it.

The script then concatenated that text after its own // <auto-generated> header:

emitted[target] = HEADER + produced_file.read_text(encoding="utf-8")   # ← the defect

So the BOM did not land at offset 0, where a BOM is ordinary and harmless. It landed mid-file, immediately before using System;, at byte 0x11a of all 56 files:

00000110  65 6e 65 72 61 74 65 64  3e 0a ef bb bf 75 73 69  |enerated>....usi|
00000120  6e 67 20 53 79 73 74 65  6d 3b 0a 0a 2f 2f 2f 20  |ng System;../// |

🚨 The symmetric trap is the write side. io.open(path, "w", encoding="utf-8-sig") adds a BOM the file never had — measured elsewhere in this fleet the same week, where it broke a JSON gate that read plain utf-8. The rule is directional, and both halves matter:

That third row is why gen-scope-proxies.py's --check comparison read stays plain utf-8 deliberately, and now says so in a comment. A drift check has to see every byte it would rewrite.

2. Why a mid-file BOM is not inert

Roslyn's lexer treats U+FEFF as whitespace, so the compiler reads an ordinary using System; and never complains. Every string-keyed dedup upstream sees a different line: core's skeleton generator keys the hoisted using directives on directive.Trim() under StringComparer.Ordinal, and .NET's string.Trim() does not strip U+FEFF — it is a format character, not whitespace. So "\ufeffusing System;" != "using System;", both directives survive into the emitted compilation unit, and the in-mesh compile reports CS0105 "the using directive appeared previously in this namespace" against content nobody authored.

That is the shape to remember: the trap does not raise. It emits a duplicate and keeps going, and the warning is then filed under the content's name rather than the tool's.

3. The instrument

One command, and it needs no mesh, no build and no reference set:

grep -c $'\xef\xbb\xbf' <the module's committed .cs files>    # 0 everywhere, or the BOM is yours

0 in each is clean. Anything else, check where — a leading BOM is ordinary, a mid-file one is this defect:

python3 -c "b=open('F','rb').read(); i=b.find(b'\xef\xbb\xbf'); print(i, 'leading' if i==0 else 'MID-FILE')"

🚨 Count before you window. grep -c first, and only then look at a window — a grep | head -20 that cuts at match 20 of 56 reads as absence.

The whole-repo sweep, and its known-good hits. Over all 2,537 tracked files (2026-09-18): 138 carry a leading BOM, which is ordinary for a file .NET tooling saved, and three carry one mid-file. All three are legitimate and none is this defect — RiskTransfer/content/videos/…mp4 is a binary false positive (restrict the sweep to text), and ReinsuranceDemo/Installer's Source/PackData.cs and Test/InstallerTests.cs hold the character on purpose: the first TrimStarts a BOM off imported CSV, the second feeds it one to prove that works. Anything else is a tool writing into content it does not own.

4. What the same investigation found underneath

A null-unsafe getter emit, 69 sites. The generator wrote

typeof(X).GetProperty(nameof(X.Y)).GetMethod;

Type.GetProperty returns PropertyInfo? and PropertyInfo.GetMethod is MethodInfo?, so that one line is two warnings — CS8602 (dereference of a possibly-null reference) and CS8601 (possibly-null assignment to a non-nullable field). It is invisible where the proxy lives: the mesh compiles a NodeType's sources as one concatenated compilation unit under NullableContextOptions.Annotations, so an authored #nullable enable earlier in the concatenation is still in force when the generated text is reached, while compile-check.py feeds each file as its own <Compile> item where the directive cannot reach across. The fix is upstream, in MeshWeaver.Plugins#2071, and it is a fix rather than a !: ?.GetMethod ?? throw new InvalidOperationException(...). The getter is guaranteed by the symbol the line was emitted from, so a null there is a metadata mismatch — naming it at type initialisation beats a NullReferenceException raised later inside Evaluate.

A generator whose provenance nothing recorded. This script does not own the generator: it resolves BusinessRules/Scope/GeneratorSource out of the sibling MeshWeaver.Plugins checkout, because shared=@... cannot cross repos. The output therefore depended on whatever commit that checkout happened to be parked on, and nothing printed which — a sibling on an older branch silently regenerates proxies without an upstream generator fix, which is precisely how #2071's null-safe emit would have been lost here again. The script now accepts --generator-source and prints the resolved path and its commit on every run:

generator source: /…/MeshWeaver.Plugins/BusinessRules/Scope/GeneratorSource
                  @ a25f4131a HEAD, origin/main, origin/HEAD

A regeneration whose provenance is not on the console is not a regeneration anyone can check.

Vendored scripts that had drifted out of the platform's API. discover_refs returns (refs, search_roots, layout); this copy still unpacked two and died on ValueError: too many values to unpack, so nothing could regenerate a proxy at all. 🚨 That is a pattern, so it was swept by count, not by the first hit: grep -rn 'discover_refs' --include='*.py' gives two call sites in this repo — gen-scope-proxies.py and run-node-tests.py — and both are fixed, together with the short_reference_set_refusal the platform's own compile-check applies (a short reference set must not pass quietly: it would emit proxies for, or run suites against, a framework nobody runs).

🚨 run-node-tests.py has a SECOND, deeper drift that is deliberately not fixed here. Past discover_refs it calls cc.usings_union(...), which the platform's compile-check.py no longer has. The replacement is not a rename: shape_authored_source / extract_using_statements made the using-hoist a MOVE rather than a copy — directives are removed from the code and placed in an import block, which is how the mesh itself hoists an alias. Porting run_set onto that shape is a correctness-sensitive change (its own comment is that "compiles under the gate" and "compiles under the test harness" must not diverge), so it is its own issue. Measured: of the seven cc.* symbols run-node-tests.py uses, six resolve and only usings_union is missing.

This means AGENTS.md's gate 5 does not currently run, on main as well as here — on main it dies one step earlier, at discover_refs. What CI actually runs of that script is run-node-tests.py --self-test, which is green.

5. The gap this leaves

🚨 No CI lane runs gen-scope-proxies.py --check. AGENTS.md calls it "the CI drift gate", and the jobs in .github/workflows/ci.yml are supersede, e2e-static, no-pins, preflight, validate, tag-modules, compile-check, test-repos, tests-ratchet, presentation, demo-data, publish-bake, gates-executed, ci-green — none of them invokes it. It cannot be wired as-is, because the check needs the sibling MeshWeaver.Plugins checkout that holds the generator, which CI does not have. The practical consequence: the committed proxies are the artefact that ships, and the only thing standing between a wrong one and production is running the script locally, with its provenance line read rather than skimmed.

Nine scope interfaces in Claims, Pricing, Reinsurance and Underwriting also have no committed proxy at all. That is a separate question from this one and is not addressed here.

Related

• MeshWeaver.Plugins#2071 — the same two defects in the repo that owns the generator, fixed first.
• Released platform selection for Reinsurance CI — how this repo's lanes choose the platform their scripts come from.