Document enrichment

Maintainer, 2026-11-08: "connect ingestion in a way that it automatically tries to do ocr on pictures and get some description what is on it (put in node.description), try to create an icon (if it's a pic, try to create simplified variant, if it is text, try to create svg)" — then "if it is pdf, take it apart and extract tables and all", "also create summary in node.description", "and all content must be searchable", and "this ingestion of any blob content should be standard behaviour for any instance on the mac studio, not just control".

This page is the design of record. The indexing it builds on is Document ingestion and Content indexing coverage.

Where it runs

Enrichment is the second stage of indexing, on the instance that indexes the file. It is wired into both indexing paths:

It is on by default on every instance that has content indexing — the switch is ContentIndexing:Enrichment:Enabled (set it to false to turn it off for one instance). It never enqueues anywhere else: the work and the files stay on the instance, and on a Mac the models it calls are the Mac's own (below).

The line

DocumentEnricher.Enqueue puts the file into one serial line per process and returns at once. The line works one file at a time, so a burst of uploads never floods the model or the owner's machine; on a Mac Studio the instance's VM is already clamped to yield to the owner (Mac Studio attach runbook). The backlog is bounded at 2,000 files; a file over the bound is logged and caught up by its next re-index.

Progress is visible: every step writes the document's enrichment record ({document}/_Enrichment, node type DocumentEnrichment), and the document's page embeds its Progress view — status and current step, what was found, the per-step timings, every failure.

The steps

Taking a PDF apart

From the text layer, with the page and on-page box of everything found:

• Tables — runs of three or more consecutive lines whose words split into two or more cells at wide gaps and whose cell left edges line up on shared columns (PdfStructure.PageTables). The first line is the header; every row is exactly as wide as the header; a cell that reaches into the next column is a merged cell, kept in its first column and counted (MergedCells). The line just above the table is its caption when it is a single short line. Prose never splits into cells (a word space is a third of an em; a cell gap is over two characters), which the tests pin.
• Sections — from the PDF's bookmarks when it has them; otherwise from heading lines set clearly larger than the body text (≥ 1.15 × the median size, a single short line); with no heading at all, one section per page (none for a one-page document).
• Figures — embedded pictures at least 64 samples on each side and 2 % of the page; a scanned page's single full-page image is the page, not a figure. At most 24.

Scanned pages have no text layer: they are read from the indexer's OCR transcription of that page. The OCR prompt asks for tables as Markdown tables, so the same pipe-table parser finds them, and the transcription's Markdown headings split it into sections. Nothing else is needed on the host — there is no separate table extractor to install: the layout analysis is in-process (PdfPig), and the only model a scan needs is the OCR engine the instance declares.

Bounds: 60 tables, 24 figures, 80 sections per document; what is cut is recorded in the record's Notes.

The description

• A picture: what it shows (the vision model's caption) followed by the gist of the text in it: "A whiteboard with a diagram. Text in the picture: “QUARTERLY PLAN — ship v2 in May”".
• A document: a real summary — what the document is and is for, its key facts and figures, the parties and the dates, and the tables it contains. A short document is summarised in one call. A long one is summarised per section first (or per 12,000-character part when it has no sections), and the summary is written from those summaries. Each section node carries its own summary as its description.
• Bounded at 1,000 characters, cut at a sentence end (DescriptionText.Bound). The platform enforces no length limit on MeshNode.Description (measured: no bound in core's contract, its write validators or the Postgres column), so this is the enrichment's own bound — the size a search result, a card and a share preview show. The full text stays in the section and table nodes.
• Without a chat model the indexer's extractive summary is used, and the record says why.

The icon

• A picture: PosterizedIconRenderer — centre-cropped to a square, scaled to 64 px and posterised to five levels per channel, as a data:image/png;base64,… icon (SkiaSharp, the stack the portal already ships). A figure inside a PDF gets the same.
• A document: the model draws a pictogram as SVG, and the answer is untrusted: SvgIconSanitizer parses it (DTD prohibited, no resolver), REBUILDS it from an allow-list of drawing elements and presentation attributes (no script, no style, no text, no foreignObject, no use, no reference that leaves the icon), bounds it at 200 elements and 12 KB, requires a numeric viewBox, and finally holds it to the platform's own write rule (IconSafety.Refusal). A refused drawing is asked for once more with the refusal; a second refusal is recorded and no icon is written.

Never over a person's words

AutoFieldRule.MayWrite: an automatic value goes only into a field that is empty or still holds exactly the value the enrichment wrote last time (the record's AutoDescription / AutoIcon). Anything else was set by a person and is kept. The check runs inside the document owner's update, so an edit made while the model was thinking wins. The record says, for the description and the icon, whether the value on the node is automatic (DescriptionIsAutomatic, IconIsAutomatic) — that is the auto-generated marker the page shows. The same rule applies to each section, table and figure node's description. The marker lives on the record, not in the document's content, because a re-index replaces the document's content wholesale.

Idempotent

The record's ContentHash + PipelineVersion is the key: a document whose record carries its bytes' hash and the current version, Done or Partial, is skipped; a file already waiting in the line is not queued twice. Raising DocumentEnricher.PipelineVersion re-enriches every document once.

Best effort

Every step's failure is caught, logged ([Enrichment] {path}: step {step} failed) and recorded on the record (Errors, status Partial); the next step still runs. Enqueue never errors, and ContentIndexingService drops a throwing enricher with a warning — indexing never fails or waits for it.

Searchable

Every part is an ordinary mesh node, so it is found by both search kinds:

• Full text reads a node's name and description: a section's title and summary, a table's caption and synopsis (its columns and row count), a figure's description and the gist of its text.
• Semantic search reads the node's embedding, which covers its content: a section's text, a table's Text (caption, headers and every row as header: cell), a figure's OCR text.
• The document's own chunks (from the indexer) already carry every word of the text layer and of the OCR transcription, with page and box, so a phrase that exists only in one table cell, or only in a photo, is found through them and lands on the document at the right page; the table node is found by its caption and headers and semantically by its cells.

Embeddings come from the instance's declared embedding provider — on a Mac the local model (Hosting/MacStudioOnboarding → Local models). The enrichment adds no embedding path of its own.

Which models

Only the platform's abstractions, never a named provider:

On a Mac every one of these resolves to the Mac's own Ollama (the chat model with vision, the OCR engine OllamaVision, the local embedding model), declared by its onboarding; elsewhere to the configured providers. Every model call is ONE tool-free completion through the Http I/O pool with reasoning effort None, so a thinking model does not spend its budget reasoning about a summary.

The nodes

Every page is a template that binds its node by path (Doc/GUI/DataBinding → Templates first, data later): nothing on the hub waits for the node and builds controls out of it. A template's grid columns are fixed, so a table's page declares 12 columns and binds the node's GridRows (the header as the first row, then the body; a wider table's further cells are joined into the last column) and its GridTemplateColumns (one share per used column, zero width for the rest). The document page's Enrichment view binds the record's Report, a Markdown rendering the enricher writes with every record update.

Tests

src/MeshWeaver.ContentCollections.Indexing.Test:

• DocumentStructureTest — a PDF table with its header, typed rows, page, box and caption; a cell phrase in its search text; prose is never a table; headings split sections; a figure inside a PDF is extracted and a 1×1 dot is not; a scanned page's tables and sections come from its OCR and its page image is not a figure; an unreadable PDF says why; Markdown tables (padding, escaped pipes, captions) and sections (fenced code ignored).
• EnrichmentRulesTest — SVG sanitising (script, event handlers, external href and url(), foreignObject, style, text, javascript: all stripped while the drawing survives; no SVG, no viewBox, no shape, unparsable, a DTD and an oversized drawing refused), the description bound and map parts, the no-overwrite rule, the indexer queues the enrichment (and again on an unchanged re-index), and a failing enricher never fails the index.

src/MeshWeaver.ContentCollections.Indexing.Graph.Test/DocumentEnrichmentTest on a real mesh: a PDF with a table becomes a summarised document with section and table nodes and a sanitised SVG icon; identical bytes are not enriched twice; a person's description survives a re-index of changed bytes; a photo gets its caption, its text's gist and a picture icon; a scanned PDF's table comes from OCR; a figure inside a PDF is its own described node; a failing model is recorded as Partial, never thrown.

Measured on a Mac

2026-11-08, the whole pipeline (LiveEnrichmentCheck: indexer + OCR + describer + enrichment on a real mesh, one file at a time as the line runs) against each Mac's own Ollama (eiger reached through a tunnel to its loopback Ollama). The same three files on both: a photo of a sign (42 KB JPEG), a 21-page report with 19 tables and a figure (367 KB PDF), and a scanned receipt (one image-only page, 50 KB PDF).

What came out, on both: the photo's description is the caption plus the sign's text, with a posterised icon. The report is taken apart into 21 sections, 19 tables and 1 figure; its summary (633–643 characters) names the parties, the dates and the tables, and it gets an SVG icon. The receipt's table (3 rows × 4 columns) comes from the OCR transcription. No step failed. A single section summary measured alone on the Mac Studio took 3.8 s (1,847 prompt tokens, 99 completion tokens). The section summaries dominate a long document, and they are bounded at 24 model calls per document. Reproduce with LiveEnrichmentCheck's class comment.

Not established here

• Table detection is geometric. Ruled tables whose columns do not share left edges (right-aligned numbers under a left-aligned header), tables split across pages, and nested header rows are detected partly or not at all; row spans are not detected (only column spans). A layout model (Docling, a VLM per page) would do better and is not wired.
• A table node's embedding reads the node's content as the platform's node embedding composes it; a very large table may be cut there. The document's chunks still carry every cell.
• Full-text search on a part node reads its name and description only (the platform's full-text index); a cell phrase is found lexically through the document's chunks, which land on the document, not on the table node.
• Ollama inference is not clamped: the line is serial and the instance's VM yields to the owner, but the model runs in the host's Ollama, which the clamp does not reach.