Content indexing coverage
The rule: every file in a content collection is either fully indexed — all of its text, chunked
whole — or recorded with the reason it is not. A file the indexer cannot read still gets its
Document node ({collection}/_Documents/…), carrying indexingGap (the reason) and chunkCount: 0,
and the same sentence is on the indexing activity's log line:
'decks/q3.pptx': NoText (0 chunk(s)) — no text extractor for '.pptx' files.
Find every file the index cannot serve with a search for nodeType:Document under the collection and
read content.indexingGap.
This page is the audit behind that rule: it was measured against the code of
src/MeshWeaver.ContentCollections.Indexing* (this repository) and MeshWeaver.ContentCollections (core),
not against other documentation. The flow itself is Document ingestion for attached
storage, and the upload → ContentIndexingObserver → ContentIndexingService.IndexFile path for
content collections.
Formats
| Format | Extracted by | Complete? |
|---|---|---|
.pdf |
PdfPig, word by word, every page (TextExtractor.ExtractPdf) |
The whole text LAYER. A scanned PDF has none and yields almost nothing — OCR (IOcrProvider) is being added beside the Excel/zip extractors; until it lands such a PDF indexes the little text it has, or is recorded as having none. |
.docx |
DocSharp, converted to Markdown (TextExtractor.ExtractDocx) |
Whole document. |
.md .markdown .txt .csv .json .html .htm .xml |
UTF-8 decode (BOM honoured) | Whole file, as text. |
.png .jpg .jpeg .gif .webp .bmp .tif .tiff |
IImageDescriber (ChatClientImageDescriber) when one is wired |
One model description, used as both the indexed text and the summary. Without a describer: recorded no text extractor for '.png' files. An empty description: recorded the image description returned no text. A FAILED describe (the model call or the client resolution throws) is a failed extraction, never an empty description: ChatClientImageDescriber lets the error through, so a vision outage during a forced rebuild keeps the earlier index instead of erasing it (ForcedRebuild_DuringAVisionOutage_KeepsTheEarlierIndex). Only "no vision model configured" answers empty. |
.xlsx .xlsm .xls .zip |
Not on main; the extractors are in review in MeshWeaver.Plugins#3826 | Today: recorded as no text extractor for '.xlsx' files. |
everything else (.pptx, .eml, .msg, .rtf, .odt, and plain-text formats outside the list — .yaml, .log, .cs, .py, .tsv, …) |
none | Recorded, never indexed: no text extractor for '{ext}' files. Plain-text formats outside the extension list are an open gap (below). |
Caps
| Cap | Where | Truncates the index? |
|---|---|---|
Chunk window 1,000 characters, overlap 150 (ContentIndexingOptions) |
TextChunker.Windows |
No — the windows cover the whole text, the last one runs to the end. |
Summary input 12,000 characters (ChatClientSummarizer.MaxTextChars) |
the Document summary only |
No — the summary is written from the head of the text; the chunks and parts carry all of it. |
| Extractive summary 280 characters | ExtractiveSummarizer |
No (summary only). |
Storage-ingestion preview 4,000 characters (IngestDocument.MaxText) |
IngestDocumentNodeType.Record |
No — a display preview; the parts carry all of it. |
Storage-ingestion original 256 MB (IngestionRuns.MaxOriginalBytes) |
IngestionRuns.Walk |
The file is not ingested. It is now named with the reason on the connection's log line (…, 1 skipped, not ingested: video/all-hands.mp4 (… bytes, over the … limit)); before, the line said only "n skipped (unreadable — see the log)". |
| Embedding input | EmbeddingProviderChunkEmbedder, the embedding providers |
No cap: each chunk is embedded whole. |
Extraction bounds of #3826 (TextExtractionLimits: MaxTextLength 8,000,000 characters, archive entries/bytes/directory records, workbook bytes, scanned cells, nesting depth) |
#3826's extractors | These are DoS bounds and stay. On that branch a bound is reported as a note INSIDE the extracted text ([truncated: …], [not extracted: …]) and in the pod log — not as a recorded status. ExtractedDocument.IndexingGap is the seam it should set so the bound reaches the Document (open item below). |
Skips — where a file is not indexed, and where that is recorded
| Case | Before | Now |
|---|---|---|
| No extractor for the format | NoText, one activity-log line naming no reason, no Document |
Document with indexingGap: no text extractor for '{ext}' files, chunkCount: 0; the log line names the reason. |
| Extraction throws (a corrupt PDF, a model that is down) | Activity Failed and the admin notification; the PREVIOUS version's chunks and Document stayed, describing other bytes |
Same failure, and the file is recorded: its chunks are removed and its Document says text extraction failed: …. A forced rebuild of UNCHANGED bytes that fails keeps the earlier good index (nothing is rewritten). |
| The file has no text (empty, whitespace) | NoText, no Document; a file that HAD text kept its old summary and parts |
Document with indexingGap: no text could be extracted…, no summary, its old parts pruned. |
| Unchanged bytes | Hash gate: Skipped, nothing re-extracted (the missing-Document heal still runs) |
unchanged |
| File gone before its activity read it | activity log warning | unchanged (nothing to index) |
| Indexing inactive on the deployment (no embedding provider / vector database) | inert by design: uploads proceed, nothing indexes (ContentIndexingPipelineExtensions.InactiveReason) |
unchanged — a configuration state, stated on the Content Indexing settings tab |
| Storage ingestion: file over 256 MB, unreadable, read timed out | counted, names only in the pod log | named with the reason on the connection's log |
| Storage ingestion: no extractable text | recorded on the IngestDocument (error) |
unchanged |
Which writes reach the pipeline
The pipeline indexes a file when IContentUploadObserver.OnUploaded is raised for it. Deletes are raised by
ContentCollection itself for every delete path; uploads are raised by the CALLER, so a write path that
does not raise is never indexed until a re-index walks its collection.
| Write path | Raises the upload? |
|---|---|
MeshOperations.Upload (MCP upload, the API) |
yes |
Executive Assistant SaveAttachment |
yes — it writes through MeshOperations.Upload |
Mail-copy attachments ({copy}/attachments, MeshMailAttachmentStore.Save and the legacy move) |
yes; the index follows the copy node and its delete removes it |
File browser upload (FileBrowser) |
yes |
Collection→collection import (ImportContentRequest), repo/GitSync content mirror (SyncContentFilesRequest), IFileContentProvider.SaveFileContent (AI file writes, data-extension writes) |
no — open item below |
Node pictures (NodeImageUpload), staging blobs, a streamed document's original (DocumentPartWriter) |
no, deliberately: derived or internal files |
Re-index and backfill
ContentIndexingObserver.ReindexAll walks every file of the named collections as one cancellable
activity and runs IndexFile for each. It is hash-gated, so a run indexes exactly the files the index
does not hold yet — which is the backfill for files uploaded before indexing existed — and re-attempts
every recorded gap (a recorded file has no stored hash). force re-extracts everything.
It is reached from the Space settings tab Content Indexing (Re-index all content, Rebuild), for the
Space's own content collection only.
Open
- Imports and repo syncs do not announce their files. Raising the upload from the owning hub's
write handlers would cover them, but a boot re-import re-sends every content file of a repository, and
each raise costs an indexing activity — the per-file burst
Doc/Architecture/ContentRoute503(core) describes. The shape that fits is ONE batched index activity per sync; that is an owner decision. - Backfill is per Space and by hand. No run walks every Space, and mail-copy
attachmentscollections have no settings tab. Whether a fleet-wide backfill runs once, on boot or on a schedule is an owner decision; it would be a Code node template triggered byRequestedStatus = Running(Activity Control Plane, core). - Plain-text formats outside the extension list (
.yaml,.log, source code) are recorded, not indexed. Sniffing valid UTF-8 in the default branch ofTextExtractorwould index them; it sits in the code #3826 rewrites, so it follows that change. - #3826's bounds should set
ExtractedDocument.IndexingGap("not indexed: {bound} exceeded") wherever they cut, so a partial file is recorded on itsDocumentand not only noted in its text. - Document view.
indexingGapis on the node's content; the Document page does not show it yet.
Proven
IndexingGapRecordedTest(unit): unsupported format, failed extraction removing the previous version's index, a failed forced rebuild of unchanged bytes keeping the earlier index, a file replaced by an empty version, an empty image description, and the fully-indexed positive control.MeshDocumentSinkTest.UnreadableFile_IsRecordedAsADocumentNode_NamingTheGap_AndDropsTheOldParts(real mesh): theDocumentnode carries the gap and the previous version's parts are pruned.ContentIndexingPipelineTest.ReindexAll_RecordsTheFileItCannotIndex_WithTheReason(real mesh, real re-index activity): the log line names the reason and the node records it.IngestionSkippedNoteTest: the storage walk names each skipped file with its reason, bounded.
Negative control: with the recording reverted, five of the unit cases, the node case and the re-index
case fail — the re-index log then reads 'decks/q3.pptx': NoText (0 chunk(s)). and no node is written.