Index backend
Document ingestion (Document ingestion) and content uploads end in a vector index: the extracted text, its chunks and their embeddings. Which backend holds that index is declared configuration, not a side effect of which module happens to be wired.
The option
| Setting | Value | Meaning |
|---|---|---|
Indexing:Backend (environment Indexing__Backend) |
unset | Default: the instance's own PostgreSQL (pgvector, the mesh database, one schema per partition). |
Postgres |
The same, declared explicitly. | |
InMemory |
A process-local index. Opt-in for evaluation and tests; it is empty after every restart, so the boot re-index refills it. Mesh-wide search paths that read the PostgreSQL content_chunks tables directly are not redirected to it, so they keep answering from PostgreSQL. |
|
| anything else | Fails loud, naming the value and the known backends: the boot service and every store resolution throw ContentIndexBackendException (a host that isolates module-service errors may log the boot one and continue, but then every consumer of the index still fails with that error). There is no fallback: indexing into a backend nobody chose is silent data placement. |
Matching is case-insensitive; an empty value counts as unset. An instance with no setting therefore indexes
in PostgreSQL — it is never "nowhere". Indexing is still inert with a stated reason when the declared
backend cannot run (no mesh database for Postgres, no embedding provider for either): the reading below
says so, and the index never quietly lands in a different backend.
The embedding provider (Embedding:Endpoint, Embedding:ApiKey) is independent of the backend. OCR and
text extraction sit in front of the index and are unaffected by this option.
Seeing the active backend
- Boot log, one line:
[ContentIndex] backend=Postgres source=default durable=True active=True(an inert one appendsinactive: <reason>), followed byaction=…once the reconcile has decided what to do. - Admin-visible reading: the node
Admin/ContentIndex(get Admin/ContentIndex, NodeTypeContentIndexState), rewritten on every boot:backend,declared,source,active,reason,indexedBackend,status, and the re-index progress counters.
Switching backend
Changing Indexing:Backend and restarting re-indexes the collections that hold indexed Document
nodes into the new backend:
- The boot reconcile compares
indexedBackend(the backend whose index is known complete) with the declared one. It moves only when a pass finishes with no failed collection, so a crash mid-way leaves the old value and the next boot continues. - The re-index is the existing hash-gated
ReindexAll: a file whose content hash is already in the target store is skipped, so a resumed or repeated pass redoes nothing, and the same content is never re-embedded twice in one backend. - Collections are processed one at a time; the reading node gets one write per collection
(
collectionsDoneofcollectionsTotal,collectionsFailed,currentCollection) — progress for an admin and the heartbeat other replicas lease against (15 minutes: a fresh heartbeat for the same durable target leaves the work to that replica; a stale one is resumed). - A pass with failed collections ends
status: Failedand is retried at the next boot.
An instance with no record yet is adopted, not re-indexed, when it runs the default: it has always
indexed in PostgreSQL. The InMemory backend re-indexes on every boot, since its memory is new each time.
Not covered by the walk (known limits):
- Chunks of ingested documents (
…/_Ingest/…) are keyed by the document path and written by each document's own hub. The walk starts fromDocumentnodes and does not enumerateIngestDocumentnodes, so those chunks are not migrated andcollectionsSkippeddoes not count them; re-parse the documents on the new backend. - A file deleted while another backend was active is removed only from that backend. Switching back copies existing files over but cannot know what was deleted meanwhile, so stale text can reappear in a backend that kept it. A forced rebuild does not remove it either (it visits existing files only).
- The
Documentinventory is read with a 100,000 limit; a list that hits it fails the switch rather than marking it complete. - Two replicas that read the same old checkpoint at the same instant can both start the pass. The hash gate makes that wasteful, not wrong.
- Extracted text is not cached across backends today, so a re-index re-extracts the files it copies; keying extraction by content hash is the OCR work's seam, not this option's.
Adding a backend
A backend is an IChunkedContentVectorStore. Add its name to ContentIndexBackends.Known, give
PostgreSqlContentIndexingExtensions.CreateStore its case, and decide its activation condition in
IsConfigured. IndexBackendSelectionTest.EveryKnownBackend_HasAStore fails until all three agree.
Proven
ContentIndexBackendsTest (default, case, blank, unknown value fails naming it), IndexBackendSelectionTest
(through the module's real registration: default is the PostgreSQL store, InMemory selects the in-memory
store, a bad value throws everywhere it is resolved), ContentIndexSwitchPlanTest (the boot decision) and
ContentIndexBackendSwitchTest (a real mesh: a fresh backend is refilled from the indexed content, the
reading shows it, a repeat is idempotent).