ESRA — The Internal Document Store

Not every corpus lives in a queryable database. The internal store is a governed document repository: an administrator enables it, initialises it, and uploads documents that become searchable content in their own right — searched through exactly the same retrieval path as any external source.

Off by default, enabled per tenant, and requiring ESRA_DATA_MANAGE to enable, initialise or configure.

Supported formats: .txt, .md, .csv, .pdf, .docx.

.doc is not supported. Legacy .doc is an OLE compound binary with no maintained managed open-source parser; the alternatives are a commercial library or a conversion sidecar, both disproportionate for a format deprecated eighteen years ago. Uploading one is rejected with an actionable message: convert to .docx and retry.

Enablement and Initialisation

Initialisation is an explicit, audited administrative action — not an implicit side effect of first use. It:

1. resolves credentials and validates every host through the outbound allowlist;
2. verifies the principal can create objects in the configured schema and is not a superuser — the store principal is the one ESRA principal that needs DDL, so it must be scoped tightly;
3. applies the store schema idempotently, recording a schema version;
4. registers the store as a synthetic source so it is searchable through the ordinary retrieval path with no special-casing.

It runs as a background job, not inline in a Portal request: DDL against a cold database can exceed any sane HTTP timeout. The Portal shows Initializing and polls.

The store's schema is applied automatically and idempotently, in a way that is safe to run against a customer-managed PostgreSQL instance without imposing platform-specific bookkeeping on it.

Search Happens at Segment Level

Documents are split into segments — pages for PDF, headings for DOCX and Markdown, header-plus-rows blocks for CSV, paragraph groups for plain text — and it is segments, not documents, that are indexed.

A 200-page PDF stored as one row dilutes ranking to near-uselessness, because term frequency is swamped by document length, and makes snippet generation re-parse the entire document per result. Segmenting fixes both. Segments roll up to their document for the result's record key; the best-scoring segment supplies the fragment.

There is no overlap between segments. Overlap helps embedding retrieval far more than lexical full-text search, and it inflates the index.

Each CSV segment repeats the header line, so every segment is self-describing.

Upload Security

File upload is the largest new attack surface ESRA introduces.

Type verification is by magic bytes, never by the client's declared content type or the file extension. A mismatch between detected type and extension is a rejection, not a coercion. Macro-enabled variants such as .docm are rejected outright.

OOXML hardening. .docx is a zip of XML, which brings two classic attacks:

• zip bomb — a maximum entry count, a maximum total uncompressed size and a maximum compression ratio, aborting on breach rather than continuing to decompress;
• XXE — every XML reader prohibits DTD processing and uses a null resolver. Embedded objects, external relationships and remote images are ignored, not fetched.

A breach yields a failed document with a MALFORMED_DOCUMENT classification — never a partial index, which would silently misrepresent the document's content.

Extracted text is untrusted and is scanned for prompt injection at ingest. Findings do not block indexing by default: the document is indexed and marked quarantined, excluded from retrieval until an administrator reviews it. A legitimate document about prompt injection would otherwise be undeliverable, while a silently indexed malicious one would reach every future retrieval. Quarantine converts a security decision into an operator decision, which is where it belongs. The behaviour (quarantine, reject, index anyway) is configurable per store.

Filenames are sanitised and used only as display metadata. Original bytes are stored as a database column, never at a filesystem path derived from user input — there is no path-traversal surface because there are no paths. Downloads always use application/octet-stream with an attachment disposition.

Antivirus scanning is out of scope and is a deployment responsibility. ESRA stores bytes and extracts text; it does not open documents in a renderer. Deployments handling untrusted uploads should place a scanning gateway in front of the Portal.

Ingestion

Extraction is always asynchronous. A 25 MB PDF can take tens of seconds and must not occupy a Portal request thread, nor be retried from scratch on a transient failure.

upload → validate → persist (Pending) → 202 Accepted
                                          ↓
worker: claim → extract → scan → segment → index → Indexed
                                        ↘ findings → Quarantined

Ingestion is built to be resilient, not merely functional:

• multiple workers can drain the ingestion queue concurrently with no risk of two of them extracting the same file, and no document is ever claimed without its content actually being handed over;
• a worker that disappears mid-extraction is detected automatically and its in-flight document is made available again, rather than being stuck forever — an ordinary deployment restart never surfaces as a batch of broken documents;
• failures are classified per document and capped by an attempt ceiling, with the classification and its remedy surfaced in the UI;
• one bad file fails that file. One unreachable store ends that tenant's turn only — it never stops the rest of the queue draining for everyone else;
• a large bulk upload is processed in bounded batches so it can never monopolize ingestion capacity, and reaching that bound is recorded rather than silently indistinguishable from a drained queue;
• a failed document keeps its original bytes whatever the retention setting says. Re-extraction is the only recovery, and discarding them would make the failure permanent.

Re-indexing by hand resets the attempt counter. A document that exhausted its retries could otherwise never be retried, which is the one case where a human has information the job does not — such as having fixed the source file.

Duplicates are rejected, not silently deduplicated — an administrator who uploads a file and sees nothing appear will upload it again. A content-hash collision within a tenant is reported with the title of the document it duplicates, rather than as a bare error.

Uploads report per file. A batch verdict cannot say which of ten files was the password-protected PDF, and refusing the whole batch over one would make the operator re-upload the nine that were fine. Each file comes back with its own outcome: stored, or refused with a reason stated as the remedy.

Query-Time Security

Two conditions are always enforced in the query itself, and neither can be disabled: tenant isolation, and a record-level filter drawn from the security scope assigned to the document at upload time by the uploading administrator. The record filter is composed exactly as it is for external sources, with the same fail-closed rule for a missing claim.

Soft-deleted documents and documents not in the Indexed state — including quarantined ones — are excluded structurally.

Management

Downloading a document's original bytes requires ESRA_DATA_MANAGE, not just ESRA_DATA_READ — the download endpoint carries its own manage-scoped authorization on top of the controller's read-scoped default, so a read-only operator can see and search a document but not pull the original file out.

Editing a document's extracted text in place is not yet exposed. The platform reserves the structure needed for that capability — previous content, revision, editor and reason — and the revision counter already increments on every (re-)extraction, but neither the API nor the Portal currently offer a way to replace a document's text directly; re-indexing from the retained original is the only way to change what is indexed today.

Document search in the management UI uses the same full-text path as retrieval, plus structured filters — one indexing implementation, not two.

The review screen shows what retrieval cannot

The management list shows every state, including the two a caller can never receive. That is the deliberate inverse of the query-time rule above: an operator's entire reason to be on this screen is the documents that are not working, and a list that hid them would leave the quarantine queue permanently empty. Store-wide counts sit above the list and double as filters, so the number of documents awaiting review is visible without navigating to them.

A document that reports Indexed with zero segments is called out. It is not searchable, and nothing about the state alone says so — the discrepancy would otherwise surface only as a search that mysteriously misses.

Releasing from quarantine is its own action, not a state field on an update. It is the one operation in the store that overrides a security finding, so it is individually visible in an access log and recorded against the document with the operator who took it. Before releasing, the operator sees exactly what the injection scanner matched. Release moves only a quarantined document, so it can never resurrect a failed one whose text was never extracted.

Re-index is offered only where the original bytes were retained. A store configured to discard them has given up re-extraction, and offering a button that silently does nothing would be worse than saying so.

Serving Document Fetch

The store is the best-served source for esra_{slug}_fetch, because segments reconstruct the document exactly: concatenated in ordinal order, paged by offset. hasMore and totalChars are therefore exact for this source, unlike external full-text sources where the total may be unknown.

Quarantined and soft-deleted documents are unfetchable for the same reason they are unsearchable.

Roadmap

Sync connectors, inbound and outbound, are planned as an additive capability on top of the store — content kept in sync from an external system will land alongside directly uploaded documents without disrupting existing content, and the Portal already distinguishes ownership and renders connector-owned content read-only in preparation.