ESRA — Operations

Audit, metrics, tracing, health and diagnostics. ESRA introduces no new pipeline for any of these — it emits into the platform's.

Audit

ESRA emits governance audit records with a dedicated action namespace, so its stages appear in the existing Call Flow trace and flow to external immutable audit targets with no new plumbing.

Retrieval:

Administrative actions are not currently audited. There is no governance-audit action namespace for admin CRUD on sources, capabilities, exposures, stores or sync — connection tests, publication, rollback, enable/disable and every other admin action against these screens emit no governance audit record today. This is a real gap, not a design choice, and it is called out here rather than papered over: an administrator's own changes to ESRA configuration currently leave no audit trail distinct from the platform's generic entity-change logging (if any). The only audit record ESRA's admin surface writes is the single hardcoded esra.preview action from Try & trace (see Diagnostics below), which records that a query was previewed — it is unrelated to CRUD and does not fill this gap.

Capability invocation: a capability tool branches out of the mapping pipeline before the request.transform / response.transform audits that would otherwise carry its MCP payloads, so the Runtime writes its own capability.invoke record — the caller's tool arguments and the response they received, captured under the same per-exposure flag and byte ceiling as every other capture and, for ESRA, after the capability's own governance (masked fields masked, restricted fields absent).

The hot-path rule

Audit must never add latency or a failure mode to retrieval. Payload capture is conditional on a per-exposure flag, writes are asynchronous, and a failed audit write never fails an invocation — the same pattern the platform's tool.invoke already follows.

Never captured: connection strings, resolved secrets, generated SQL, raw provider errors. Prompts and result fragments are captured only under an explicit per-capability retention classification; a hash or redacted form is the default.

Warning messages carry a fixed string and, at most, a source name — never an id-to-host mapping and never error text.

Metrics

Every metric carries the tenant dimension. Tenant is never omitted, including on failures and denials.

This is the exhaustive list of instruments ESRA currently defines. Selected-source count, query fan-out, deduplication ratio, rerank candidate count, per-source retry/timeout rate as a distinct metric, circuit-breaker openings, and ingestion lag/queue depth are not currently instrumented — none of those exist as counters or histograms in code today, however useful they would be. Some of that information is visible elsewhere (e.g. circuit state on the source grid, described under Circuit breaking below; per-stage detail in the invocation trace), just not as a metric.

What the model has cost

A durable running total per capability — input tokens, output tokens, invocation count, model-call count — is tracked separately from the point-in-time metrics above. It answers a question neither the metrics nor the invocation trace can: the metric counters report current spend and are aggregated away, and the trace is a read model over an audit trail that is scanned in a bounded window and purged on a retention schedule, so it can say what one run cost and never what a capability has cost in total.

It is reported on the Portal's capability list and detail screens, and summed across every capability — retired ones included — on the overview, since a retired capability's spend is still spent. Counting happens at the one point every model call passes through on its way out, so a call that timed out or was retried is billed exactly like one that succeeded. Where a connector reports no usage the count falls back to the platform's four-characters-per-token estimate, and the total says how much of it was estimated rather than presenting the two as the same kind of number. Accrual is off the hot path — buffered in memory and flushed by a background writer that coalesces by capability — on the same terms as audit.

Tracing

One span per stage, plus one per provider query, as children of the MCP invocation span. A single distributed trace therefore covers MCP invocation → policy evaluation → orchestration → selection → planning → each provider query → aggregation → reranking → audit dispatch.

Health

Contributed to the Runtime host: provider registration, service-LLM reachability, internal-store availability, configuration validity.

Source connectivity is probed on a schedule, not on the readiness path. A failing optional source degrades a retrieval; it does not fail readiness. The four-state model — healthy, degraded, unhealthy, disabled — is surfaced per source in the Portal.

The sweep runs as a scheduled background job and probes every published, enabled source. It exists because health is not decoration: scope resolution excludes an unhealthy source, so without it a database that went away overnight keeps being selected and keeps failing every invocation that selects it until an operator probes it by hand.

The sweep is configurable and can be switched off for a deployment whose sources sit behind a connection-rate-limiting gateway. The Portal then shows when the probe last actually ran, rather than implying the recorded value is current.

Circuit breaking

A separate signal from health above, and a faster one: health is a scheduled probe's opinion, checked every few minutes; circuit state is the live resilience pipeline's view, updated the moment a source starts or stops failing. A source can show Healthy from the last probe with an Open circuit, because it started failing seconds after the probe ran.

Each source has its own circuit breaker: under a real outage, the breaker turns what would otherwise be every concurrent query burning its own retry budget and timing out independently into fast, immediate failures after an initial run of errors — until a single probing query decides the source has recovered. Breaker behavior is configurable per source on the Ceilings tab and defaults to on. State (Closed / Open / Half-open) is surfaced on the source grid and detail screen, refreshed on the same schedule as the health sweep rather than instantaneously — a deliberate choice, so that surfacing circuit state never adds a database write to a retrieval itself.

Result caching

Off by default, per source, on the same Ceilings tab: "Allow result caching" and a cache TTL (default 30 seconds, deliberately short — this collapses a burst of near-identical queries against a slow-changing source, not a data warehouse). Turning it on for a source whose content changes continuously just serves stale rows for no benefit; it is a setting for a reference catalog, not a ticketing system.

Results are cache-isolated per tenant, source, and caller, which is what makes this safe to turn on even for a source using a claim-narrowed record filter — two callers who are entitled to see the same data may share a cache entry, but a caller entitled to different data structurally cannot receive it, cache or not. Only raw, pre-mask rows are ever cached — masking re-applies on every read, cache hit or not, so a masking-policy change takes effect immediately rather than waiting out the TTL. Only Results-mode queries are eligible; a grounded answer always regenerates from current evidence, never from a cached result.

A cache hit still writes the invocation's audit record and stage trace — a cached answer is still a served answer for audit purposes, marked so in the esra.provider.query record's cacheHit dimension. Only the round-trip to the source itself was skipped.

Index and schema advisories

The scheduled health probe also flags conditions that make a mapping searchable but suboptimal, never conditions that make it unsearchable — those already surface as Degraded/Unhealthy above. Three closed-set advisory findings: a PostgreSQL index that exists but is a plain column rather than an expression matching the mapping's computed search vector, so it cannot actually accelerate the query; a matching expression index that was built against a text-search configuration the mapping no longer names; and, on either engine, a claim-narrowed record filter's column with no index covering it.

Advisory means exactly that: never a blocker, never a contributor to a source's health state, never a reason a source drops out of scope. The source grid shows a count of these findings as an amber dot, distinct from the health badge, since a source can be perfectly Healthy and still carry one — "reachable but suboptimal" is not a state the health badge alone can express. Each finding on the source detail screen's findings panel names the affected relation and column and states the concrete remediation as text an operator can hand to their DBA; ESRA never runs the DDL itself.

Errors and Warnings

Closed sets. Nothing outside them reaches a caller.

Connection failures name a remedy

Four failures happen before the driver is involved — an unallowlisted host, an unresolvable secret, an unparseable template, and a source configured for an isolation level its database forbids. Each carries its own classification rather than falling through to UNKNOWN, and the Portal's connection test states what to change:

Anything genuinely unrecognised still reports UNKNOWN. A confident wrong answer sends an operator further from the cause than an honest admission of ignorance.

A fifth, CIRCUIT_OPEN, is not a connection failure at all: it means the query never reached the driver — the source's circuit breaker was already open from prior failures and failed the call fast. It is permanently excluded from the retryable set, since retrying around an open circuit would defeat the point of tripping it; the remedy is to look at what opened the circuit, not to retry.

Errors — the invocation fails:

Warnings — the invocation succeeds with status: degraded:

Diagnostics

Try a query

On an exposure's Try & trace tab, an administrator runs one real retrieval and sees every stage of it. It is not a simulation: the same orchestrator, the same configuration and the same providers a caller reaches, running as one of the exposure's own client applications — so narrowing, audience and tool policy all apply. A preview that approximated the pipeline would be most convincing exactly where it was wrong, and an operator whose preview returns nothing has learned the truth about their configuration rather than about a bypass.

What it changes is the audit disposition, not the execution. The stage records are captured and returned instead of being written, and the preview writes one esra.preview operator-action record carrying the prompt, the status, the result count and the duration — never the results. An administrator testing a query must never appear in the trail as a caller having retrieved data: that trail answers "who read this record", and it is the one question it has to answer truthfully.

It requires ESRA_DATA_MANAGE, not the exposure role. Previewing returns real records from a real source, so the authority it needs is the authority to see that data — not the authority to decide who else may.

The response shows what a caller never sees: the resolved effective scope with narrowing notes, the generated plans, each query's rendered parameter metadata, per-source timings and result counts, all four per-result scores — provider, normalized, semantic and final — the deduplication merge, and every fallback taken.

An exposure that is not published, or has no client application bound, reports why it cannot run rather than reporting no results. The two need different responses from the operator: one is a configuration change, the other is a fact about the corpus.

Invocation traces

The same tab lists recent invocations and opens any one of them as a stage trace. This is a read model over the governance audit records ESRA already writes — there is no second store, because a second copy of the same facts could disagree with the audit trail, and the trail is the one that has to be true. Preview records are excluded from the listing, for the same reason they are not invocations.

The timestamp in the When column expands that run's stages inside its own row — stage, outcome, duration and detail — and clicking it again collapses them. The stages belong to the run above them, so they are a child row rather than a panel under the table: with several runs listed, a trace rendered at the bottom said nothing about which one produced it. Both the listing and the trace read a tenant's audit trail, so both take the exposure under inspection: it names the tenant for a platform administrator, whose session pins none. When either cannot be read, the panel says why — an empty listing means the exposure has never been called, and it must not be shown for a request that simply failed.

The trace is what separates four situations that look identical from an empty result set: the corpus has no answer, the selector skipped every source, the plan was rejected, or a provider query timed out. Each calls for a different response, and the results alone tell them apart not at all.

Outcomes are shown as OK, Partial, Failed and Not run, each explaining what it means for the answer. Partial is the one worth knowing: the step produced a result but not its best one, and its commonest cause is that no model is configured for the tenant, so the deterministic path ran — a setting rather than a fault. The audit trail keeps the pipeline's own words (Success, Degraded, Failed, Skipped); the screens do not.

Each row carries an outcome — succeeded, degraded or failed, rolled up from the stages, worst wins — and a failed one names the stage that broke, the class of failure, and the reason that stage recorded. It names the first failure rather than the last: a stage that fails usually makes the ones after it fail too, and what an operator needs is the cause. Degraded is reported as degraded rather than as success, which matters most when it is a model stage that quietly fell back to the deterministic path.

Callers get status and warnings. Everything else is administrator-only, deliberately.

What each step records

With Capture request and response payloads on for the exposure, the call flow carries the caller's tool arguments — the prompt among them — and the response they received, on the MCP Call step. The response is captured after governance: masked fields are masked and restricted fields are absent, exactly as the caller saw them.

The retrieval steps carry their own detail whether or not capture is on, because none of it is caller content: which sources were in effective scope, which of them were selected and which were passed over, how many queries were planned and how many rejected, and per provider query the source, the entity, the number of records and whether the result was truncated.

The terms a query searched for follow the capture setting, since they are the caller's own words. They are recorded as ESRA planned them — what had to match, what could match, exact phrases, and what was excluded — rather than as the statement that ran. Excluded terms are worth reading first: they are the usual reason a record somebody expected is missing. The generated SQL is never recorded, and neither is an entity's underlying table: the trail carries no database structure.

The wider call flow

Governance → Call Flow shows an execution end to end, and a retrieval call now appears there in full: the governance check, then every ESRA stage — effective scope, source selection, planning, each provider query, aggregation, reranking, the answer — each as its own node with its own timing, followed by the response. Repeated stages are repeated nodes, because a call that fanned out to two sources and had one time out is not the same call as one that queried once.

Configurability

Nothing is a hard-coded constant. Every numeric and behavioural default is a configured value at three narrowing levels:

platform ceiling   (appsettings, operator-controlled — the maximum any tenant may set)
      ↓
tenant default     (Portal ESRA settings)
      ↓
per-entity override (source / capability / exposure / store)

A value set above the level above it is a validation error, never a silent clamp — consistent with the narrowing-only rule for exposures.