Jobs Worker Semantic Enrichment Processing

Purpose

This functional area processes background requests for mapping enrichment.

Each enrichment request corresponds to a tracked enrichment run that moves through a defined lifecycle as the LLM pipeline executes.

Main Capabilities

• receive queued enrichment work carrying a run identifier
• validate mapping state before processing
• generate and apply enrichment output through a chunked LLM pipeline
• distill an attached reference document once into a compact glossary, then inject only the relevant slice into each chunk
• record per-chunk LLM activity as child log entries under the parent run
• publish real-time progress snapshots after each processed chunk
• apply partial success when only some chunks succeed
• mark runs as completed, canceled, or failed with a structured error message
• receive cross-process abort signals and immediately stop in-flight LLM calls
• filter the LLM prompt to a caller-specified subset of endpoints, leaving unselected endpoints unchanged
• checkpoint every completed pass so a run interrupted mid-flight resumes instead of re-spending its token budget
• detect and recover runs abandoned by a worker that died without reporting an outcome

Functional Value

This keeps AI-assisted mapping refinement asynchronous and operationally manageable, with fine-grained visibility into every LLM call within an enrichment request.

Enrichment Runs

Every enrichment request creates a top-level enrichment run entity before the job is enqueued.

A run has a lifecycle:

• Requested — run was created and the job was enqueued
• Processing — the job has started executing and is actively making LLM calls
• Completed — all applicable enrichment was applied successfully
• Canceled — the operator canceled the run before or during execution; any partial LLM work is discarded
• Failed — the job could not complete, either due to a precondition failure or a processing error

The run records the provider, model, total item count, processed item count, failed item count, suggestion counts, and timestamps for when work started and completed. It also records a last-heartbeat timestamp and a completed-item count, which together let the platform tell a slow-but-healthy run apart from an abandoned one (see Interrupted Run Recovery).

When the job cannot proceed because the requested enrichment context is no longer valid, the run is marked failed with a descriptive error message. No side effects are applied to the mapping.

Interrupted Run Recovery

An enrichment run can be interrupted without the worker ever reporting an outcome — the process crashes, or the message broker withdraws the delivery because the run exceeded the broker's maximum unacknowledged-delivery time. Because a run is claimed exclusively before the first LLM call, and duplicate deliveries are deliberately refused so work already paid for is never replayed, an interrupted run would otherwise remain in Processing indefinitely: the mapping stays locked in Enriching and the Portal shows a progress bar frozen at the last completed chunk with no error surfaced.

Four mechanisms work together to prevent this.

Two health signals, because there are two failure modes. A claimed run reports liveness on a timer for as long as a worker holds it, and separately records progress each time a pass completes.

Liveness alone is not enough. Reporting it only on pass completion would make a worker that died moments ago indistinguishable from one still waiting on a slow response, forcing a threshold too long to be useful. But a timer alone is not enough either: a worker blocked forever inside a call that never returns keeps heartbeating perfectly while completing nothing, so it would look healthy indefinitely.

Both are therefore tracked, with separate thresholds. A run is treated as abandoned when its liveness signal stops (the process died) or when its progress signal stops for much longer (the process is alive but hung). The progress threshold must exceed the slowest single pass including its retries, otherwise a legitimately slow chunk is mistaken for a hang and taken away from a working run.

Claim and reclaim. A delivery may claim a run that is still Requested, or reclaim one that is Processing but has stopped heartbeating past the configured threshold. A delivery arriving while the original worker is still heartbeating is refused as a duplicate — reclaim applies only to runs that are genuinely abandoned.

Checkpoints. Each completed pass — the reference digest, the overview pass, and every operation chunk — is persisted as a checkpoint keyed by run and pass identity as soon as the LLM returns it. When a run is claimed and checkpoints already exist, the pipeline replays them and issues LLM calls only for the passes that never finished. A 29-chunk run interrupted after 7 chunks resumes at chunk 8 rather than restarting at chunk 1.

Checkpointing is treated as an optimisation, never a correctness requirement. If a checkpoint cannot be written, the run continues and only loses the ability to resume past that pass. If a stored checkpoint cannot be read back — for example after a contract change — that pass is simply re-run rather than failing the whole run.

Re-queueing. A refused delivery is acknowledged and discarded, which means the message that could later have reclaimed an abandoned run is already gone by the time the run goes silent. Recovery therefore cannot rely on a redelivery arriving at the right moment. Instead, each run records the parts of its originating request that cannot be derived from the run itself, and a scheduled sweep re-publishes the job for any run found abandoned. The freshly published job claims the run and resumes from its checkpoints.

Before re-queueing, the sweep signals the previous attempt to abort. A hung worker is by definition still running, so without this it would keep consuming its LLM budget and could still write results for a run another delivery has since taken over.

The sweep looks across all tenants, since a failed worker takes down whichever tenants' runs it happened to be holding, and then acts on each run inside that run's own tenant scope so every read and write remains tenant-checked.

The number of re-queues is bounded. A run that keeps dying — a payload the LLM cannot process, or a provider that is simply unreachable — exhausts its recovery attempts and is then failed for good: marked failed with a message stating how many items completed, its checkpoints cleared, and its mapping reset out of Enriching so an operator can retry. The same permanent failure applies to a run with no recorded request to re-publish, such as one created before recovery existed.

While a run is being recovered its mapping deliberately stays locked in Enriching. Releasing it earlier would let an operator start a second, competing run against the same mapping while the first is still being resumed.

The sweep re-checks staleness at the moment it writes, so a run that resumes heartbeating between the sweep query and the update is left untouched. It also reports the in-flight picture on every pass that finds work — how many runs are healthy and how stale the oldest heartbeat is — so an operator watching logs can tell a slow run from a stuck one without querying the database.

Recovery covers abandonment specifically. A run that fails outright — an LLM error that survives retries, or a validation failure — is still marked failed and has its checkpoints cleared, because a retry is issued as a new run with a new identifier.

Enrichment Cancellation

Operators can cancel an in-progress enrichment run from the Portal. The cancellation is a two-stage cross-process signal.

When a cancellation request arrives at the Portal API:

1. The mapping is reset out of the Enriching state.
2. The active enrichment run is looked up and marked Canceled in the database.
3. A MappingEnrichmentAbortCommand message is published on the message bus.

In the Jobs Worker, a dedicated MappingEnrichmentAbortConsumer listens for the abort command. On receipt, it looks up the in-flight run in an in-memory JobAbortRegistry and signals its cancellation token. The main job consumer links this abort token to the MassTransit consumer cancellation token, so the cancellation propagates into the active LLM call.

The job service also checks for a Canceled run status at the very start of execution and before processing resumes. This catches the case where the cancellation arrived before the job started, preventing unnecessary LLM calls.

If the job's ProcessAsync call receives an OperationCanceledException and the run is already in the Canceled state, the job returns a Cancelled outcome cleanly without escalating to a failure. If the cancellation came from an external source unrelated to the abort signal, the exception is re-thrown normally.

Chunked LLM Pipeline

Rather than sending the entire mapping in a single LLM call, the enrichment pipeline breaks the work into discrete passes and chunks.

Processing follows this sequence:

1. When a reference document is attached and reference distillation is enabled, a one-time reference-digest pass runs first. It sends the full document (plus the operation/field inventory) to the LLM once and receives a compact glossary that maps opaque source tokens (fields, parameters, action codes, endpoints) to concise meanings. Each glossary entry carries a key (the source token), a kind, a meaning, an endpoints list scoping where that meaning is valid, and an optional note. Because an opaque token such as field001 can mean different things on different endpoints (for example FirstName on one operation and customerCode on another), the pass emits a separate scoped entry per meaning; entries with an empty endpoints list are treated as global. The note captures important detail beyond the plain meaning — taxonomy/enum values, constraints (length, format, ranges), or endpoint-specific logic. This runs before planning so the chunks that follow are sized against the small glossary rather than the full document.
2. A chunk planner receives the enrichment request and divides the imported API operations into groups. When the request carries a SelectedOperationKeys filter, only the matching operations are included; unmatched operations are silently excluded from all chunks.
3. Each group is tested against a prompt budget estimator. The fixed, non-operation portion of the prompt (system instructions, mapping metadata, and — when distillation is disabled — the verbatim reference document) is reserved as overhead; only the operation content is split. If the estimated prompt size still exceeds the configured limit, the group is recursively split until each part fits or cannot be split further. If the fixed overhead alone meets or exceeds the budget, a distinct reference_context_exceeds_budget warning is emitted instead of blaming operation-field truncation.
4. When mapping name or description enrichment is requested and the separate metadata pass option is enabled, an additional overview pass is planned before the operation chunks.
5. The executor runs the reference-digest pass (if planned) and the overview pass first, then executes operation chunks, with optional concurrency up to the configured limit. Each operation chunk injects only the glossary entries relevant to its fields, parameters and endpoints — an entry scoped to specific endpoints is included only when the chunk contains one of them, so per-endpoint meanings do not bleed across operations. Injected entries render their endpoint scope and note alongside the meaning.
6. Each chunk execution writes a child log entry recording the pass kind, chunk identifier, prompt content, raw LLM output, suggestion counts, retry count, and completion status.
7. Transient LLM failures are retried up to the configured retry limit with a configurable delay between attempts. Invalid JSON responses can also trigger a retry if that option is enabled.
8. After all chunks have run, a proposal merger combines successful results. Endpoint suggestions are deduplicated across chunks by path, method, and operation ID. Field suggestions are deduplicated by direction, field name, schema name, and status code.
9. The merged proposal is filtered to the targets requested by the operator, then applied to the mapping as a system-generated draft.

Chunks that fail after retries are counted as failed items but do not prevent successful chunks from being applied. This means enrichment can produce partial results when only some chunks succeed.

Real-Time Progress Reporting

After each overview pass or operation chunk completes, the worker emits a progress snapshot for the active run.

Functionally, each progress snapshot carries:

• the enrichment run identifier
• the mapping identifier and tenant scope
• total item count for the run
• processed item count so far
• cumulative input, output, and total token counts

These progress updates are used to drive live Portal progress bars and in-progress dashboard metrics while the worker is still executing. They complement, rather than replace, the persisted run and chunk records used for historical review.

The same point in the pipeline also refreshes the run's heartbeat, so progress reporting and liveness detection cannot drift apart: any pass that advances the progress bar also proves the run is alive.

Validation Before Processing

Before any of the checks below, the job claims the run exclusively. The claim admits a run that is still Requested, or one that is Processing but abandoned (see Interrupted Run Recovery). A delivery that cannot claim the run returns immediately without touching the LLM, which is what prevents a duplicate or redelivered message from re-spending a token budget another delivery already paid for.

Once claimed, the job checks:

• the mapping exists and belongs to the expected tenant
• the mapping is in the Enriching status (any other status indicates a duplicate or stale job)
• the imported API definition exists and matches the mapping

If any check fails, the job returns early and the run is marked failed. The mapping status is reset to its previous state when appropriate to unblock it for future enrichment attempts.

Enrichment Targets

The job request carries a set of enrichment targets specifying which parts of the mapping the operator wants to improve.

Supported targets:

• mapping name
• mapping description
• endpoint names
• input field names
• input field descriptions
• output field names
• output field descriptions

The proposal is filtered to only the requested targets before it is applied. If the filtered proposal contains no applicable content, the run is marked failed and no changes are written.

In addition to content-type targets, the job request can carry a set of SelectedEndpointIds — stable endpoint Guid identifiers from the mapping's current version. When present, the job service resolves these IDs to METHOD::PATH::OPERATIONID operation keys and passes them to the chunk planner as SelectedOperationKeys. The planner then filters the imported API operations to that subset before building prompt chunks.

When SelectedEndpointIds is null or empty, all operations from the imported API are included in the LLM pipeline.

Attachment Context Injection

When the Portal API processes an enrichment request that includes attached reference documents, it extracts the text content from each file and forwards it to the Jobs Worker alongside the enrichment request.

The Jobs Worker forwards the combined reference text to the enrichment pipeline as a dedicated reference context, separate from operator-supplied custom instructions. Rather than embedding the full document into every chunk, the pipeline distills it once (see the Chunked LLM Pipeline sequence): a single reference-digest pass produces a compact glossary of the opaque source tokens — each entry scoped by the endpoints it applies to and optionally annotated with a note — and each operation chunk then injects only the glossary entries relevant to its own fields, parameters and endpoints. This keeps the authoritative document meaning available to the LLM while avoiding the per-chunk cost of re-sending a large document, which previously consumed most of the prompt budget and could starve the operation content.

If reference distillation is disabled (EnableReferenceDistillation = false) or the digest pass fails to produce a usable glossary, the pipeline falls back to injecting the reference document text directly into each operation chunk as before.

When no attachments are provided, the prompt is constructed as usual with no changes.

Configuration

The chunked enrichment pipeline is configurable through the SemanticEnrichment configuration section.

Key settings:

• ChunkSizeOperations — the number of API operations to include in each initial chunk before adaptive splitting (default 3)
• MaxPromptCharacters — the prompt budget limit in characters; chunks that exceed this are split adaptively (default 24000)
• EnableReferenceDistillation — when true, an attached reference document is distilled once into a compact glossary instead of being embedded into every operation chunk; set to false to restore legacy verbatim injection (default true)
• MaxReferenceDigestInputCharacters — maximum characters of attached reference text fed to the single reference-digest pass; longer input is truncated with a warning (default 48000)
• MaxReferenceDigestCharacters — maximum characters of the produced digest glossary (default 6000)
• MaxFieldsPerRequestBody — maximum number of request body fields to include for a single operation in a prompt (default 60)
• MaxFieldsPerResponseStatus — maximum number of response fields to include per response status in a prompt (default 60)
• SeparateMappingMetadataPass — when true, runs mapping name and description enrichment in a dedicated overview pass before operation chunks (default true)
• MaxChunkRetries — maximum retry attempts per chunk on transient LLM failures (default 2)
• RetryDelaySeconds — delay between retry attempts in seconds (default 2)
• RetryOnInvalidJson — when true, invalid JSON responses from the LLM are retried (default true)
• MaxConcurrentChunks — maximum number of chunks to execute concurrently; sequential execution is the default (default 1)
• TimeoutSeconds — HTTP timeout for LLM calls in seconds (default 300)
• Temperature — LLM sampling temperature (default 0.2)

Interrupted-run recovery is configured separately, through the Mapping:EnrichmentRunRecovery section:

• StaleRunThreshold — how long a processing run may go without a heartbeat before it is presumed abandoned (default 3 minutes). Keep it comfortably above HeartbeatInterval so that a few missed ticks — a garbage collection pause, a brief database blip — cannot cause a live run to be taken from its owner.
• HeartbeatInterval — how often a claimed run reports liveness while it is being processed (default 30 seconds)
• ProgressStallThreshold — how long a run may go without completing a pass before it is treated as hung, even while still reporting liveness (default 20 minutes). Must exceed the slowest single pass including retries.
• ReaperInterval — how often to sweep for abandoned runs (default 5 minutes)
• ReaperBatchSize — maximum number of runs handled in a single sweep (default 50)
• MaxRecoveryAttempts — how many times an abandoned run may be re-queued to resume before it is failed permanently (default 3)

Note that TimeoutSeconds bounds a single LLM call, not the run as a whole. A run consisting of many passes can satisfy every per-call timeout and still take hours end to end, which is why the message broker's maximum unacknowledged-delivery time must be sized against total run duration rather than per-call duration.