AI API Mapper
Technical walkthrough →
Technical walkthrough →
/ docs · functional specification

The full platform
functional reference.

Every functional area of AI API Mapper documented at implementation depth so architects, security reviewers, and platform engineers can evaluate the product with real technical context.

Jobs Worker · Overview
On this page

Jobs Worker

Purpose

The Jobs Worker is the background processing component of the platform. It runs asynchronous work that should not happen inline with portal requests or runtime invocations.

Functionally, it supports platform automation, scheduled maintenance, and deferred processing.

Main Responsibilities

The Jobs Worker currently handles:

  • semantic enrichment job processing
  • scheduled retention and cleanup work
  • message-driven background execution
  • scheduler health and job registration

Detailed documents for each Jobs Worker area:

Semantic Enrichment Processing

One of the main Jobs Worker responsibilities is processing mapping semantic enrichment requests.

Each request is tied to a tracked enrichment run that records the full lifecycle from request to completion.

Functionally, that means:

  • receiving a queued enrichment request carrying a run identifier
  • loading the target mapping and imported API definition
  • validating that the request is still current and applicable
  • planning an execution strategy that divides API operations into prompt-budget-aware chunks
  • running a separate overview pass for mapping-level metadata when configured
  • executing each chunk as an individual LLM call with retry support for transient failures
  • recording each LLM call as a child chunk log entry under the parent run
  • publishing live progress snapshots after each processed chunk, including cumulative token usage
  • merging partial results from successful chunks even when some chunks fail
  • checkpointing each completed pass so an interrupted run resumes rather than re-spending its token budget
  • applying the merged proposal to the mapping as a system-generated draft
  • marking the run completed or failed with a structured error message

This keeps enrichment work asynchronous and avoids making Portal users wait for long-running AI operations.

An enrichment run can be interrupted without its worker ever reporting an outcome — a process crash, or the broker withdrawing the delivery because the run outlived its maximum unacknowledged time. Runs report liveness on a timer while they are being processed, and a scheduled sweep re-queues any run that goes silent so it resumes from its checkpoints instead of restarting. Runs that cannot be recovered, or that have exhausted their recovery attempts, are failed and their mappings released. See Jobs Worker Semantic Enrichment Processing for the full model.

Retention And Data Lifecycle

The Jobs Worker also enforces scheduled data lifecycle rules.

Supported retention activities include:

  • governance audit retention
  • inactive refresh-token session retention
  • device flow session expiry and purge

Device flow sessions that remain in a Pending state after their expiry time are transitioned to Expired status. Expired, denied, and approved sessions are purged from the database once they are older than the configured retention window.

The retention process is scheduled through Quartz and can be enabled or disabled through configuration.

Messaging And Execution Model

The Jobs Worker uses a message-driven model for background work.

Functionally it supports:

  • MassTransit-based consumer execution
  • configurable transport options for background messaging
  • in-memory transport for single-host development and RabbitMQ transport for broker-backed deployments
  • retry behavior for background processing

This makes it the main asynchronous execution engine for cross-component platform workflows.

Scheduling Support

The Jobs Worker includes a Quartz scheduler host for recurring jobs.

Its functional role is to:

  • register recurring jobs at startup
  • trigger maintenance workloads on a schedule
  • expose scheduler health

Supported Background Workloads

Current workload categories include:

  • mapping semantic enrichment
  • recovery of enrichment runs abandoned by a dead worker
  • data lifecycle retention (governance audit records, refresh token sessions, device flow sessions)
  • governance audit handling in broker-backed deployments

Operational Value

The Jobs Worker improves the platform by:

  • separating long-running and scheduled work from user-facing interactions
  • making enrichment and retention reliable and repeatable
  • reducing load on the Portal and Runtime hosts
  • providing a clear operational home for background automation

Protocols And Platform Patterns

From a functional standpoint, the Jobs Worker participates in:

  • MassTransit message processing
  • Quartz scheduling
  • platform health and metrics endpoints
  • a lightweight host status endpoint that reports scheduler and transport mode

Platform Data Store Support

The Jobs Worker uses the shared platform database for enrichment state, retention processing, and other background platform data. This shared data model stores the records that the worker updates as it processes enrichment, retention, and related background workloads.

It is not directly used by portal end users or AI callers, but it is essential to the platform’s operational completeness.

PreviousGovernanceRuntimeNextSemantic enrichmentJobs Worker