Skip to main content

Job Orchestrator

Overview​

The Job Orchestrator is a new addition to IOMETE’s job management platform, designed to enable

  • Priority-based scheduling: Prioritize business-critical jobs over routine workloads.
  • Resource-aware execution: Submit jobs only when sufficient cluster resources are available.

Key Features​

Priority-Aware Scheduling​

Easily configure job priority (High vs Normal) to ensure critical jobs are executed first. The system maintains separate queues for each priority level, with high-priority jobs always processed before normal-priority ones.

Resource-Aware Execution​

Jobs are submitted only when cluster capacity is available, reducing the chances of failure due to insufficient compute resources. The orchestrator continuously monitors CPU & memory availability before scheduling job execution.

FIFO Queue Processing​

Within each priority level, jobs are processed in First-In-First-Out (FIFO) order. The system will process the first job in the queue completely before moving to the next job, even if resources are available for subsequent jobs.

Queue behavior:

  • High-priority queue is always processed before normal-priority queue.
  • Within each queue, jobs execute in the order they were submitted.
  • If the first job in queue cannot run due to resource constraints, subsequent jobs wait until resources become available or the blocking job completes.

Per-Job Opt-In​

Each job independently opts in or out via its Deployment Flow setting, allowing gradual migration and testing.

Using the New Flow​

How to Enable​

New Spark jobs created in the console use Priority-Based by default. To move an existing job:

  1. Navigate to Spark Jobs and open the job's configuration.
  2. Change Deployment Flow from Legacy to Priority-Based.
  3. Select your Priority:
    • High - for time-sensitive, business-critical tasks
    • Normal - for regular data processing workloads
Spark job form with Priority-Based selected as the deployment flow and Normal priority | IOMETESpark job form with Priority-Based selected as the deployment flow and Normal priority | IOMETE
tip

Start by testing the orchestrator with non-critical jobs before migrating production workloads.

How to Disable​

Change Deployment Flow to Legacy in the job's configuration to opt out of orchestration for a specific job.

tip

To move all jobs in a domain back to the Legacy flow at once, use this API endpoint:

curl --location 'https://<IOMETE_URL>/api/v1/domains/<DOMAIN_NAME>/spark/jobs/migrate-from-prefect' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <TOKEN>' \
--data '{}'

Please replace:

  • <IOMETE_URL> with your actual IOMETE instance URL
  • <DOMAIN_NAME> with the actual domain name
  • <TOKEN> with your authentication token

Configuration​

Helm Chart Settings​

Configure orchestrator settings in your values.yaml:

services:
  jobOrchestrator:
    # Verbose (INFO-level) stdout logging for server/worker pods (default: false, i.e. ERROR-level)
    verboseLogging: false
    settings:
      # Maximum jobs deployed in a single batch
      batchSize: 20
      # Job run cleanup configuration
      jobRunCleanup:
        enabled: true
        # Retention period in seconds (default: 30 days)
        retentionPeriod: 2592000

Settings Reference​

SettingDefaultSinceDescription
verboseLoggingfalse3.19.0Enables verbose (INFO-level) stdout logging for job orchestrator server and worker pods. Leave disabled for reduced (ERROR-level) log volume.
settings.batchSize203.15.0Maximum jobs validated and deployed per batch.
settings.jobRunCleanup.enabledtrue3.15.0Enable periodic cleanup of completed queue runs and logs.
settings.jobRunCleanup.retentionPeriod25920003.15.0Retention period in seconds (30 days). Older runs are automatically deleted.

S3 Log Storage [3.17.0+]​

Job orchestrator logs can be persisted to S3-compatible object storage (MinIO, Dell ECS, etc.).

services:
  jobOrchestrator:
    s3Logging:
      enabled: true  # disabled by default
      # Optional overrides — credentials and endpoint are derived from storage.s3CompatibleSettings
      # by default, or from the deprecated storage.minioSettings or storage.dellEcsSettings
      # when s3CompatibleSettings is not set.
      # Override only if using a separate bucket.
      # endpoint: "https://your-s3-endpoint"
      # accessKey: "your-access-key"
      # secretKey: "your-secret-key"
      # bucket: "your-bucket"
      # folderPath: "job-orchestrator-logs"

SSL Database Connection [3.17.0+]​

The job orchestrator can connect to its backing PostgreSQL database over SSL. Enable it in your values.yaml:

javaTrustStore:
  enabled: true
  caCertFileName: your_file.crt   # key name of the PEM certificate chain in the java-truststore secret

clusterDatabase:
  ssl:
    enabled: true

The caCertFileName must match the key name of a PEM certificate chain file stored in the java-truststore Kubernetes secret. See Custom Java Truststore for setup instructions.

System Properties​

Configure queue behavior via Admin Portal → System Configuration:

PropertyDefaultSinceDescription
job-orchestrator.queue.high.scheduling-share-percentage903.12.0Percentage of scheduling slots allocated to high-priority jobs. Remaining slots go to normal-priority jobs.
job-orchestrator.queue.head-timeout-seconds36003.15.0Time (seconds) a job can remain blocked at queue head before timeout action.
job-orchestrator.queue.head-retry-count03.15.0Number of retry attempts before cancelling a blocked job. Set to 0 for immediate cancellation after timeout.

Advanced Features​

Weighted Round Robin Scheduling [3.12.0+]​

To prevent normal-priority job starvation, the orchestrator uses Weighted Round Robin (WRR) scheduling between priority queues.

Default behavior (90/10 split):

  • 90% of scheduling slots go to high-priority jobs
  • 10% go to normal-priority jobs

This ensures normal-priority jobs still progress even when the high-priority queue is busy.

Queue Head Blocking Prevention [3.15.0+]​

Jobs blocked at the queue head due to quota limits are now automatically retried or cancelled after configurable thresholds. This prevents a single resource-intensive job from indefinitely blocking the entire queue.

How it works:

  1. When a job at queue head cannot be scheduled due to quota limits, the timeout counter starts.
  2. After head-timeout-seconds elapses, the system retries scheduling (if head-retry-count > 0).
  3. After exhausting retries, the job is automatically cancelled with a clear reason.

Job Queue Visibility [3.15.0+]​

Job details now show the specific resource blocking deployment when a job is waiting in the queue:

  • CPU - Insufficient CPU quota
  • Memory - Insufficient memory quota
  • Pods - Pod count limit reached
  • Storage - Storage quota exceeded

The UI also displays queue timeout retries, cancellation reasons, and reschedule events.

Job queue visibility | IOMETEJob queue visibility | IOMETE

Scheduling Reliability [3.15.0+]​

Jobs incorrectly scheduled due to stale quota data are now automatically retried. This reduces failures caused by timing mismatches between quota checks and actual resource allocation.

Cleanup & Maintenance [3.15.0+]​

The orchestrator automatically cleans up completed queue runs and logs to prevent unbounded data growth. Configure retention via jobRunCleanup settings in Helm values.

How to Troubleshoot Common Issues​

1. Job Orchestrator Won't Start - PostgreSQL Extension Error​

Error Message:

sqlalchemy.exc.DBAPIError: (sqlalchemy.dialects.postgresql.asyncpg.Error)
<class 'asyncpg.exceptions.FeatureNotSupportedError'>: extension "pg_trgm"
is not allow-listed for users in Azure Database for PostgreSQL
HINT: to learn how to allow an extension or see the list of allowed extensions,
please refer to https://go.microsoft.com/fwlink/?linkid=2301063
[SQL: CREATE EXTENSION IF NOT EXISTS pg_trgm;]
(Background on this error at: https://sqlalche.me/e/20/dbapi)

Cause: The Job Orchestrator uses the pg_trgm extension for partial string search functionality in the database.

Solution: Enable the extension in your PostgreSQL database: Allow extensions in Azure PostgreSQL

2. Jobs Stuck in Queue Due to Resource Constraints​

Problem: Jobs remain queued even when cluster appears to have available resources.

Potential Causes:

  • First job in queue requires more resources than currently available
  • Resource fragmentation across different node types
  • Quota limits blocking deployment

Automatic Handling (3.15.0+):

Starting from version 3.15.0, the orchestrator automatically handles blocked jobs:

  1. Jobs blocked at queue head are monitored via head-timeout-seconds (default: 1 hour)
  2. After timeout, the system retries based on head-retry-count setting
  3. If retries are exhausted, the job is automatically cancelled with a clear reason
  4. The UI shows which resource (CPU, memory, pods, storage) is blocking deployment

Manual Solutions:

  • Check the job details page to see which resource is blocking deployment
  • Monitor cluster capacity through Grafana dashboards
  • Adjust job resource requirements or scale cluster capacity
  • Temporarily change problematic jobs to Legacy deployment flow

3. All Jobs Stuck Due to One Problematic Job​

Automatic Resolution (3.15.0+)

With Queue Head Blocking Prevention enabled, this issue is now handled automatically. Jobs that cannot be scheduled are retried or cancelled after the configured timeout, preventing queue blockage.

Manual Fix (if needed):

  1. Navigate to the problematic job's configuration page
  2. Change the Deployment Flow from Priority-Based to Legacy
  3. This removes the job from the orchestrator queue and allows other jobs to proceed
  4. Debug the problematic job separately before re-enabling orchestration

ON THIS PAGE