Distributed Job Scheduler

April 10, 2026#ontic#corefwk#scheduler#distributed-systems

Notes on ONTIC's current scheduler implementation, its limitations, and some good external references on distributed job scheduling.

Notes on ONTIC's current scheduler implementation, its limitations, and some good external references on distributed job scheduling.

Resource

Ontic Implementation

We are designing a cron job scheduler. The job of the scheduler is defining a set number of jobs and executing them at a specified time.

Here are the important points:

  • How we are registering jobs
  • How we are executing jobs

Terminology

  • JobDefinition

    • Definition of a job.
    • Whenever you create a job, you create a JobDefinition.
    • It contains cronExpression for when the job needs to be scheduled and dirty for whether the job still needs a trigger registered.
    • Whenever a new job is added we set dirty=true.
    • A job with dirty true means it does not have a trigger registered yet.

  • JobRunInfo

    • After the job trigger is executed, JobRunInfo stores whether the job was successful or failed and the failure reason.
    • It contains jobTriggerId because it is associated with a trigger.
    • So each trigger will have one JobRunInfo.

  • JobTrigger

    • Everything about job execution.
    • Contains JobDefinition, previousFireTime, nextFireTime, JobRunInfo, and jobStatus.
    • If you want to execute a job you should create a trigger first.

Jobs Registration

  • In ONTIC only developers run jobs, so the registration part is controlled by InternalController, which is why it is not exposed externally.
  • You first define what job you want to schedule in MandatoryJobs (enum).
  • A typical job enum includes:
    • jobName
    • cronExpr
    • jobLevel (GLOBAL, ORG, SPACE)
    • jobData (job metadata)
  • A typical job can be scheduled at 3 levels:
    • GLOBAL
    • ORG
    • SPACE
  • You basically create a JobDefinition specifying the level you want the job to run at.
  • Just defining the enum does not schedule the job.
  • You still need to create a JobDefinition for it. This is the first step of job registration.
  • You register the job by running the internal curl retriggerJobs() method and passing the jobName that you added in MandatoryJobs.
  • retriggerJobs() basically deletes any existing JobDefinition and JobTrigger associated with that jobName, then creates a new JobDefinition with dirty=true based on the appropriate level.

Unregister Jobs

  • It is also possible to unregister jobs.
  • Mark that JobDefinition as dirty=true and enabled=false, then delete the corresponding JobTrigger.

Jobs Execution

  • OnticScheduler is responsible for executing all jobs.
  • It is deployed separately and executes all jobs by accessing the scheduler Mongo, which is at the global level.
  • The crux of job execution is:
    • Create trigger
    • Trigger all jobs
    • Reconcile stuck jobs
  • If you look at SchedulerConfig, which is a configuration class, you will see all methods are @Scheduled with their respective timings.

Trigger Scheduler

10 sec initial delay, runs every 2 mins

  • Every job needs a trigger to run.
  • This fetches all dirty jobs, which are the newly created ones, and creates triggers for them and saves them in Mongo.
  • It sets the trigger status to WAITING, sets nextFireTime, and so on.

Trigger Executor

15 sec initial delay, runs every 30 sec

  • Every 30 seconds it executes all triggers.
  • It checks the jobTrigger collection and if nextFireTime < currentTime + 30 sec, it collects that trigger for execution.
  • It first collects all WAITING job triggers and executes them.
  • Then it collects all FAILED job triggers and executes them.
  • For each trigger execution:
    • Status is first changed to QUEUED
    • A messaging event JobExecutionEvent is created and pushed to Kafka on the job_execution topic
    • During execution of FAILED jobs we first check if max retry count is reached
    • If max retry count is reached, mark the trigger as ABORTED
    • Else, re-trigger it

SchedulerTopicSubscriber

Listening on job_execution

  • Acquire lock on JobTrigger by updating the status to IN_PROGRESS
  • Execute job
    • Mandatory jobs implement OnticJob, which has the execute() method
  • Post execution:
    • Set JobRunInfo
    • Release lock
    • If success, set jobStatus=DONE
    • Else set jobStatus=FAILED

Trigger Reconciler

20 sec initial delay, runs every 15 mins

  • It executes stuck jobs, meaning jobs whose status is IN_PROGRESS but whose nextFireTime is already in the past.
  • Find jobs whose nextFireTime is older than the last 10 mins and which are IN_PROGRESS
    • These jobs are basically stuck during subscriber execution, so fail them, increase failure count, and update the JobTrigger
  • Find jobs whose nextFireTime is older than the last 30 mins and which are QUEUED
    • These were never picked for execution, maybe because events were not consumed or were lost, so just fail them

Failure Scenarios and Re-Designing

  • Single Point of Failure

    • Only one worker node (SchedulerTopicSubscriber)
    • This creates a single point of failure, because if that node goes down, no jobs will be executed

  • Limited Scalability

    • With only one worker node, the system cannot scale horizontally to handle higher job loads

  • Lack of Proper Coordination

    • There is no dedicated coordination layer with leader election
    • That can create issues if multiple scheduler instances are ever run

  • Rudimentary Job Recovery

    • Trigger Reconciler only runs every 15 minutes
    • Stuck jobs can remain inconsistent for too long

  • Tight Coupling

    • JobDefinition, JobTrigger, and JobRunInfo seem tightly coupled
    • That makes it harder to evolve the system independently

  • No Mention of Priorities

    • There does not seem to be a mechanism for prioritizing jobs
    • Important jobs may get delayed

  • Simplistic Load Balancing

    • The system relies on Kafka partitioning for load balancing
    • That might not be ideal for jobs with varying resource requirements

  • Lack of Resource Awareness

    • There is no indication that the system considers job resource requirements or worker capacity

  • No Explicit Failure Handling Strategy

    • There is retry logic
    • But there is no clear strategy for permanent failures or more complex error scenarios

  • Minimal Monitoring

    • There does not appear to be comprehensive monitoring and alerting for the scheduler itself

  • Manual Job Registration

    • The requirement to manually register jobs through InternalController could be cumbersome when the number of jobs grows

Design a Distributed Job Scheduler

These articles are really good: