Distributed Job Scheduler

April 10, 2026 #ontic#corefwk#scheduler#distributed-systems Published

Notes on ONTIC's current scheduler implementation, its limitations, and some good external references on distributed job scheduling.

Resource

Ontic Implementation

We are designing a cron job scheduler. The job of the scheduler is defining a set number of jobs and executing them at a specified time.

Here are the important points:

  • How we are registering jobs
  • How we are executing jobs

Terminology

  • JobDefinition

    • Definition of a job.
    • Whenever you create a job, you create a JobDefinition.
    • It contains cronExpression for when the job needs to be scheduled and dirty for whether the job still needs a trigger registered.
    • Whenever a new job is added we set dirty=true.
    • A job with dirty true means it does not have a trigger registered yet.

  • JobRunInfo

    • After the job trigger is executed, JobRunInfo stores whether the job was successful or failed and the failure reason.
    • It contains jobTriggerId because it is associated with a trigger.
    • So each trigger will have one JobRunInfo.

  • JobTrigger

    • Everything about job execution.
    • Contains JobDefinition, previousFireTime, nextFireTime, JobRunInfo, and jobStatus.
    • If you want to execute a job you should create a trigger first.

Jobs Registration

  • In ONTIC only developers run jobs, so the registration part is controlled by InternalController, which is why it is not exposed externally.
  • You first define what job you want to schedule in MandatoryJobs (enum).
  • A typical job enum includes:
    • jobName
    • cronExpr
    • jobLevel (GLOBAL, ORG, SPACE)
    • jobData (job metadata)
  • A typical job can be scheduled at 3 levels:
    • GLOBAL
    • ORG
    • SPACE
  • You basically create a JobDefinition specifying the level you want the job to run at.
  • Just defining the enum does not schedule the job.
  • You still need to create a JobDefinition for it. This is the first step of job registration.
  • You register the job by running the internal curl retriggerJobs() method and passing the jobName that you added in MandatoryJobs.
  • retriggerJobs() basically deletes any existing JobDefinition and JobTrigger associated with that jobName, then creates a new JobDefinition with dirty=true based on the appropriate level.

Unregister Jobs

  • It is also possible to unregister jobs.
  • Mark that JobDefinition as dirty=true and enabled=false, then delete the corresponding JobTrigger.

Jobs Execution

  • OnticScheduler is responsible for executing all jobs.
  • It is deployed separately and executes all jobs by accessing the scheduler Mongo, which is at the global level.
  • The crux of job execution is:
    • Create trigger
    • Trigger all jobs
    • Reconcile stuck jobs
  • If you look at SchedulerConfig, which is a configuration class, you will see all methods are @Scheduled with their respective timings.

Trigger Scheduler

10 sec initial delay, runs every 2 mins

  • Every job needs a trigger to run.
  • This fetches all dirty jobs, which are the newly created ones, and creates triggers for them and saves them in Mongo.
  • It sets the trigger status to WAITING, sets nextFireTime, and so on.

Trigger Executor

15 sec initial delay, runs every 30 sec

  • Every 30 seconds it executes all triggers.
  • It checks the jobTrigger collection and if nextFireTime < currentTime + 30 sec, it collects that trigger for execution.
  • It first collects all WAITING job triggers and executes them.
  • Then it collects all FAILED job triggers and executes them.
  • For each trigger execution:
    • Status is first changed to QUEUED
    • A messaging event JobExecutionEvent is created and pushed to Kafka on the job_execution topic
    • During execution of FAILED jobs we first check if max retry count is reached
    • If max retry count is reached, mark the trigger as ABORTED
    • Else, re-trigger it

SchedulerTopicSubscriber

Listening on job_execution

  • Acquire lock on JobTrigger by updating the status to IN_PROGRESS
  • Execute job
    • Mandatory jobs implement OnticJob, which has the execute() method
  • Post execution:
    • Set JobRunInfo
    • Release lock
    • If success, set jobStatus=DONE
    • Else set jobStatus=FAILED

Trigger Reconciler

20 sec initial delay, runs every 15 mins

  • It executes stuck jobs, meaning jobs whose status is IN_PROGRESS but whose nextFireTime is already in the past.
  • Find jobs whose nextFireTime is older than the last 10 mins and which are IN_PROGRESS
    • These jobs are basically stuck during subscriber execution, so fail them, increase failure count, and update the JobTrigger
  • Find jobs whose nextFireTime is older than the last 30 mins and which are QUEUED
    • These were never picked for execution, maybe because events were not consumed or were lost, so just fail them

Failure Scenarios and Re-Designing

  • Single Point of Failure

    • Only one worker node (SchedulerTopicSubscriber)
    • This creates a single point of failure, because if that node goes down, no jobs will be executed

  • Limited Scalability

    • With only one worker node, the system cannot scale horizontally to handle higher job loads

  • Lack of Proper Coordination

    • There is no dedicated coordination layer with leader election
    • That can create issues if multiple scheduler instances are ever run

  • Rudimentary Job Recovery

    • Trigger Reconciler only runs every 15 minutes
    • Stuck jobs can remain inconsistent for too long

  • Tight Coupling

    • JobDefinition, JobTrigger, and JobRunInfo seem tightly coupled
    • That makes it harder to evolve the system independently

  • No Mention of Priorities

    • There does not seem to be a mechanism for prioritizing jobs
    • Important jobs may get delayed

  • Simplistic Load Balancing

    • The system relies on Kafka partitioning for load balancing
    • That might not be ideal for jobs with varying resource requirements

  • Lack of Resource Awareness

    • There is no indication that the system considers job resource requirements or worker capacity

  • No Explicit Failure Handling Strategy

    • There is retry logic
    • But there is no clear strategy for permanent failures or more complex error scenarios

  • Minimal Monitoring

    • There does not appear to be comprehensive monitoring and alerting for the scheduler itself

  • Manual Job Registration

    • The requirement to manually register jobs through InternalController could be cumbersome when the number of jobs grows

Design a Distributed Job Scheduler

These articles are really good: