Scheduling Model

This page describes a durable schedule. It covers storage, one fire for each slot, timezones, and operation.

Durable Schedules

A Cyclone schedule is a row in cyclone_schedules. The row remains after a restart, a failover, or a deploy. The schedule scanner is a goroutine inside Client.Start. The scanner reads each due schedule and fires it.

The important fields are:

Field	Purpose
id	The stable identifier, from WithScheduleID or from a hash.
kind	The handler kind to call, or the workflow marker for ScheduleWorkflow.
cron_spec	The cron expression or the cron descriptor.
location	The IANA timezone name. The API default is UTC.
input	The JSON payload for each fired run.
next_run_at	The time of the next fire.
last_run_id	The run ID of the latest fired run.
consecutive_failures	The count of scanner failures, enqueue failures, and workflow submit failures.
enabled	False after PauseSchedule or the deprecated Unschedule. ResumeSchedule sets it to true.
def_version	Registration pins the workflow-definition fingerprint (migration 012). The value is NULL for a step or action schedule, and for a row from before that migration.

workflow.Schedule is the method form of ScheduleWorkflow. For that schedule, the firing process must match the pinned fingerprint. The process refuses the fire when its builder fingerprint is different. The process does not submit a DAG that does not match. A later registration pins the local definition again.

Fire Records

Cyclone records each fire in cyclone_schedule_fires:

(schedule_id TEXT, scheduled_for TIMESTAMPTZ, run_id BIGINT)
PRIMARY KEY (schedule_id, scheduled_for)

The primary key on (schedule_id, scheduled_for) allows one fire for each schedule and slot. Two scanner processes can race on the same due schedule. Only one INSERT into cyclone_schedule_fires succeeds.

Exactly-Once Fire per Slot

The scanner claims a due schedule with SELECT FOR UPDATE SKIP LOCKED. In that same transaction, the scanner does these writes:

1. The scanner inserts a new one-node run, or a workflow run.
2. The scanner inserts one row into cyclone_schedule_fires. The unique constraint blocks a second fire for that slot.
3. The scanner sets cyclone_schedules.next_run_at to the next due time.
4. The scanner sets last_run_id.

All of those writes are in one transaction. When two scanners race on one schedule, SKIP LOCKED lets only one scanner continue. The other scanner moves to the next schedule.

You can start more than one Client.Start process, as in a deployment with more than one instance. Those processes do not make duplicate fires.

Timezones

The location field stores an IANA timezone name. "America/New_York", "Europe/Berlin", and "UTC" are examples. The scanner reads the cron expression in that timezone.

loc, _ := time.LoadLocation("America/New_York")
cyclone.Schedule(ctx, client, "0 9 * * 1-5", briefStep, in,
    cyclone.WithScheduleID("daily-brief"),
    cyclone.WithScheduleLocation(loc),
)

The scanner keeps local time across a DST change. A schedule for 09:00 in America/New_York fires at 09:00 local time. That local time stays 09:00 when DST is active. That local time stays 09:00 when DST is not active.

Missed Fire Behavior

The scanner can be stopped when a slot time passes. A deploy, a maintenance window, or a crash can stop the scanner. On the next wake, Cyclone fires that schedule one time. The fire belongs to the oldest missed slot. That slot is the next_run_at value that the row still holds. next_run_at then moves directly to the next future fire. Cyclone does not backfill the slots between those two times.

Example:

• The schedule is @hourly. The previous fire was at 13:00, so next_run_at is 14:00.
• The scanner is down from 14:00 until 16:30.
• At the restart at 16:30, Cyclone fires one time. The fire row stores scheduled_for = 14:00.
• Cyclone skips the 15:00 slot and the 16:00 slot. next_run_at becomes 17:00.

This catch-up fits most idempotent schedules, such as a report, a cleanup, or a sync. A job that must occur one time for every slot, with no skipped slot, needs a different design. An external trigger with guaranteed delivery is one example.

Multi-Client Safety

More than one Client.Start process can use the same database. SELECT FOR UPDATE SKIP LOCKED on cyclone_schedules controls scanners that wake together.

• Only one scanner claims each schedule row.
• Only one scanner fires the run for that row.
• The unique constraint on cyclone_schedule_fires is a second check against a duplicate fire.

Postgres locks are the only coordination that you need.

Schedule Poisoning

consecutive_failures increases when the scanner cannot create the fired run. The count includes a scanner error. The count includes an enqueue error. The count includes a workflow submit error. A handler failure after the run exists is stored on that run and on its steps. That failure is not stored on the schedule.

The schedule stays enabled. An operator can correct the cause and watch the next fire.

Monitor consecutive_failures with this query:

SELECT id, kind, consecutive_failures, next_run_at
FROM cyclone_schedules
WHERE consecutive_failures > 0;

Send an alert when consecutive_failures is above your limit. For example, use 3 as that limit. The alert shows a poison schedule early.

Schedule Properties

• Cyclone stores each schedule in Postgres. The schedule remains after a restart.
• A fire record blocks a duplicate for each due slot.
• After a handler failure, the fired run uses the normal step retry path.
• After downtime, the schedule fires one time on the next scanner wake.
• You can read cyclone_schedules.consecutive_failures. You can also read the queue views and the kind views.
• You can create a schedule inside a transaction with ScheduleTx. step.ScheduleTx is the method form of ScheduleTx.
• Code can manage a schedule with Schedule, ScheduleAction, ScheduleWorkflow, ScheduleTx, PauseSchedule, ResumeSchedule, and TriggerSchedule. The method forms are step.Schedule, action.Schedule, workflow.Schedule, and step.ScheduleTx.

Related

• Schedules
• Scheduled Job
• Schema