Cyclone

Appearance

Run Deadlines ​

A per-step timeout (WithTimeout) bounds one execution of one step. It does not bound a whole run. A workflow that retries can last longer than the timeout of one step. A workflow that fans out can last longer than the timeout of one step. A workflow that parks on a signal can last longer than the timeout of one step. The product timeout × attempts × depth is easy to calculate incorrectly.

WithRunDeadline bounds the run directly. Pass it at submit:

go
runID, err := cyclone.Submit(ctx, client, FulfillOrder, in,
    cyclone.WithRunDeadline(30*time.Minute),
)

The deadline is wall-clock time from submit. When the run is still in progress after 30 minutes, the engine ends the run. The engine ends the run at the point the run has reached.

What Happens When a Deadline Fires ​

A background scanner finds each run in state running whose deadline has passed. The scanner fails each such run.

  • The scanner dead-letters each step that is available, running, or parked on a wait. The error kind is deadline.
  • The scanner cancels an in-progress handler through its context, the same way as Cancel. A handler that checks ctx.Done() stops at once. A handler that ignores ctx.Done() continues until it returns. Cyclone discards that late result.
  • The run moves to failed. GetRun reports the failure as the deadline (ErrorKind == "deadline"). This failure is distinct from an ordinary step failure.
go
st, _ := client.GetRun(ctx, runID)
// st.State == cyclone.RunFailed
// st.ErrorKind == "deadline"

The dead-lettered steps appear in the DLQ view. cyclone dlq list --reason deadline selects them. ExhaustedQuery{Reason: cyclone.ErrorKindDeadline} selects them. The step rows store error_kind = "deadline". The step rows store error = "run deadline exceeded". Read the rows with the CLI:

sh
cyclone dlq list --reason deadline       # every step killed by a run deadline
cyclone steps <runID>                    # see every step in the killed run, error text and all

Group the rows in SQL:

sql
-- Across all runs: every step killed by a run deadline. The view records the
-- terminal timestamp as completed_at for every terminal state, discarded
-- included; there is no discarded_at column.
SELECT run_id, node, kind, error, completed_at
  FROM cyclone_v_steps
 WHERE error_kind = 'deadline'
 ORDER BY completed_at DESC;

cyclone_v_runs does not carry error_kind. GetRun computes the error kind of a run from its failed step. The view does not store that error kind. To find these runs in SQL, read the steps:

sql
SELECT DISTINCT r.id, r.workflow, r.finished_at
  FROM cyclone_runs r
  JOIN cyclone_steps s ON s.run_id = r.id
 WHERE s.error_kind = 'deadline'
 ORDER BY r.finished_at DESC;

No Deadline by Default ​

A run with no WithRunDeadline has no deadline. A zero duration also means no deadline. A negative duration also means no deadline. The scanner does not change that run. The scanner does not change a run that reaches a terminal state before the deadline. The deadline applies only to a run that is still in progress.

Child Runs ​

The deadline of a parent run applies to its child nodes. When the deadline elapses, the engine dead-letters a waiting child node with error kind deadline. The engine dead-letters a wait the same way. The engine cancels the child run in the same transaction. A child run has no deadline of its own. WithMaxWait on the child node limits that wait.

A Pause Stops the Deadline ​

The deadline does not elapse while the run is paused. Resume moves deadline_at forward by the length of the pause. The run keeps the time that it had left. A run with 20 minutes left at the pause has 20 minutes left after Resume. See Pause and Resume Runs.

Testing Without Waiting ​

cyclonetest treats the deadline as one more time gate. AdvanceTime moves that gate.

go
runID, _ := cyclone.Submit(ctx, env.Client(), wf, in, cyclone.WithRunDeadline(time.Hour))
env.RunUntilIdle()                      // run parks, still running
env.AdvanceTime(2 * time.Hour)          // jump past the deadline
st := env.RunUntilComplete(runID)       // scanner fails it
// st.State == cyclone.RunFailed, st.ErrorKind == "deadline"

See Timeouts, Cancellation, and Leases for the per-step timeout. See Testing with cyclonetest for the AdvanceTime clock.

Cyclone stores each run in Postgres.