Lifecycle of a Run

This page shows how a run and its steps move through states. The path goes from submission to completion or to failure.

Run Lifecycle

Cyclone inserts a run in state running. The submission and the first steps use the same transaction. A run has no pending state. One discarded step fail-fasts the run to failed, unless every downstream node declared OnDepFailure(CancelBranch). In that case, Cyclone cancels only that branch. The run can still reach completed. Steps have a pending state. Runs do not have a pending state.

An operator can pause a run in state running. The operator can resume that run later. A paused run is not terminal. A paused run starts no new work. A step in state running at the pause can still complete the run. That step can also fail the run. See Paused run.

Step Lifecycle

The diagram leaves out the wait states. Pause also moves waiting_signal, waiting_timer, and waiting_child steps to paused. Resume moves those steps back.

State Descriptions

Pending

A step is pending at submit time when an upstream dependency is not complete. The step waits for all parents to reach completed. Then the step moves to available.

A step in a one-node run starts in available. A standalone job is a one-node run. A step with no upstream dependency also starts in available. These steps do not start in pending. These steps are claimable immediately.

Available

A step is available when every upstream dependency is completed and now() >= scheduled_at. Workers poll for available steps in priority order, then in FIFO order. A step stays available until a worker claims it.

The metric cyclone_v_kinds.oldest_available_age shows the age of the oldest unclaimed step. That age is the primary backlog indicator.

Running

A worker has claimed the step. The claim writes:

• state = 'running'
• lease_expiry = now() + leaseTTL
• lease_owner = clientID
• lease_generation incremented by 1
• started_at = now()

The claim does not change the attempt count. The attempt count changes only when an execution fails.

A heartbeat goroutine for this step renews lease_expiry every leaseTTL/3 while the handler executes. The handler receives a cyclone.Context with RunID(), Node(), Attempt(), and CompletionTx().

Completed

The handler returned nil. Cyclone then does these actions:

1. Cyclone stores the return value as JSONB in cyclone_steps.output.
2. Cyclone sets the step to completed in the same transaction.
3. Cyclone makes a downstream step available when this step was its last incomplete dependency.
4. If this step is the last step, Cyclone marks the run completed.

Cyclone does all of these actions in one Postgres transaction. The handler uses that same transaction through ctx.CompletionTx().

Retrying

retrying is not a distinct state. The step moves back to available with scheduled_at = now() + backoff(attempt). Cyclone has already incremented the attempt count. On the next claim, the worker calls the handler again.

This path starts when the handler returns a non-permanent error and attempt < maxAttempts.

Discarded

A step is discarded when:

• The handler returned a Permanent(err) error, or
• The handler returned an error and attempt == maxAttempts, or
• The lease expired again after WithMaxReclaims reclaims. The recovery sweep then dead-letters the step with error_kind = "reclaim_limit". The sweep does not reclaim the step one more time.

A discarded step triggers fail-fast. Cyclone cancels every non-terminal step in the run. The run moves to failed. OnDepFailure(CancelBranch) is the exception. The exception applies when every node downstream of the discard declares that policy. Then Cyclone cancels only that branch. The run continues.

Operators reopen discarded steps with:

client.Retry(ctx, runID, "node-id")      // one node
client.RetryAll(ctx, runID)               // all discarded in run

Cancelled

A step is cancelled when:

• The operator called client.Cancel(ctx, runID), or
• Another step in the same run was discarded (fail-fast propagation), or
• One dependency of this node was discarded and this node declared OnDepFailure(CancelBranch) (branch cancellation).

Cancellation affects only non-terminal steps. Those states are pending, available, running, waiting_signal, waiting_timer, waiting_child, and paused. Cyclone changes all of those steps in one transaction. A cancelled child node cancels its active child run in the same transaction. The handler context of a step in state running is cancelled on the next heartbeat. That happens when the renewal no longer matches a running row. The handler can observe ctx.Done() and return. If the handler does not return, the CAS guard discards the late completion.

Waiting for a child run

A child node starts a second workflow as a child run. When its dependencies complete, the node goes from pending to waiting_child. No worker claims that node. No worker holds that node. A worker that registered the workflow inserts the child run, its graph, and its parent link in one transaction.

When the child run completes, the engine completes the node with the result of the child run. The engine discards the node when the child run fails. The engine discards the node when the child run is cancelled. The engine discards the node when the WithMaxWait limit elapses. The error kind is permanent. Then OnDepFailure applies, as for any discarded node.

The engine ends the node in a new transaction after the commit that ended the child run. That transaction locks the parent run first, as every child operation does. After a crash between the two commits, the wait scanner finds the ended child run and ends the node.

Paused

A step is paused while its run is paused. run.Pause moves each available, waiting_signal, waiting_timer, and waiting_child step of the run to paused. The claim query does not read state paused. The wait scanners do not read state paused. No worker claims the step. No wait completes. A step that fails, snoozes, or is reclaimed during the pause goes to available and then to paused. Those two changes occur in the same transaction.

run.Resume moves each paused step back to the state for its kind. A timer goes to waiting_timer. A signal wait goes to waiting_signal. A child node goes to waiting_child. Any other step goes to available. A timed wait keeps the time that it had left. The WithMaxWait limit of a child node also keeps the time that it had left.

Paused run

run.Pause stops new work in a run in state running:

• No worker claims a step of the run. No pending step becomes claimable.
• No wait completes. The wait timers stop. The run deadline stops. Resume continues them with the time that they had left.
• A step in state running at the pause continues. That step keeps its lease and its attempt timeout.
• Signal stores signals in the mailbox. The waits receive those signals after Resume.

A step that ends during the pause can end the run. When that step completes the last node, the run completes. No work remains. A paused state would keep Wait and Result waiting with no work left. When a discard fails the run, the run fails as it does with no pause. Cyclone cancels the paused steps and the pending steps. A pause does not hold a finished run. A pause does not hide a failure.

run.Resume moves the paused steps back and sets the run to running. run.Resume also promotes each pending step whose dependencies completed during the pause. A pause applies to one run. A pause does not pause the child runs of that run. A waiting child node is paused. When its child run ends during the pause, the node does not move until Resume. See Pause and Resume Runs.

Failed run

The run moves to failed when a step is discarded and OnDepFailure(CancelBranch) does not contain the discard. The discard is not contained when a downstream node uses the default FailRun policy. The discard is also not contained when the discarded node is a leaf with no dependents. Cyclone then cancels all non-terminal steps in the same transaction. The run stays failed until an operator calls Retry or RetryAll. That call reopens the discarded nodes. The run then returns to running.

A contained discard does not fail the run. Cyclone cancels only the declaring branch. The run completes when no non-terminal step remains.

Retention and purge

Terminal runs stay in the database with no time limit, unless one of these is true. The terminal states are completed, failed, and cancelled.

• You set WithRetention(d) on a step. The run is then eligible for purge d after the run finishes.
• You set WithRetentionSweep(RetentionConfig{...}) on the client. A background goroutine then does a bounded delete on the interval. The delete honors WithRetention on each step. The delete uses DefaultAge when a step has no WithRetention.
• Or the operator calls client.PurgeTerminalRuns(ctx, ...) manually.

A paused run is not terminal. Retention does not delete it.

A terminal child run stays until its parent run completes or is deleted. Until then, the parent node can still need the result. A retry can reuse that result.

State Transition Table

From	To	Trigger
(new)	pending	Submit or Enqueue (step has upstream deps)
(new)	available	Submit or Enqueue (no deps or standalone job)
pending	available	Last upstream dep reaches completed
available	running	Worker claims with FOR UPDATE SKIP LOCKED
running	completed	Handler returns nil
running	available	Retryable error; attempt < maxAttempts; backoff elapsed
running	discarded	Permanent(err), or attempt == maxAttempts
running	available	Lease expiry → recovery sweep reclaims (same attempt, no backoff, reclaims + 1)
running	discarded	Lease expiry after WithMaxReclaims reclaims → recovery sweep dead-letters it (reclaim_limit, same attempt)
running	cancelled	Fail-fast from another discarded node, branch-cancellation under CancelBranch, or client.Cancel
available	cancelled	Fail-fast, branch-cancellation, or client.Cancel
pending	cancelled	Fail-fast, branch-cancellation, or client.Cancel
waiting_signal	cancelled	Fail-fast, branch-cancellation, or client.Cancel
waiting_timer	cancelled	Fail-fast, branch-cancellation, or client.Cancel
discarded	available	client.Retry or client.RetryAll
pending	waiting_child	Last upstream dep of a child node reaches completed
waiting_child	completed	The child run completed; the node stores its result
waiting_child	discarded	The child run failed or was cancelled, the WithMaxWait limit elapsed, or the run deadline elapsed
waiting_child	cancelled	Fail-fast, branch-cancellation, or client.Cancel; the child run is cancelled too
discarded, cancelled	pending	client.Retry or client.RetryAll of a child node; the node reuses its child run
available, waiting_signal, waiting_timer, waiting_child	paused	run.Pause
running	paused	A failed attempt, a snooze, or a reclaim in a paused run
paused	available, waiting_signal, waiting_timer, waiting_child	run.Resume; the kind of the step gives the state
paused	cancelled	Fail-fast, branch-cancellation, or client.Cancel
(new run)	running	Submit or Enqueue
(run) running	completed	All steps completed, or every discard was contained by CancelBranch and nothing non-terminal remains
(run) running	failed	A step was discarded and the discard was not contained by CancelBranch
(run) running	cancelled	client.Cancel
(run) failed	running	client.Retry / client.RetryAll reopens discarded nodes
(run) running	paused	run.Pause
(run) paused	running	run.Resume
(run) paused	completed	A step in state running at the pause completed the last node
(run) paused	failed	A step in state running at the pause was discarded and was not contained by CancelBranch
(run) paused	cancelled	client.Cancel

Related

• Retries, Leases, and Recovery
• States
• Retries and Permanent Failures
• Timeouts, Cancellation, and Leases
• Pause and Resume Runs