Pause and Resume Runs

Pause a run to stop its work for a short time. For example, pause it while a dependency of the run is in maintenance. Resume continues the run from the point where it stopped.

run := client.OpenRun(savedID)
if err := run.Pause(ctx); err != nil {
    return err
}

// Do the maintenance.

if err := run.Resume(ctx); err != nil {
    return err
}

Pause and Resume are methods of cyclone.RunHandle and cyclone.ResultRun. A handle from workflow.Open, step.Open, or action.Open checks the run identity before each call. The handle does the same check for Cancel. A run of a different workflow or kind returns ErrRunMismatch. The call does not change that run. There is no package function for a pause. There is no Client method for a pause.

The CLI has the same two operations:

cyclone pause 42
cyclone resume 42
cyclone runs --state paused

What a Pause Stops

After Pause returns, the run starts no new work.

• No worker claims a step of the run.
• A pending step does not become claimable. This is also true when its last dependency completes.
• No wait completes.
• Wait timers stop.
• WithMaxWait limits stop.
• The run deadline from WithRunDeadline stops.

Pause moves each available step of the run to the step state paused (StepPaused). Pause also moves each waiting step of the run to paused. Pause also moves each waiting child node to paused. The claim query does not read that state. The timer, signal, and child scanners do not read that state. A run that is not paused pays no extra cost on the claim path.

What Continues During a Pause

A step that is in progress at the pause continues.

• Its heartbeat stays in effect. Its lease stays in effect. Its attempt timeout from WithTimeout also stays in effect.
• When the step completes, Cyclone stores its output. Its dependent steps start after Resume.
• When the step fails and has attempts left, Cyclone charges the attempt. The retry waits for Resume. This is also true when the backoff is zero.
• When its worker stops, the recovery sweep reclaims the step. The step then waits for Resume.

The step event of a retry, a snooze, or a reclaim during the pause reports the state available. The same transaction then moves the step to paused.

Signal continues to accept signals. See Signals.

A Step Can End a Paused Run

A pause stops new work. A pause does not keep a run that has no work left. A pause does not hide a failure.

• When the in-progress step completes the last node, the run completes. Wait returns. Result returns. No work remains for Resume to start. A paused state would make a caller wait for an operator who has nothing to do.
• When a step uses every attempt and that failure fails the run, the run fails. The run fails the same way with no pause. Cyclone cancels the paused steps. Cyclone cancels the pending steps. RetryAll opens the run again as a run in state running. RetryAll does not open it as a paused run.
• When OnDepFailure(CancelBranch) contains the failure, Cyclone cancels that branch. The run stays paused while other steps remain.

A paused run can become completed, failed, or cancelled without Resume. Resume on such a run returns ErrNotPaused.

Signals

Signal on a paused run returns nil. Signal stores the signal in the mailbox of the run. The wait does not complete during the pause. The mailbox keeps only the signals that the wait can use. WaitSignal keeps one signal. WaitAny keeps one signal. WaitN keeps the rest of the quorum. A WaitN keeps the payloads that it gathered before the pause.

After Resume, the wait scanner delivers the stored signals in arrival order. A signal that no wait declares still returns ErrSignalNotDeclared.

Timers and the Run Deadline

Each wait timer keeps the time that it had left at the pause. The run deadline keeps that remaining time too. Resume moves the scheduled_at of each timed wait forward by the length of the pause. Resume moves the deadline_at of the run forward by the same length. For example, pause a one-hour WaitTimer after 10 minutes. The timer completes 50 minutes after Resume. The length of the pause does not change that remaining time.

A retry delay is a clock time, not a wait timer. WithProcessIn is a clock time. WithProcessAt is a clock time. These clock times do not stop. A step whose time passes during the pause becomes claimable at Resume.

Unique Keys and Sequences

A paused step is still active work.

• The step keeps its WithUnique key. An enqueue with the same key returns ErrDuplicate.
• The step keeps its place in its WithSequence key. A later job with the same key waits until you resume the paused run, or until you cancel it.

Fan-Out Nodes

A pause also holds the item tasks of a fan-out node.

• Pause moves the available item tasks to paused. A pending item task stays pending. An item task that is in progress continues.
• When the upstream step completes during the pause, the engine expands the fan-out node in the same transaction. This is the same as with no pause. The item tasks that the WithMaxParallel limit permits are paused. Those item tasks are not available.
• When an item task completes during the pause, the engine releases no pending item task. When an item task fails and has attempts left, its retry waits for Resume.
• A paused item task keeps its place in the WithMaxParallel limit. Resume makes the paused item tasks available. Resume then releases pending item tasks up to the limit. a pause never lets more than n item tasks be in progress at the same time.
• A paused item task has no output. the fan-out node stays pending. When the last item task completes during the pause, the node completes. The steps after that node start after Resume.
• When an item task is discarded during the pause, the engine discards the fan-out node. The engine cancels the paused item tasks of that node. The engine cancels the pending item tasks of that node. This is the same as with no pause. Then OnDepFailure applies. When the run fails, RetryAll opens it again as a run in state running.
• An empty list completes the fan-out node during the pause. A list that fails a fan-out limit discards the node during the pause.

Cancel cancels the fan-out node and all of its item tasks. Cancel also cancels the paused item tasks.

Errors

Call	Run state	Result
Pause	running	The run is paused
Pause	paused	ErrAlreadyPaused, no change
Pause	completed, failed, or cancelled	ErrAlreadyTerminal, no change
Resume	paused	The run is running
Resume	running, completed, failed, or cancelled	ErrNotPaused, no change
Cancel	paused	The run and its paused steps are cancelled
Retry, RetryAll	paused	ErrNotFailed, no change
Any	no such run	ErrRunNotFound

The errors match with errors.Is. The message names the run.

Read the Paused State

run.Status reports RunPaused. client.GetRun reports RunPaused. FinishedAt is nil. client.ListRuns with RunQuery{State: "paused"} lists the paused runs. The steps that the pause stopped report StepPaused.

A paused run is not terminal. RunPaused.Terminal() is false. Wait continues to wait. Result continues to wait. Output returns ErrNotCompleted.

Retention does not delete a paused run, because the run is not terminal. A run that stays paused stays in the database. This query finds a long pause:

SELECT id, workflow, paused_at
  FROM cyclone_runs
 WHERE state = 'paused'
   AND paused_at < now() - interval '1 hour'
 ORDER BY paused_at;

Child Runs and Compensation

A pause applies to one run. A run that a step submits with EnqueueCompletion or SubmitCompletion is a separate run. The pause does not reach that run.

A pause does not reach the child runs of the run either.

• Pause moves each waiting child node to paused. No child run starts for that node. Its WithMaxWait limit stops.
• A child run that is in progress at the pause continues. To stop that child run too, pause it with its own handle, for example client.OpenRun(childID).Pause(ctx). StepStatus.ChildRunID gives its ID.
• When a child run ends during the pause, the parent node keeps the result in the child run. The parent node does not move until Resume. Other waits behave the same way. Resume then completes the node, or Resume discards the node.
• A pause of a child run does not pause its parent run. The parent node waits for the child run. The parent node does the same for a slow child run.

Cancel on a paused parent run also cancels its active child runs.

Compensation (#45) does not exist yet. When compensation exists, a paused run must not start compensation work.

Deploy Before You Pause

Apply the pause migration. Then deploy the new build to every worker before you pause a run. A worker of an older build does not know the paused state. That worker can start the dependent steps of a paused run. That worker cannot complete a paused run. That worker cannot fail a paused run.

A producer of an older build reads a paused run as terminal. A CLI of an older build does the same. Its Wait returns. Its Signal returns ErrAlreadyTerminal. Its Cancel returns ErrAlreadyTerminal. Those calls change nothing. Upgrade these processes too. See Migrations.

Test a Pause

cyclonetest.Env.AdvanceTime moves a paused run with the clock. A test can then check that the timers keep their remaining time.

run, _ := wf.Submit(ctx, env.Client(), in, cyclone.WithRunDeadline(time.Hour))
_ = run.Pause(ctx)
env.AdvanceTime(2 * time.Hour)   // the deadline does not fire during the pause
_ = run.Resume(ctx)              // about one hour remains
env.AdvanceTime(2 * time.Hour)
st := env.RunUntilComplete(run.ID())
// st.State == cyclone.RunFailed, st.ErrorKind == "deadline"

RunUntilComplete waits for a terminal state. For a paused run, RunUntilComplete fails the test at its timeout. ElapseWait on a wait of a paused run removes the time that the wait has left. The wait lapses at Resume.