Failure Recovery

How to respond to worker crashes, Postgres restarts, stuck steps, and discarded jobs.

Worker Crash

What happens automatically:

1. The worker process stops. All goroutines stop.
2. Postgres ends the connections from the crashed process. Postgres rolls back the in-flight transactions.
3. Heartbeats stop. lease_expiry passes on all steps that the crashed worker owned.
4. The recovery sweep on any live client detects state = 'running' AND lease_expiry < now().
5. The sweep reclaims the steps. The state becomes available. The sweep clears the lease and does not change scheduled_at. The same attempt starts again, immediately, with no backoff.

Recovery is automatic. No operator action is needed if other clients are up and the crash was transient. Crash Loop covers a crash that the handler itself causes.

Monitor the cyclone_step_events_total{type="reclaimed"} rate. You define that counter from WithOnStepEvent. Cyclone exports no Prometheus metrics itself. A spike indicates a worker crash. A sustained high rate indicates repeated crashes or severe overload.

Operator action if workers are not recovering:

# Check for stuck steps (running + lease expired):
SELECT * FROM cyclone_v_steps WHERE lease_expired;

# If no live clients are running, steps won't recover until a client starts.
# Start a client to trigger the recovery sweep.

Postgres Restart

What happens:

1. Postgres ends all client connections.
2. pgxpool attempts to connect again automatically.
3. Postgres rolls back in-flight completion transactions.
4. Steps that were running stay running in the database. The rollback does not change their state.
5. After the pool connects again, the recovery sweep finds expired leases and reclaims them.

Recovery is automatic once the pool connects again. The time to recovery is leaseTTL plus reconnect time plus one sweep tick. The sweep ticks on the poll interval.

Monitor pgxpool connection errors in the logs. Also monitor a cyclone_step_events_total{type="reclaimed"} spike after the restart.

Network Partition

From Cyclone's perspective, a network partition between the application and Postgres looks the same as a Postgres restart. Postgres ends the connections. The pool connects again when the partition heals. Lease recovery then proceeds.

This case needs no special handling.

Handler Panic

The worker goroutine catches panics inside handler functions:

defer func() {
    if r := recover(); r != nil {
        err = &panicError{value: r, stack: debug.Stack()}
    }
}()

Cyclone stores the panic value in cyclone_steps.error as a compact one-liner (handler panicked: <v>). Cyclone writes the full goroutine stack to cyclone_steps.panic_stack (migration 005). Cyclone does not append the stack to the error text. The error stays readable, and you can query the stack:

SELECT node, kind, error, panic_stack
FROM cyclone_v_steps
WHERE run_id = $runID AND panic_stack IS NOT NULL;

The CLI also surfaces the panic stack:

cyclone steps <runID>
cyclone steps --format json <runID>

PANIC_STACK appears only under --format json, where it carries the captured goroutine stack when the step panicked.

Cyclone treats the panic as a retryable error and increments the attempt. After backoff, a worker claims the step again and the handler starts again.

Operator action: find the root cause in the panic stack. Fix the handler code. If the step is now discarded (attempts exhausted), retry:

cyclone retry <runID> <node>
# or via Go:
client.Retry(ctx, runID, node, cyclone.RetryAdditionalAttempts(3))

Stuck Running Step

A step with state = 'running' and lease_expiry < now() is stuck. cyclone_v_steps.lease_expired shows this. Its worker likely crashed, or the connection was lost.

Check:

SELECT * FROM cyclone_v_steps WHERE lease_expired;

The recovery sweep handles this automatically. If a Client.Start is active, the sweep reclaims the step on the next poll tick.

If no live client is up, start one to trigger the sweep. Or force the reclaim manually:

-- Only if no live workers are running and you're certain the worker is dead:
UPDATE cyclone_steps
SET state = 'available', lease_expiry = NULL, lease_owner = NULL,
    scheduled_at = now()
WHERE state = 'running' AND lease_expiry < now()
  AND run_id = $run_id AND node = $node;

Stranded Run

A run can stay running with no available step, no running step, and no waiting step. At the same time, every dependency of at least one pending step has completed. Nothing in the engine reports that run. The recovery sweep only selects running steps. The wait scanner only selects waiting_* steps. The deadline scanner only selects runs with a deadline_at. A pending step is nobody's responsibility. run_has_active_steps counts pending as active. The run looks alive. RetryAll refuses because the run is not failed.

This state means an engine invariant broke. When every dependency of a step is completed, promote_ready_steps must make that step available. One known cause is a Postgres server whose default_transaction_isolation is repeatable read. The last sibling completion then keeps a snapshot from before its predecessor committed. NewClient now refuses that setting. A manual UPDATE, a partially applied migration, or a future bug can still get there.

Check:

cyclone repair stranded-runs --dry-run

Repair:

cyclone repair stranded-runs           # default batch of 100
cyclone repair stranded-runs --batch 500

Each repair uses its own transaction. The repair locks the run row and does the normal advance. The advance promotes the steps that must already be available. If nothing is left to do, the advance completes the run or fails it. Repair of one run never blocks another. The command is safe on a live database. Client.RepairStrandedRuns(ctx, batch) is the same operation from Go.

Every repair is logged at WARN with the run id. Investigate the cause. The run moves again, but something put it in that state:

1. Check SHOW default_transaction_isolation on the server and on the application role. It must be read committed.
2. Check for manual UPDATEs against cyclone_steps in your audit log or pg_stat_statements.
3. Check that every migration applied cleanly (SELECT * FROM cyclone_schema_version).

Client.Stats reports the count as Health.StrandedRuns. Alert on it. See Monitoring and Alerting for the query.

A paused run is not stranded. The check selects only running runs. A paused run waits for Resume by design.

Expired Lease (Normal Recovery)

The recovery sweep handles this automatically. Handler duration is not a cause of reclaims. Each running step has its own heartbeat goroutine. The goroutine renews the lease every leaseTTL/3. A long handler keeps its lease. A high reclaim rate means renewals are not reaching Postgres, or the step's own WithTimeout elapsed. Investigate:

1. Check for workers that crash or that the OOM killer stops. Each restart strands every lease that the worker held.
2. Check that workers can reach Postgres. A renewal that errors is retried on the next tick. A sustained outage lets the lease lapse.
3. Check for steps that hit WithTimeout. Renewal stops at started_at + timeout. The sweep then takes the step. That event is a timeout, not a crash. It shows up as timed_out, not reclaimed.

Crash Loop

A handler can stop its worker on every execution. It can exhaust memory, trip a runtime fatal error, or call os.Exit. The next live client reclaims that step and starts it again. A reclaim charges no attempt. Without a bound, this loop never ends. The step never reaches discarded. Every worker that takes the step stops.

Symptoms: a sustained reclaimed rate for one kind. The reclaims count on one step rises. Workers restart in step with that count.

SELECT run_id, node, kind, reclaims, attempt, lease_expired
FROM cyclone_v_steps
WHERE reclaims > 0
ORDER BY reclaims DESC
LIMIT 20;

Prevent it with WithMaxReclaims(n) on every client that calls Start. The expiry after the nth reclaim discards the step with error_kind = "reclaim_limit". The discard fails the run like any other dead letter. The discard stops the loop:

cyclone dlq list --reason reclaim_limit

Fix the handler before you retry. cyclone retry, client.Retry, and RetryAll reset reclaims to 0. A retried step that still crashes its worker gets the full bound again. Cyclone discards the step only after that bound.

Discarded Nodes

A step in discarded state with a run in failed state requires operator intervention.

Diagnose:

cyclone steps <runID>
# or:
SELECT node, kind, error, attempt, max_attempts
FROM cyclone_steps
WHERE run_id = $runID AND state = 'discarded';

Fix the root cause (infrastructure issue, bad input, code bug).

Retry:

# Single node:
cyclone retry <runID> <node>

# All discarded nodes in a run:
cyclone retry <runID> --all

// With Go:
client.Retry(ctx, runID, "send_email",
    cyclone.RetryAdditionalAttempts(3),
    cyclone.RetryReason("retry after SMTP fix"),
)
client.RetryAll(ctx, runID,
    cyclone.RetryAdditionalAttempts(5),
    cyclone.RetryReason("bulk retry after infra recovery"),
)

Retrying Safely

A retry reuses the same (runID, node) pair. Your ON CONFLICT (run_id, node) DO NOTHING deduplication works across operator retries. External idempotency keys work across operator retries too.

A retry of a parent run reuses the child run of each child node that it reopens. A failed or cancelled child run starts again, with its run ID. A retry never starts a second child run for a node. When a child run completed with no result for its node, the retry returns ErrChildNotRetryable and changes nothing. RetryExhausted skips such a run and does not count it. Retry the parent run, not the child run.

Before you retry a batch of failed runs:

1. Confirm that the root cause is fixed.
2. Confirm that the handler is idempotent for the affected inputs.
3. Add RetryAdditionalAttempts if the original max_attempts was exhausted.

Dead-Letter Queue (Exhausted Steps)

Cyclone has no separate dead-letter table. The dead-letter queue is a view over discarded steps in cyclone_steps (attempts used up). The view groups those steps by kind and failure reason.

List exhausted steps:

cyclone dlq list
cyclone dlq list --kind send_email
cyclone dlq list --reason permanent
cyclone dlq list --since 24h --limit 50

steps, err := client.ListExhausted(ctx, cyclone.ExhaustedQuery{
    Kind:   "send_email",
    Reason: "retryable",
    Since:  24 * time.Hour,
    Limit:  100,
})
for _, s := range steps {
    fmt.Printf("run %s / %s: %s (%d/%d attempts)\n",
        s.RunID, s.Node, s.Error, s.Attempts, s.MaxAttempts)
}

Reason filters by failure classification. "permanent" selects Permanent(err). "retryable" selects retries that are exhausted. "deadline" selects a WithRunDeadline cutoff. Leave Reason empty for all three.

Bulk requeue:

cyclone dlq retry --kind send_email --since 24h --attempts 3

count, err := client.RetryExhausted(ctx, cyclone.ExhaustedQuery{
    Kind:  "send_email",
    Since: 24 * time.Hour,
}, cyclone.RetryAdditionalAttempts(3))
fmt.Printf("requeued %d runs\n", count)

RetryExhausted reopens each affected failed run with RetryAll. Retry operates at run granularity. A matching dead step reopens its whole run. The reopen includes sibling discarded nodes that the filter did not match.

Pausing a Run During an Outage

When a dependency of a run is down or in maintenance, pause the run instead of cancelling it. The run keeps its state, its outputs, and the time left on its timers and deadline.

cyclone pause 42      # no new step of run 42 starts
# Repair the dependency.
cyclone resume 42     # the run continues where it stopped

A step that is in progress at the pause continues. If it fails during the pause, its retry waits for Resume. A step that exhausts its attempts still fails the run. Resume on a run that ended during the pause returns ErrNotPaused. The CLI reports that error. See Pause and Resume Runs.

Cancelling Safely

client.Cancel is idempotent. Calling it on an already-terminal run returns ErrAlreadyTerminal.

err := client.Cancel(ctx, runID)
if errors.Is(err, cyclone.ErrAlreadyTerminal) {
    // Already done; ignore
}

Cancellation sets non-terminal steps to cancelled and the run to cancelled. Cancellation cancels the context of in-flight handlers. Write handlers that observe ctx.Done() and return. Cancel also cancels a paused run and its paused steps.

Related

• Retries, Leases, and Recovery
• Retries and Permanent Failures
• Timeouts, Cancellation, and Leases
• CLI