Timeouts, Cancellation, and Leases

How per-attempt timeouts, graceful cancellation, and lease-based crash recovery work together.

Per-Attempt Timeout with WithTimeout

WithTimeout caps the wall-clock time of one handler invocation:

var ImportStep = cyclone.NewStep("import", importFn,
    cyclone.WithTimeout(5*time.Minute),
)

// Override per-enqueue:
cyclone.Enqueue(ctx, client, importStep, in, cyclone.WithTimeout(10*time.Minute))

The per-enqueue value wins when both are set.

How Timeout Is Measured

Cyclone enforces a timeout twice. It uses two different clocks.

Enforcement	Clock	Effect
The handler's context deadline	the worker's own clock, started when it begins the attempt	cancels ctx with context.DeadlineExceeded so a cooperative handler returns
started_at + timeout on the step row	the database clock (now())	stops lease renewal and lets the recovery sweep move the step on, cooperation or not

The database clock is the authority. The claim writes started_at as now(). The heartbeat and the recovery sweep both compare against now() in the same database. No worker system clock can shorten a step's durable timeout. No worker system clock can extend it. Cyclone stores timeout as an INTERVAL on the step row. It is not an absolute deadline column.

Handler Context Cancellation

When the deadline passes, Cyclone cancels the handler's context.Context with context.DeadlineExceeded. A cooperative handler checks ctx.Done() and returns:

var LongRunningStep = cyclone.NewStep("long_running",
    func(ctx cyclone.Context, in Input) (Output, error) {
        for {
            select {
            case <-ctx.Done():
                return Output{}, ctx.Err() // returns context.DeadlineExceeded
            default:
            }
            // ... do a unit of work
        }
    },
)

Cyclone treats a timed-out attempt as a retryable error. The attempt counter increments. The step goes back to available after backoff. Or the step goes to discarded when attempts are exhausted. The deadline stays authoritative when the handler sees the cancellation and returns nil.

The handler ctx comes from the context passed to Start. Cancelling that context cancels every in-flight handler. Only Shutdown() with a live Start context leaves handlers executing. See Graceful Shutdown.

Non-Cooperative Handlers

Cyclone cannot kill a goroutine. A handler that ignores ctx.Done() keeps executing. The step moves on without that handler:

1. After started_at + timeout passes on the database clock, the heartbeat stops renewing the lease.
2. The recovery sweep selects the step. The step matches an elapsed timeout, not only an expired lease. The sweep consumes the attempt. It sets the step to available with backoff, unless the attempts are exhausted. In that case it sets the step to discarded.
3. The abandoned goroutine returns later. Its completion no longer matches a running row of the same lease generation. The CAS guard discards that completion.

The durable transition lands about one sweep tick after the timeout elapses. The sweep ticks on the poll interval (WithPollInterval, default [100ms, 1s)). This delay has nothing to do with the lease TTL.

Cooperative Cancellation

client.Cancel(ctx, runID) sets the run to cancelled. It also sets every non-terminal step to cancelled. Those states are pending, available, running, waiting_signal, and waiting_timer. Cyclone does this in one transaction. The state change is immediate. Nothing waits for a sweep.

Cancel also cancels a paused run. Its paused steps become cancelled in the same transaction.

An in-flight handler learns about the cancel on its next heartbeat. The renewal no longer matches a running row. The worker then cancels the handler context. A handler that checks ctx.Done() exits promptly:

// In the handler:
select {
case <-ctx.Done():
    return Output{}, ctx.Err() // cancelled by operator
default:
}

A handler that ignores ctx.Done() continues until it returns. The same CAS guard drops its result. The step is no longer running. A late completion fails the CAS.

Lease TTL

The lease TTL is crash-detection latency. It is how long a step owned by a dead process sits running before a live client can reclaim it.

// Set globally for all workers on this client:
client, _ := cyclone.NewClient(pool, mux,
    cyclone.WithWorkers(4),
    cyclone.WithLeaseTTL(2*time.Minute), // default: 30s
)

Size the TTL by how fast you want the steps of a crashed worker back. Do not size it by handler duration. A handler that executes for an hour is fine on the default 30-second TTL. The heartbeat keeps renewing its lease while the process is alive. A short TTL recovers faster and writes more heartbeats. A long TTL does the reverse. A long TTL leaves steps stranded longer after a crash.

Heartbeats

Every claimed step gets its own heartbeat goroutine. The goroutine ticks every leaseTTL/3. It uses a random initial phase and ±10% jitter. Concurrent workers do not renew in lockstep. The cadence comes from the TTL alone. It has no relation to WithPollInterval.

UPDATE cyclone_steps
 SET lease_expiry = now() + ($1::bigint * interval '1 microsecond'),
 heartbeat_at = now()
 WHERE id = $2
 AND state = 'running'
 AND lease_owner = $3
 AND lease_generation = $4
 AND (timeout IS NULL OR started_at + timeout > now());

Three things follow from that WHERE clause:

• One renewal per step, addressed by id. There is no bulk update of every step that a client owns.
• Cyclone stamps lease_generation at claim time. A renewal matches nothing after another worker reclaims the step and claims it again.
• The started_at + timeout guard stops renewals after the step timeout elapses. That stop lets the recovery sweep take the step from a wedged handler.

When a renewal matches zero rows, the worker cancels the handler context. The row can be completed, cancelled, reclaimed, or timed out. A duplicate execution then stops promptly.

What Happens When a Worker Crashes

1. The worker process dies.
2. Its heartbeat goroutines die with it. Nothing pushes lease_expiry forward again.
3. lease_expiry passes.
4. The recovery sweep on any live client detects state = 'running' AND lease_expiry < now().
5. Under the run row lock and the step row lock, the sweep checks expiry again. It then reclaims the step. It sets state = 'available'. It clears lease_owner and lease_expiry. It leaves scheduled_at untouched.
6. Any worker that serves the queue claims the step on its next poll. The handler starts again with the same attempt number.

A crash is a lease expiry, not a handler failure. A reclaim does not increment attempt. A reclaim does not apply backoff. The step is claimable the instant the sweep commits. The worst case from crash to re-availability is leaseTTL plus one sweep tick.

Each reclaim increments reclaims (StepStatus.Reclaims). With WithMaxReclaims(n), the expiry after the nth reclaim discards the step. It does not make the step available. The step gets error_kind = "reclaim_limit". A handler that crashes its worker every time cannot loop forever. See Bounding Crash Loops.

Expired Lease Recovery

The recovery sweep executes automatically inside every Client.Start loop. You do not need to configure it. It ticks on the poll interval. It takes at most 100 candidates per pass.

To trigger recovery by hand in a test:

env.ExpireLease(runID, "my-node") // sets lease_expiry to the past
// The sweep will reclaim the step on the next tick.

Difference Between Timeout, Cancellation, Discard, and Crash Reclaim

Event	Attempt consumed?	Step next state	Error stored?
Timeout (WithTimeout exceeded)	Yes	available after backoff, or discarded when exhausted	Yes
Handler returns retryable error	Yes	available after backoff, or discarded when exhausted	Yes
Handler returns Permanent(err)	Yes (the last one)	discarded	Yes
client.Cancel	No	cancelled	No
Crash (lease expires, reclaim)	No	available, claimable immediately	No
Crash past WithMaxReclaims	No	discarded, error_kind = "reclaim_limit"	Yes
client.Retry (discard → reopen)	No. The retry raises max_attempts instead	available	Cleared

Testing Timeout and Lease Expiry

Use cyclonetest options to control timing:

// Short lease for faster tests:
env := cyclonetest.New(t,
    cyclonetest.WithWorkers(2),
    cyclonetest.WithLeaseTTL(200*time.Millisecond),
)

// Manually expire a lease to test recovery:
env.ExpireLease(runID, "slow_node")
// Wait for recovery to run and the step to restart:
env.RunUntilComplete(runID)

A short TTL does not shorten a handler. The heartbeat renews every leaseTTL/3. A handler that outlives several TTLs still executes exactly once.

Test the timeout response of a handler:

var slowStep = cyclone.NewStep("slow", func(ctx cyclone.Context, in Input) (Output, error) {
    select {
    case <-ctx.Done():
        return Output{}, ctx.Err()
    case <-time.After(30 * time.Second):
        return Output{Done: true}, nil
    }
}, cyclone.WithTimeout(100*time.Millisecond))

env := cyclonetest.New(t, cyclonetest.WithWorkers(1))
env.Mux().Register(slowStep)

runID, _ := cyclone.Enqueue(context.Background(), env.Client(), slowStep, Input{})
run := env.RunUntilComplete(runID)
// Handler should have been cancelled by timeout; step retried until discarded.

Related

• Retries, Leases, and Recovery
• Retries and Permanent Failures
• Graceful Shutdown
• States