Graceful Shutdown

How to stop Cyclone workers cleanly. Let in-flight handlers complete before the process exits.

Why Shutdown Matters

A Cyclone worker holds leases on in-flight steps. If the process exits without a drain, the leases expire. The recovery sweep then reclaims the steps. That delay is the lease TTL. During that window, the steps appear stuck. Graceful shutdown avoids that delay. It gives in-flight handlers a chance to complete normally.

Use a Signal Context

The usual Go pattern is signal.NotifyContext. It cancels a context on SIGINT or SIGTERM:

package main

import (
    "context"
    "log"
    "os"
    "os/signal"
    "syscall"
    "time"

    "github.com/jackc/pgx/v5/pgxpool"
    "github.com/wafer-labs/cyclone"
)

func main() {
    sigCtx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()

    runCtx := context.Background()

    pool, _ := pgxpool.New(runCtx, os.Getenv("DATABASE_URL"))
    defer pool.Close()

    if err := cyclone.Migrate(runCtx, pool); err != nil {
        log.Fatal(err)
    }

    mux := cyclone.NewRegistry()
    // ... register handlers ...

    client, _ := cyclone.NewClient(pool, mux, cyclone.WithWorkers(8))

    // Every handler context descends from the one passed to Start, so Start gets a
    // context that outlives the signal. Shutdown, not cancellation, does the draining.
    go func() {
        if err := client.Start(runCtx); err != nil {
            log.Printf("client stopped: %v", err)
        }
    }()

    // Wait for signal.
    <-sigCtx.Done()
    stop() // release the signal listener

    // Graceful shutdown with a timeout.
    shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    if err := client.Shutdown(shutdownCtx); err != nil {
        log.Printf("shutdown: %v", err)
    }
}

Stop Accepting Work

Shutdown stops the worker loops. Cancelling the context of Start also stops the worker loops. The two calls are not interchangeable:

• client.Shutdown(ctx) tells the workers to stop claiming. It then blocks until every in-flight handler has returned. Shutdown does not cancel handler contexts. The handler stops when it returns. If it does not return, its own WithTimeout deadline stops it. Cyclone then commits that outcome in the normal way.
• Cancel the context that you passed to Start. Cyclone then cancels every in-flight handler at the same time. Each handler context comes from that context. A cooperative handler aborts mid-attempt. Cyclone still records that failure before Start returns. A handler that finished anyway still has its completion recorded before Start returns.

Keep the signal context out of Start. Use the signal context to decide when to shut down. Pass Start a context that outlives the signal, as in the example above. Both paths record the same bookkeeping. Only Shutdown lets handlers finish their current work.

Results are recorded either way

The result write of the step does not use the context that you pass to Start. The write uses a context detached from that one. The detached context drops cancellation. It keeps values such as trace ids. Start does not return until those writes finish. The heartbeat of the step stays alive for the same window. The lease cannot lapse under a slow commit.

Cyclone bounds the write at 30 seconds. Do not use that limit as a time budget. It exists only so a wedged database cannot hold shutdown open forever. Your WithErrorHandler and WithOnStepEvent hooks execute outside that limit. A slow reporter cannot consume it.

This behavior was different before. Cancelling the context of Start made the write fail immediately with context canceled. A step whose handler had finished stayed running with a live lease. Nothing in that process touched the step again. Another worker's recovery sweep reclaimed it one leaseTTL later. The handler executed a second time. The second execution applied effects outside a CompletionTx again. Those effects happened twice. If you pin this behavior in tests, cancel the context of Start. Then wait for Start, or call Shutdown, before you assert on step state.

Wait for In-Flight Handlers

client.Shutdown(shutdownCtx) blocks until every in-flight handler goroutine exits:

shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
if err := client.Shutdown(shutdownCtx); err != nil {
    log.Printf("shutdown did not complete cleanly: %v", err)
    // Process will exit; in-flight steps will be reclaimed after lease expiry.
}

Shutdown returns nil when every goroutine exits before the deadline. It returns an error when the deadline passes first. That error is context deadline exceeded. Goroutines are still active in that case.

Shutdown Timeout

Set the shutdown timeout to slightly less than the termination grace period of your orchestrator:

Orchestrator	Setting	Recommendation
Kubernetes	terminationGracePeriodSeconds	Set terminationGracePeriodSeconds to max_handler_duration + 10s. Set the shutdown timeout to terminationGracePeriodSeconds - 5s.
systemd	TimeoutStopSec	Same logic
Docker	docker stop --time	Same logic

If your longest handler takes up to 5 minutes, set terminationGracePeriodSeconds: 310. Pass a 305-second timeout to Shutdown.

Kubernetes Example

spec:
  terminationGracePeriodSeconds: 310
  containers:
    - name: worker
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 5"] # let the SIGTERM reach the app first

The process receives SIGTERM. The signal context fires. client.Shutdown stops the workers from claiming new work. It drains in-flight work within 305 seconds.

systemd Example

[Service]
ExecStart=/usr/local/bin/my-worker
TimeoutStopSec=310
KillSignal=SIGTERM

Non-Cooperative Handlers

A handler that does not check ctx.Done() continues until it returns on its own. If the shutdown deadline expires before it finishes:

• client.Shutdown returns with a deadline exceeded error.
• The process exits after your main function returns.
• The lease of the step expires.
• The recovery sweep on another client reclaims the step.
• The handler starts again on the next worker that claims it.

This is safe because of lease recovery and idempotent handlers. It is inefficient. Write handlers that respect cancellation.

Relationship to Leases

Shutdown gives handlers a chance to finish normally and to commit their completion transactions. If shutdown is unclean, the process was killed. Leases are the safety net. The recovery sweep reclaims the step after leaseTTL passes. The handler starts again. A reclaim does not increment the attempt counter.

A long lease TTL means a longer reclaim delay after an unclean shutdown. A short lease TTL means faster recovery and more heartbeat traffic. The default (30 seconds) is a reasonable balance.

Common Mistakes

Not calling Shutdown at all:

// WRONG: process exits immediately after signal; in-flight handlers killed
<-sigCtx.Done()
os.Exit(0)

Calling Shutdown with a zero timeout:

// WRONG: deadline already expired; Shutdown returns immediately
shutdownCtx, cancel := context.WithTimeout(context.Background(), 0)
defer cancel()
client.Shutdown(shutdownCtx) // noop

Calling Shutdown with the signal context after it is already cancelled:

// WRONG: sigCtx is already cancelled; Shutdown returns deadline exceeded immediately
<-sigCtx.Done()
client.Shutdown(sigCtx) // already done, deadline already passed

Always create a fresh context for shutdown.

Passing the signal context to Start:

// WRONG: on SIGTERM this cancels every in-flight handler before Shutdown can drain them
go client.Start(sigCtx)
<-sigCtx.Done()
client.Shutdown(shutdownCtx)

Handler contexts descend from the context of Start. Cancelling the context of Start cancels in-flight handlers. Give Start a context that outlives the signal.

Related

• Retries, Leases, and Recovery
• Production Checklist
• Timeouts, Cancellation, and Leases