Production Checklist

Pre-launch checklist for deploying Cyclone to production.

Before Launch

Run migrations

Call cyclone.Migrate(ctx, pool) on startup. Verify it completed without error:

if err := cyclone.Migrate(ctx, pool); err != nil {
    log.Fatalf("migration failed: %v", err)
}

Verify in Postgres:

SELECT * FROM cyclone_schema_version ORDER BY version;

• [ ] cyclone.Migrate runs on every startup with no errors
• [ ] cyclone_schema_version contains expected version rows
• [ ] DB user has sufficient privileges (CREATE TABLE, SELECT/INSERT/UPDATE/DELETE)

Configure pool size

The pool must satisfy MaxConns >= txSlots + 6. By default, txSlots equals the total worker count.

For 8 workers with default txSlots=8:

MaxConns >= 8 + 6 = 14

Set it in the DSN or with pgxpool.Config:

DATABASE_URL='postgres://...?pool_max_conns=25'

• [ ] Pool MaxConns is correctly sized for your worker count
• [ ] NewClient does not return a pool size validation error on startup

Verify transaction isolation

Cyclone requires default_transaction_isolation = 'read committed'. Two separate things depend on it, and they fail in different ways.

The connection default governs claims, heartbeats, and wait scans. Those are single statements with no surrounding BEGIN. They execute as implicit transactions at the connection default. On a server, database, or role set to repeatable read, they abort with SQLSTATE 40001 as soon as two workers contend for the same rows. The work is retried instead of done.

Transactions that Cyclone opens itself govern completions. Cyclone starts every one of them at read committed, including the handler's ctx.CompletionTx(). A completion keeps a correct snapshot even where the connection default is wrong. A completion that escaped that pin would take its snapshot before its sibling committed. It would not see the join's dependencies as all completed. It would commit anyway. The join would stay pending forever. No error is logged.

Check the effective value for your application role:

SHOW default_transaction_isolation;

Pin it on the pool regardless of the server default:

cfg, _ := pgxpool.ParseConfig(dsn)
cfg.ConnConfig.RuntimeParams["default_transaction_isolation"] = "read committed"
pool, _ := pgxpool.NewWithConfig(ctx, cfg)

Or reset the role/database default:

ALTER ROLE app RESET default_transaction_isolation;

NewClient reads the setting back from a live connection at startup and refuses to start when it is anything but read committed. A database that is unreachable at that moment only logs a warning. The check does not replace the boxes below.

• [ ] SHOW default_transaction_isolation returns read committed for the application role
• [ ] Pool pins RuntimeParams["default_transaction_isolation"], or the role/database default is read committed

Configure queues and workers

client, _ := cyclone.NewClient(pool, mux,
    cyclone.WithWorkerQueues(map[string]int{
        "default":  8,
        "bulk":     2,
    }),
)

• [ ] All queues referenced in step definitions have workers configured
• [ ] Worker count is appropriate for your workload (see Queue Sizing)

Bound crash recovery

A reclaim after an expired lease charges no attempt. A handler that crashes its worker on every execution loops forever unless reclaims are bounded:

client, _ := cyclone.NewClient(pool, mux,
    cyclone.WithMaxReclaims(3), // discard with error kind reclaim_limit on the next expiry
)

• [ ] WithMaxReclaims set to the same value on every client that runs Start
• [ ] The value leaves room for the reclaims a healthy step can see during a deploy or a database failover

Configure logging

logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
    Level: slog.LevelInfo, // suppress DEBUG in production
}))
client, _ := cyclone.NewClient(pool, mux,
    cyclone.WithLogger(logger),
    // ...
)

• [ ] Logger set to INFO or WARN in production
• [ ] Logs include run_id, node, kind fields for correlation
• [ ] Log pipeline configured to index structured fields

Configure metrics

client, _ := cyclone.NewClient(pool, mux,
    cyclone.WithOnStepEvent(func(e cyclone.StepEvent) {
        stepEventsTotal.With(prometheus.Labels{
            "type":  string(e.Type),
            "state": string(e.State),
            "kind":  e.Kind,
            "queue": e.Queue,
        }).Inc()
    }),
)
go updateQueueGauges(ctx, client) // periodic Stats() call

• [ ] WithOnStepEvent connected to your metrics system
• [ ] Queue depth gauges published with periodic client.Stats() calls
• [ ] Dashboards created for queue depth, discard rate, lease reclaims

Configure retention

Without retention configuration, terminal runs accumulate indefinitely.

client, _ := cyclone.NewClient(pool, mux,
    cyclone.WithRetentionSweep(cyclone.RetentionConfig{
        DefaultAge: 30 * 24 * time.Hour,
        Interval:   10 * time.Minute,
        BatchSize:  1000,
    }),
    // and/or on individual steps:
    // cyclone.WithRetention(72*time.Hour) on NewStep
)

• [ ] Retention period chosen based on debugging/compliance needs
• [ ] WithRetentionSweep configured on the client, or manual purge scheduled
• [ ] Staging environment tested with a purge, to confirm that the purge does not affect active runs

Set shutdown timeout

// In main():
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()

// ... start client ...

<-ctx.Done()
shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
client.Shutdown(shutdownCtx)

Kubernetes terminationGracePeriodSeconds must be greater than the shutdown timeout:

spec:
  terminationGracePeriodSeconds: 35

• [ ] client.Shutdown called on signal
• [ ] Shutdown timeout is shorter than orchestrator's grace period
• [ ] Longest handler is shorter than the shutdown timeout

Set alerts

Alert	Metric/Query	Threshold
Queue backlog	cyclone_queue_oldest_available_seconds	> 5 minutes
Jobs discarded	cyclone_step_events_total{state="discarded"} rate	> 0 for 5 min
Lease reclaims	cyclone_step_events_total{type="reclaimed"} rate	> 5 in 15 min
Crash loops	SELECT count(*) FROM cyclone_steps WHERE state = 'discarded' AND error_kind = 'reclaim_limit'	> 0
Schedule failures	SELECT consecutive_failures FROM cyclone_schedules	> 3
DB bloat	pg_stat_user_tables.n_dead_tup (cyclone_steps)	> 1M

• [ ] Backlog alert configured
• [ ] Discard rate alert configured
• [ ] Lease reclaim alert configured
• [ ] On-call runbook references this checklist

Test retry and cancel flows

Before launch, verify in staging:

// Verify retry works:
client.Retry(ctx, failedRunID, "send_email", cyclone.RetryAdditionalAttempts(1))

// Verify cancel works:
client.Cancel(ctx, runningRunID)

// Verify idempotency: run a step twice and check no duplicate side effects

• [ ] client.Retry and client.RetryAll tested in staging
• [ ] client.Cancel tested in staging
• [ ] Idempotency verified under simulated retries

Backup and restore

Cyclone tables live in your application database. Standard Postgres backup procedures cover them.

Consider:

• Jobs in progress are in the running state. After a restore, the leases have expired. The recovery sweep starts those steps again on the next startup. This behavior is expected.

• If you restore to a point in time before a completed step, that step starts again. Design handlers for this.

• [ ] cyclone_* tables included in backup schedule

• [ ] Restore procedure tested and documented

• [ ] The team knows that a point-in-time restore can cause steps to start again

Related

• Monitoring and Alerting
• Queue Sizing
• Failure Recovery
• Retention
• Graceful Shutdown