Observability

Start with logs and Stats. Add lifecycle events when you need metrics per kind or queue. SQL views are there when you want to debug a live run from psql.

Logs

Cyclone uses Go's log/slog. Pass your logger when you create the client.

logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
    Level: slog.LevelInfo,
}))

client, err := cyclone.NewClient(pool, mux,
    cyclone.WithWorkers(8),
    cyclone.WithLogger(logger),
)

Useful fields include run ID, node, kind, queue, attempt, duration, and error.

For local debugging, set DEBUG before you start the process. The default logger then emits debug events.

Step Events

Use WithOnStepEvent to count step transitions.

client, err := cyclone.NewClient(pool, mux,
    cyclone.WithWorkers(8),
    cyclone.WithOnStepEvent(func(e cyclone.StepEvent) {
        stepEvents.WithLabelValues(e.Kind, e.Queue, string(e.Type), string(e.State)).Inc()
    }),
)

Keep the callback fast. It executes in worker goroutines. Do not make a network call from it. Increment a metric. Or send the event on a buffered channel. Or return.

Events cover the normal lifecycle and waits:

Event	Use it for
claimed	A worker started a handler
completed	The handler finished successfully
retrying	A retryable failure starts again
discarded	Attempts exhausted, a permanent failure, or the lease expired past WithMaxReclaims (ErrorKind == "reclaim_limit")
cancelled	Run cancellation or fail-fast cancellation
timed_out	The handler exceeded WithTimeout
reclaimed	The lease expired. Cyclone recovered the step
requeued	An operator called Retry or RetryAll
snoozed	The handler returned cyclone.Snooze
signalled	A signal wait consumed a payload
wait_lapsed	A bounded WaitSignal hit its WithMaxWait cap
timer_fired	WaitTimer fired

Subscribe: Event Channel for Multiple Consumers

Subscribe returns a filtered, buffered channel of StepEvent. Use it when a consumer reacts to step transitions outside the one WithOnStepEvent hook:

evCh, cancel := cyclone.Subscribe(client, cyclone.EventCompleted, cyclone.EventDiscarded)
defer cancel()

go func() {
    for ev := range evCh {
        fmt.Printf("step %s/%s: %s\n", ev.Kind, ev.Node, ev.Type)
    }
}()

Multiple subscribers can coexist. Each subscriber gets a buffered channel (capacity 64). A full channel drops events silently. A slow consumer never blocks a worker. Pass no event types to receive all events. The returned cancel function removes the subscription. It also closes the channel.

WithErrorHandler: Centralized Error Reporting

WithErrorHandler fires once per failed attempt. It receives the original Go error, not only the message. Use it for Sentry, Rollbar, or a custom error aggregator:

client, err := cyclone.NewClient(pool, mux,
    cyclone.WithErrorHandler(func(ctx context.Context, ev cyclone.StepEvent, err error) {
        // ev carries the step metadata; err is the raw handler error,
        // unwrappable to its concrete type (for example *panicError for panics).
        sentry.CaptureException(err, sentry.Context(ctx), &sentry.Event{
            Extra: map[string]interface{}{
                "run_id":  ev.RunID,
                "node":    ev.Node,
                "kind":    ev.Kind,
                "attempt": ev.Attempt,
            },
        })
    }),
)

The callback is synchronous. Cyclone catches a panic from the callback. A misbehaving reporter never crashes a worker. Cyclone does not call the callback for a successful attempt. It does not call the callback for a Snooze reschedule. A run deadline dead-letters a step. That event arrives with cyclone.ErrRunDeadlineExceeded. WithMaxReclaims discards a step. That event arrives with cyclone.ErrReclaimLimitExceeded. Neither step has a handler error of its own.

A fan-out node that fails a fan-out limit arrives once. The error matches cyclone.ErrTooManyItems, or it wraps a *PayloadTooLargeError. A fan-out node that fails because an item task failed does not arrive again. The hook already received the failure of the item task.

Stats

Client.Stats gives a snapshot by kind and queue. Poll it from a metrics goroutine.

stats, err := client.Stats(ctx)
if err != nil {
    return err
}

for _, q := range stats.Queues {
    queueAvailable.WithLabelValues(q.Queue).Set(float64(q.Available))
    queueRunning.WithLabelValues(q.Queue).Set(float64(q.Running))
    queueWaitingSignal.WithLabelValues(q.Queue).Set(float64(q.WaitingSignal))
    queueWaitingTimer.WithLabelValues(q.Queue).Set(float64(q.WaitingTimer))

    if q.OldestAvailable != nil {
        queueOldestAge.WithLabelValues(q.Queue).Set(q.OldestAvailable.Seconds())
    }
}

WaitingSignal and WaitingTimer are not backlog in the usual sense. They are parked workflow state. Alert on them only when the count or the age does not match your product flow.

OpenTelemetry (cycloneotel)

The cycloneotel module turns step events into OpenTelemetry spans and metrics. cycloneotel is a separate Go module. The core cyclone module stays dependency-free. You opt into the OTel SDK only where you use it:

go get github.com/wafer-labs/cyclone/cycloneotel

Wire the recorder at construction with the step-event hook:

rec, err := cycloneotel.New() // uses the OTel global tracer/meter providers
if err != nil {
    return err
}
client, err := cyclone.NewClient(pool, mux, cyclone.WithOnStepEvent(rec.Hook()))

Or attach it to a client that is already built. Use Subscribe:

stop, err := cycloneotel.Instrument(client)
defer stop()

Pass cycloneotel.WithTracerProvider or cycloneotel.WithMeterProvider for an explicit provider. Tests can use an in-memory exporter.

What it records:

• Spans: one span per finished node attempt. The span name is the node. Cyclone backdates the span to the real start of the attempt. All attempts of one run share one trace. Cyclone derives that trace from the run id. A retry is a new span in the same trace. Span status follows the event type:

  • EventCompleted → Ok
  • EventDiscarded / EventTimedOut → Error. This is a terminal failure.
  • EventRetrying → Error. This span is the attempt that failed. The next attempt is a separate Ok or Error span in the same trace. A retryable failure therefore produces an Error span. The run is not failed yet. The terminal status of the run is on the run row. It is not on this span.

  Every non-Ok span carries the attributes cyclone.error and cyclone.error_kind.

• Metrics: the counters cyclone.steps.completed and cyclone.steps.failed, and the histogram cyclone.step.duration. Each instrument has the labels cyclone.kind and cyclone.queue.

SQL Views

The migration creates read-only views for debugging:

-- Runs still in progress.
SELECT * FROM cyclone_v_runs
WHERE state = 'running'
ORDER BY created_at;

-- One run's steps.
SELECT node, kind, state, attempt, error, signal_name
FROM cyclone_v_steps
WHERE run_id = $1
ORDER BY node;

-- Queue backlog.
SELECT queue, available, running, waiting_signal, waiting_timer, oldest_available_age
FROM cyclone_v_queues
ORDER BY available DESC;

-- Expired leases.
SELECT run_id, node, kind, lease_expired
FROM cyclone_v_steps
WHERE lease_expired;

These views are also good support queries. When somebody says "my order is stuck", start with cyclone_v_steps.

What to Alert On

The first useful alerts are:

• Available jobs are old on a queue that normally drains quickly.
• Discarded steps are increasing.
• Lease expiries are increasing. No deploy explains them. No crash loop explains them.
• Schedule consecutive_failures is non-zero.
• wait_lapsed fires for a workflow that expects the signal.

Do not alert on every retry. A retry is often the system doing its job.

Next

• Monitoring and Alerting
• Failure Recovery
• Signals and Waits
• Retries