Monitoring and Alerting

Key metrics, SQL queries, and example Prometheus alerting rules for Cyclone in production.

Key Metrics

Cyclone exports no Prometheus metrics of its own. Every cyclone_* series named on this page is one you define, from the WithOnStepEvent hook or a periodic client.Stats() call. The one built-in exporter is cycloneotel, and it records OpenTelemetry instruments — cyclone.steps.completed, cyclone.steps.failed, and cyclone.step.duration — not the names below.

Every step alert here reads one counter, incremented once per event:

cyclone.WithOnStepEvent(func(e cyclone.StepEvent) {
    stepEvents.With(labels{
        "type":  string(e.Type),
        "state": string(e.State),
        "kind":  e.Kind,
        "queue": e.Queue,
    }).Inc()
})

The state label makes discards and retries selectable. A discard from a timeout reports timed_out, not discarded. The event type alone under-counts both.

Metric	Why it matters
Queue depth (available count)	Primary backlog indicator; high = workers saturated
Running age	Stuck workers or slow handlers
Discard rate	Handler failures that exhausted retries
Retry rate	Transient failures. A high rate can show an infrastructure fault
Lease reclaim rate	Worker crashes or severe load
Schedule fail count	Broken scheduled handlers
DB dead tuples	Autovacuum effectiveness on high-churn tables

Queue Depth

// From periodic client.Stats() call:
for _, q := range stats.Queues {
    queueAvailable.With(labels{"queue": q.Queue}).Set(float64(q.Available))
    if q.OldestAvailable != nil {
        queueOldestAge.With(labels{"queue": q.Queue}).Set(q.OldestAvailable.Seconds())
    }
}

-- Direct SQL:
SELECT queue, available, oldest_available_age
FROM cyclone_v_queues
ORDER BY available DESC;

Alert: oldest_available_age > 300s on the default queue for 5 minutes.

Running Age

-- Find steps that have been running longer than expected. cyclone_v_steps does not
-- carry the queue, so join cyclone_steps when you want to group by it.
SELECT v.run_id, v.node, v.kind, s.queue, v.attempt,
       now() - v.started_at AS running_age
FROM cyclone_v_steps v
JOIN cyclone_steps s ON s.run_id = v.run_id AND s.node = v.node
WHERE v.state = 'running'
ORDER BY running_age DESC;

Alert: any step in running for more than 2× its configured WithTimeout.

Discarded Count

SELECT kind, count(*) as discarded
FROM cyclone_v_kinds
WHERE state = 'discarded'
ORDER BY discarded DESC;

Alert: increase(cyclone_step_events_total{state="discarded"}[5m]) > 0. Keying on the state counts a timed-out discard too.

Retry Rate

Alert: rate(cyclone_step_events_total{type=~"retrying|timed_out",state="available"}[5m]) > 0.5 for 10 minutes. A timed-out attempt with attempts left reports timed_out and returns to available. An attempt with none left is a discard.

A high retry rate for one kind shows a handler that keeps failing. Investigate the error:

SELECT kind, error, count(*) as occurrences
FROM cyclone_steps
WHERE state IN ('available', 'running') AND attempt > 1
GROUP BY kind, error
ORDER BY occurrences DESC;

Lease Expiry Count

-- Currently stuck (running + lease expired). The queue lives on cyclone_steps, not
-- on the view, so read lease_expired from the view and the queue from the table.
SELECT s.queue, count(*) AS stuck_steps
FROM cyclone_v_steps v
JOIN cyclone_steps s ON s.run_id = v.run_id AND s.node = v.node
WHERE v.lease_expired
GROUP BY s.queue;

Alert: increase(cyclone_step_events_total{type="reclaimed"}[15m]) > 5. Sustained lease expirations show worker instability (crashes, OOMs, or severe overload).

A step that keeps losing its lease is a crash loop, not instability. With WithMaxReclaims set, it ends as a dead letter with error kind reclaim_limit. Alert on any row:

SELECT kind, count(*) AS crash_loops
FROM cyclone_steps
WHERE state = 'discarded' AND error_kind = 'reclaim_limit'
GROUP BY kind;

Schedule Failures

SELECT id, kind, consecutive_failures, next_run_at
FROM cyclone_schedules
WHERE consecutive_failures > 0
ORDER BY consecutive_failures DESC;

Alert: SELECT count(*) FROM cyclone_schedules WHERE consecutive_failures > 3 returns > 0.

Alert well below 10. At 10 consecutive failures, Cyclone skips the slot and loses that fire. Client.Stats reports the same count as Health.FailingSchedules. cyclone stats prints it.

A definition mismatch or an unregistered workflow does not increment the counter. Those are deployment-topology conditions. The slot stays claimable by a peer. Cyclone records them in last_error with consecutive_failures = 0:

SELECT id, kind, last_error, next_run_at
FROM cyclone_schedules
WHERE enabled AND consecutive_failures = 0 AND last_error IS NOT NULL
ORDER BY next_run_at;

Alert on any row that persists past the end of a deployment. During a rollout this is expected. It clears when the matching build fires the slot. After the rollout, it means no running process can fire that schedule. Cyclone emits the WARN line cyclone schedule fire refused by this process once for each schedule, slot, and reason. Cyclone emits it again every 10 minutes while the refusal persists.

Stranded Runs

A run is stranded when it is still running and it has no available, running, or waiting step. It also has at least one pending step whose every dependency has completed. Nothing in the engine reports it. The recovery sweep only selects running steps. The wait scanner only selects waiting_* steps. The deadline scanner only selects runs with a deadline_at. The queue-depth metrics and the running-age metrics still read healthy. No step is queued, and none is late. The run has stopped.

SELECT r.id, r.workflow, r.created_at
FROM cyclone_runs r
WHERE r.state = 'running'
  AND NOT EXISTS (
    SELECT 1 FROM cyclone_steps x
    WHERE x.run_id = r.id
      AND x.state IN ('available', 'running', 'waiting_signal', 'waiting_timer', 'waiting_child'))
  AND EXISTS (
    SELECT 1 FROM cyclone_steps s
    WHERE s.run_id = r.id
      AND s.state = 'pending'
      AND NOT EXISTS (
        SELECT 1 FROM cyclone_deps d
        JOIN cyclone_steps p ON p.run_id = d.run_id AND p.node = d.dep_node
        WHERE d.run_id = s.run_id AND d.node = s.node AND p.state <> 'completed'));

Alert on any row that persists for longer than one poll interval. A run can appear here for a moment between a completion commit and the next promotion. Require the condition to hold. Do not alert on a single sample.

A paused run is not stranded. The query selects only running runs. A run whose child node waits for its child run is not stranded either. waiting_child is a waiting state.

Client.Stats reports the count as Health.StrandedRuns. cyclone stats prints it. Repair with cyclone repair stranded-runs. See Failure Recovery.

Child Nodes With No Child Run

A child node waits in waiting_child until a worker that registered its workflow starts the child run. A worker that did not register the workflow leaves the node waiting. This query lists waiting child nodes whose child run did not start:

SELECT s.run_id, s.node, s.child_workflow
FROM cyclone_steps s
WHERE s.state = 'waiting_child'
  AND NOT EXISTS (
    SELECT 1 FROM cyclone_runs c
    WHERE c.parent_run_id = s.run_id AND c.parent_node_id = s.node);

Alert on a row that persists for longer than a few poll intervals. It means that no running worker registered the workflow. Give the workflow to WithHandlers on a worker, or set WithMaxWait on the child node to bound the wait.

Paused Runs

A paused run starts no new work until an operator resumes it. Retention does not delete it. A pause that nobody ends holds the run, its unique keys, and its sequence places.

SELECT id, workflow, paused_at
FROM cyclone_runs
WHERE state = 'paused'
  AND paused_at < now() - interval '1 hour'
ORDER BY paused_at;

Alert: any row older than your longest planned maintenance. cyclone runs --state paused lists the same runs. See Pause and Resume Runs.

DB Dead Tuples

SELECT relname, n_dead_tup, n_live_tup,
       round(100.0 * n_dead_tup / nullif(n_live_tup + n_dead_tup, 0), 1) as dead_pct
FROM pg_stat_user_tables
WHERE relname IN ('cyclone_steps', 'cyclone_runs')
ORDER BY n_dead_tup DESC;

Alert: dead_pct > 20 on cyclone_steps. This shows that autovacuum is not keeping up. Review the settings, or start a manual VACUUM ANALYZE cyclone_steps.

Example Prometheus Alerting Rules

groups:
  - name: cyclone
    rules:

      - alert: CycloneQueueBacklog
        expr: cyclone_queue_oldest_available_seconds > 300
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Cyclone queue {{ $labels.queue }} has steps waiting > 5 minutes"
          description: "Oldest available step age: {{ $value | humanizeDuration }}"

      - alert: CycloneJobsDiscarded
        expr: increase(cyclone_step_events_total{state="discarded"}[5m]) > 0
        labels:
          severity: warning
        annotations:
          summary: "Cyclone {{ $labels.kind }} jobs are being discarded"

      - alert: CycloneHighRetryRate
        expr: rate(cyclone_step_events_total{type=~"retrying|timed_out",state="available"}[5m]) > 0.5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High retry rate for {{ $labels.kind }} ({{ $value | humanize }} retries/sec)"

      - alert: CycloneLeaseReclaims
        expr: increase(cyclone_step_events_total{type="reclaimed"}[15m]) > 5
        labels:
          severity: warning
        annotations:
          summary: "Elevated cyclone lease reclaims — possible worker instability"

      - alert: CycloneScheduleFailures
        # Check via a recording rule that queries cyclone_schedules.consecutive_failures
        expr: cyclone_schedule_consecutive_failures > 3
        labels:
          severity: warning
        annotations:
          summary: "Schedule {{ $labels.id }} has failed {{ $value }} times"

Dashboard Queries

Queue health overview:

SELECT queue, available, running, oldest_available_age, lease_expired
FROM cyclone_v_queues
ORDER BY available DESC;

Per-handler error rates:

SELECT kind, state, count
FROM cyclone_v_kinds
WHERE state IN ('discarded', 'available')
ORDER BY kind, state;

Recent failed runs:

SELECT r.id, r.workflow, r.finished_at,
       s.node, s.error, s.attempt, s.max_attempts
FROM cyclone_runs r
JOIN cyclone_steps s ON s.run_id = r.id AND s.state = 'discarded'
WHERE r.state = 'failed'
ORDER BY r.finished_at DESC
LIMIT 20;

Related

• Observability
• Production Checklist