CLI Reference

The cyclone CLI inspects runs and manages them. It is a thin wrapper over the public API and the SQL views.

Install or Build the CLI

# Build locally:
go build -o cyclone ./cmd/cyclone

# Or run without building:
go run ./cmd/cyclone <command> [args]

# Or install to $GOPATH/bin:
go install ./cmd/cyclone

Environment Variables

Variable	Required	Description
CYCLONE_DSN	Yes	Postgres connection string

export CYCLONE_DSN='postgres://user:pass@host:5432/db?sslmode=require'

runs: List Runs

cyclone runs [flags]

Lists runs from cyclone_v_runs, newest first.

Flag	Default	Description
--state	(all)	Filter by run state: running, paused, completed, failed, cancelled
--limit	50	Maximum rows to return (max 100)
--cursor		Keyset cursor. It lists runs older than this id. Omit it for the newest page. The command prints more -> cyclone runs --cursor <id> when another page exists

# All running runs:
cyclone runs --state=running

# Last 20 failed runs:
cyclone runs --state=failed --limit=20

# All recent runs (default limit):
cyclone runs

Output columns: ID, WORKFLOW, STATE, STEPS, PARENT, CREATED, FINISHED

PARENT shows the parent run and the parent node of a child run, for example 42/fulfillment. It is empty for any other run. --format json carries the two values as parent_run_id and parent_node.

steps: Inspect Steps of a Run

cyclone steps <runID>

Lists all steps for the given run from cyclone_v_steps.

cyclone steps 42

Output columns: NODE, KIND, STATE, ITEMS, CHILD, ATTEMPT, RECLAIMS, LEASE_EXPIRED, ERROR

The item tasks of a fan-out node follow the node, in item order: process[0], process[1], and so on. ITEMS shows fan-out progress as completed item tasks over all item tasks, for example 7/10. It is empty for any other step. --format json carries it as items with completed and total. It also carries fan_out_node on each item task.

CHILD shows the child run of a child node. It is empty for any other step. It is also empty for a child node whose child run did not start. --format json carries it as child_run_id. Use cyclone steps <childRunID> to see the steps of the child run.

The CLI prints ATTEMPT as n/m (attempt over max attempts) in one column. There is no separate MAX_ATTEMPTS column. There is no QUEUE column. RECLAIMS is the number of times the recovery sweep reclaimed the step after its lease expired. WithMaxReclaims bounds that count. --format json carries it as reclaims. PANIC_STACK appears only under --format json. It carries the captured goroutine stack when the last failure was a panic.

LEASE_EXPIRED is a boolean. The CLI prints it as true or false. It is true when the step is running but its lease has expired. That step can be stuck.

dlq: Dead-Letter Queue (Exhausted Steps)

cyclone dlq list [flags]
cyclone dlq retry [flags]

dlq list: list exhausted steps

Lists discarded steps (attempts used up) across all failed runs. The list groups them by kind and failure reason.

Flag	Default	Description
--kind	(all)	Filter by handler kind name
--reason	(all)	Filter by failure classification: permanent, retryable, deadline, or reclaim_limit
--since	(all time)	Only steps discarded in this window, for example 24h or 1h30m
--limit	100	Maximum rows (capped at 1000)

cyclone dlq list
cyclone dlq list --kind send_email
cyclone dlq list --reason permanent --since 24h --limit 50

--reason deadline lists steps that a WithRunDeadline cutoff stopped (error_kind = "deadline"). You can also query them directly: SELECT * FROM cyclone_v_steps WHERE error_kind = 'deadline'.

--reason reclaim_limit lists steps whose lease expired again after WithMaxReclaims reclaims. That usually means a handler that stops its worker. The RECLAIMS column (reclaims in JSON) shows how many reclaims each dead letter had.

dlq retry: bulk requeue exhausted steps

The command reopens every failed run that contains a step matching the filters. It grants more attempts. It operates at run granularity. It reopens every discarded step in an affected run, not only the matched step.

Flag	Default	Description
--kind	(all)	Filter by handler kind name
--reason	(all)	Filter by failure classification
--since	(all time)	Only steps discarded within this window
--attempts	1	More execution attempts to grant. The command also resets reclaim counts

cyclone dlq retry --kind send_email --since 24h --attempts 3
cyclone dlq retry --reason permanent --attempts 5

Output: requeued N run(s) from the dead-letter queue

A run with a child node whose child run cannot give a result again is skipped and not counted. See ErrChildNotRetryable.

cancel: Cancel a Run

cyclone cancel <runID>

Cancels the run and all of its non-terminal steps. It wraps client.Cancel.

cyclone cancel 42

A cancel of a run that is already terminal is not an error. The command prints run <id> already terminal; nothing to cancel and exits 0. A script that cancels in advance does not have to special-case that line. An unknown run id is the error case.

The Go API is different. client.Cancel returns ErrAlreadyTerminal. The CLI swallows that error.

pause: Pause a Run

cyclone pause <runID>

Stops new work in the run until cyclone resume. A step that is in progress at the pause continues. It wraps RunHandle.Pause. See Pause and Resume Runs.

A run that is already paused or terminal is not an error. The command prints run <id> already paused; nothing to pause or run <id> already terminal; nothing to pause and exits 0. An unknown run id is the error case.

resume: Resume a Paused Run

cyclone resume <runID>

Continues a paused run. Wraps RunHandle.Resume.

A run that is not paused is not an error. The command prints run <id> is not paused; nothing to resume and exits 0. An unknown run id is the error case.

The Go API is different. Pause returns ErrAlreadyPaused or ErrAlreadyTerminal. Resume returns ErrNotPaused. The CLI reports those errors and exits 0.

retry: Requeue Discarded Steps

# Retry a single discarded step:
cyclone retry <runID> <node>

# Retry all discarded steps in a failed run:
cyclone retry <runID> --all

# Retry a specific step:
cyclone retry 42 send_email

# Retry all discarded steps:
cyclone retry 42 --all

Flags:

Flag	Description
--all	Retry all discarded nodes (uses client.RetryAll)
--attempts	More attempts to grant (default: 1; a value below 1 counts as 1)
--reason	Audit reason, shown on the requeued step event. The step does not store it. The reopen sets error = NULL

cyclone retry 42 send_email --attempts=3 --reason="SMTP restored"
cyclone retry 42 --all --attempts=5

schedules: List or Manage Recurring Schedules

With no flags, the command lists every schedule. Pass exactly one of the flags below to act on one schedule by id.

Flag	Description
--pause	Pause the schedule with this id. You can resume it. The scanner stops firing it
--resume	Resume a paused schedule. This sets next_run_at to the next future tick
--trigger	Fire the schedule once, now, outside the normal scan. The pending slot fires exactly once

cyclone schedules
cyclone schedules --pause hourly-report
cyclone schedules --resume hourly-report
cyclone schedules --trigger hourly-report

migrate: Apply Schema Migrations

cyclone migrate            # apply every pending migration
cyclone migrate --dry-run  # print the current schema version and change nothing

Flag	Description
--dry-run	Report the current schema version without applying anything

stats: Backlog and Health

cyclone stats
cyclone --format json stats

Prints the backlog figures Client.Stats returns. See Monitoring and Alerting for what each one means and what to alert on.

purge: Delete Old Terminal Runs

cyclone purge --older-than 720h
cyclone purge --older-than 720h --batch-size 5000

Flag	Default	Description
--older-than	(required)	Age cutoff, for example 720h
--batch-size	1000	Rows deleted per batch

This command is PurgeTerminalRuns. It uses one flat age cutoff for every run. It ignores per-step WithRetention. It can delete a step that is still inside its retention window. The background sweeper from WithRetentionSweep is the one that honors per-step retention. See Retention and Purging.

Global Flags

Flag	Default	Description
--format	table	Output format: table or json. It applies to every listing command. Usage errors and parse errors always go to stderr, so a JSON stream stays clean for jq
--version		Print the CLI version and exit
--help		Print usage for the CLI or a subcommand

Dead-Letter Output Examples

$ cyclone dlq list --kind send_email --since 24h
 RUN   WORKFLOW   NODE        KIND         REASON         ATTEMPTS  RECLAIMS  DISCARDED              ERROR
 105   order      send_email  send_email   retryable      5/5       0         2024-01-15 14:23:01Z    dial tcp: connection refused
 108   notify     send_email  send_email   permanent      1/5       0         2024-01-15 14:25:42Z    email is required
 111   notify     send_email  send_email   reclaim_limit  0/5       3         2024-01-15 14:31:07Z    reclaim limit exceeded: lease expired after 3 reclaims

$ cyclone dlq retry --kind send_email --since 24h --attempts 3
requeued 2 run(s) from the dead-letter queue

Output Examples

$ cyclone runs --state=running --limit=5
 ID    WORKFLOW      STATE    STEPS           PARENT         CREATED
 105   crawl         running  2/3 completed                  2024-01-15 14:23:01
 106   order-fulfil  running  1/4 completed                  2024-01-15 14:24:15
 107   fulfillment   running  0/2 completed   106/ship       2024-01-15 14:24:16

$ cyclone steps 105
 NODE     KIND       STATE      ITEMS  CHILD  ATTEMPT  RECLAIMS  LEASE_EXPIRED  ERROR
 fetch    fetch      completed                1/5      0         false
 parse    parse      completed                1/5      0         false
 save     save       running                  2/5      1         true

$ cyclone cancel 105
cancelled run 105

$ cyclone cancel 105
run 105 already terminal; nothing to cancel

$ cyclone pause 107
paused run 107

$ cyclone pause 107
run 107 already paused; nothing to pause

$ cyclone resume 107
resumed run 107

$ cyclone resume 107
run 107 is not paused; nothing to resume

$ cyclone retry 106 charge --attempts=3 --reason="payment gateway restored"
retried node charge in run 106

$ cyclone retry 106 --all --attempts=3
retried all exhausted nodes in run 106

A cancel of an already-terminal run is not an error. The command prints the line above and exits 0. A script that cancels in advance does not have to special-case that line.

Exit Codes

Code	Meaning
0	Success
1	Any failure: invalid arguments, database connection error, run not found

There is no exit code 2. Every error path prints cyclone: <message> to stderr and exits 1. A cancel of an already-terminal run is a success, not a failure.

repair: Advance Runs That Can No Longer Make Progress

cyclone repair stranded-runs --dry-run   # report the count, change nothing
cyclone repair stranded-runs             # repair up to --batch runs
cyclone repair stranded-runs --batch 500

Flag	Default	Description
--batch	100	Maximum runs repaired in one pass
--dry-run		Report the count without repairing

A run is stranded when it is still running and it has no available, running, or waiting step. It also has at least one pending step whose every dependency has completed. Nothing else in the engine reports that run or fixes it. Each repair uses its own transaction. The command is safe on a live database.

cyclone stats reports the same count as stranded_runs, next to failing_schedules. Monitoring and Alerting describes both counts. Failure Recovery gives the repair procedure.

Operational Recipes

Find all runs stuck in running state for > 1 hour:

# Using psql directly:
psql "$CYCLONE_DSN" -c "
    SELECT id, workflow, created_at
    FROM cyclone_runs
    WHERE state = 'running'
      AND created_at < now() - interval '1 hour'
    ORDER BY created_at;"

Retry all failed runs from the last 24 hours:

# Get failed run IDs, then retry each:
psql "$CYCLONE_DSN" -t -c "
    SELECT id FROM cyclone_runs
    WHERE state = 'failed'
      AND finished_at > now() - interval '24 hours'" | \
while read runID; do
    echo "Retrying run $runID"
    cyclone retry "$runID" --all --attempts=3 --reason="bulk retry after outage"
done

Check for lease-expired steps:

psql "$CYCLONE_DSN" -c "SELECT run_id, node, kind FROM cyclone_v_steps WHERE lease_expired;"

Related

• Failure Recovery
• Observability
• Retries and Permanent Failures