Migrations Reference

How Cyclone's migration system works and what to expect when upgrading.

How Migrations Work

cyclone.Migrate(ctx, pool) applies the embedded SQL migration files with the tern migration library. It:

1. Creates cyclone_schema_version (the migration tracking table) if it does not exist.
2. Reads the current maximum applied version.
3. Applies each pending migration in order.
4. Commits each migration in its own transaction.

Migrations are idempotent. A call to Migrate on an already-migrated schema is a no-op. It is safe to call on every startup.

// In main(), before NewClient:
if err := cyclone.Migrate(ctx, pool); err != nil {
    log.Fatalf("migration failed: %v", err)
}

Migration Versions

Migrations are numbered in order. The version in cyclone_schema_version is the highest applied number.

SELECT * FROM cyclone_schema_version ORDER BY version;

Version 1: initial schema

• Creates cyclone_runs, cyclone_steps, cyclone_deps, cyclone_signals
• Creates cyclone_schedules, cyclone_schedule_fires, cyclone_step_inputs
• Creates cyclone_v_runs, cyclone_v_steps, cyclone_v_kinds, cyclone_v_queues
• Applies Cyclone-specific indexes, constraints, and autovacuum settings

Version 2 — signals and wait nodes (WaitSignal / WaitTimer)

• Extends cyclone_steps_state_check to admit waiting_signal and waiting_timer. The migration adds the check as NOT VALID and then validates it. The rewrite does not block writes
• Adds cyclone_steps.wait_for (INTERVAL, nullable) — the activation-relative wait duration
• Adds cyclone_signals.consumed_at (TIMESTAMPTZ, nullable) — mailbox consume timestamp
• Adds idx_signals_unconsumed, idx_steps_signal_timeout, and the one-waiter-per-name guard the sender and scanner rely on

Version 3 — fan-in assembly mode (Join2..4 / FromAll)

• Adds cyclone_step_inputs.assembly_mode (TEXT NOT NULL DEFAULT 'tuple'). tuple is the a/b/c/d object that Join2..4 build. array is the JSON array from FromAll
• The worker chooses the shape at claim time. The mode is stored on each binding. The DEFAULT keeps every existing binding in tuple mode

Version 4 — default max_attempts 25 → 5

• Changes the column default on cyclone_steps.max_attempts and cyclone_schedules.max_attempts
• Existing rows stay as they are. A default applies only to an insert that omits the column. In-flight work keeps the budget it was created with

Version 5: adds panic_stack column

• Adds cyclone_steps.panic_stack (TEXT, nullable) for goroutine stacks from recovered handler panics
• Re-issues cyclone_v_steps to expose the new column
• Panic error text stays in error as a one-liner; the full stack lands in panic_stack

Version 6 — run-level deadline (WithRunDeadline)

• Adds cyclone_runs.deadline_at (TIMESTAMPTZ, nullable). It is the absolute cutoff now()+d at submit. NULL means no run deadline
• Adds the partial index idx_runs_deadline ON cyclone_runs (deadline_at) WHERE state = 'running' AND deadline_at IS NOT NULL. It backs the deadline scanner
• The column is nullable and has no default. Existing runs are unchanged. A run submitted with no deadline is never eligible

Version 7 — quorum waits (WaitN)

• Adds cyclone_steps.wait_count (INT, nullable) — quorum N for WaitN nodes; NULL means single-delivery WaitSignal (equivalent to N=1)
• Partial payloads accumulate in the existing output column until the quorum is met; no view re-issue needed

Version 8 — first-of waits (WaitAny)

• Adds cyclone_steps.signal_names (TEXT[], nullable). These are the candidate signal names for a WaitAny first-of wait
• signal_name stays NULL for a WaitAny node. The DAG validator enforces per-name uniqueness across single-name waits and multi-name waits at submit. The existing idx_steps_signal_name_unique index does not apply

Version 9 — per-branch failure policy (OnDepFailure)

• Adds cyclone_steps.on_dep_failure (TEXT, nullable). It is the per-node dependency-failure policy. NULL means fail_run (the default fail-fast). 'cancel_branch' contains the failure in this node's downstream branch
• The value is stored on the step. The engine's one terminal-decision path can read the declared policy with no separate lookup

Version 10 — FIFO-per-key sequences (WithSequence)

• Adds cyclone_steps.sequence_key (TEXT, nullable). It is the gating key from WithSequence (standalone enqueues only). It gates claim so one run per key is active at a time
• Adds the partial index idx_steps_sequence ON cyclone_steps (sequence_key, scheduled_at, id) WHERE sequence_key IS NOT NULL AND state IN ('available','running'). It backs the claim-time gate

Version 11 — runtime hardening

• Adds cyclone_steps.lease_generation (BIGINT NOT NULL DEFAULT 0). Every claim increments it. Worker completion and heartbeat transitions compare it. That comparison fences an execution whose lease was reclaimed and claimed again
• Adds 256-byte queue-name constraints on cyclone_steps.queue and cyclone_schedules.queue. They match API validation. They keep NOTIFY payloads below the PostgreSQL limit
• Rollout requirement: stop all v0.1.0 workers and drain them before you apply this migration. Then deploy the upgraded workers. An old worker does not send lease_generation. It is not safe at the same time as the new fence.

Version 12 — schedule definition pinning

• Adds cyclone_schedules.def_version (TEXT, nullable). It is the workflow-definition fingerprint pinned at ScheduleWorkflow registration. A firing process refuses the fire when its registered builder produces a different fingerprint. Cyclone records that refusal in last_error. The slot stays for a process that has the pinned definition. The column is NULL for a step schedule, an action schedule, and a row registered before this migration. A new registration pins those rows.
• Rollout: apply it before you deploy a binary that includes it. The new schedule queries select the column. Unlike migration 11, no worker drain is required. Old workers ignore the column. Unpinned rows fire on every worker.

Version 13 — enqueue constraint and cascade indexes

• Repairs rows that the old API could write. An empty unique_key or sequence_key becomes NULL. max_attempts < 1 becomes 1. It also adds cyclone_steps_max_attempts_check (CHECK (max_attempts >= 1) NOT VALID)
• Adds idx_signals_run and idx_schedule_fires_run. The ON DELETE CASCADE that retention triggers then does not scan the child tables in sequence

Version 14 — crash-reclaim accounting (WithMaxReclaims)

• Adds cyclone_steps.reclaims (INT NOT NULL DEFAULT 0), the number of expired-lease reclaims a step has had. The constant default makes this a catalog-only change. Existing rows are not rewritten. They start at 0
• Re-issues cyclone_v_steps to expose reclaims
• error_kind has no CHECK constraint. The new reclaim_limit kind needs no constraint change
• Rollout: no worker drain is required. A worker built before it ignores the column. It does not count reclaims. It does not reset the count on an operator retry. Only an upgraded client enforces a bound

Version 15: workflow result node (DefineResult)

• Adds cyclone_runs.result_node (TEXT, nullable), the ID of the node whose output is the run result. Submit writes it in the run insert, in the same transaction as the graph
• The column is NULL for a run from Define, for a standalone run, and for every run from before this migration. A result handle returns ErrNoOutput for such a run. OutputNode still reads its nodes
• The column is nullable and has no default, so this is a catalog-only change. Existing rows are not rewritten. There is no index. Cyclone reads the column only by the run primary key
• Rollout: apply it before you deploy a binary built with it. The run insert of that binary names the column for every workflow, also for a workflow from Define. No worker drain is required. A binary built before it does not name the column. Its run inserts leave the column NULL. Its claim, completion, and retry queries do not read it. A workflow with no result keeps its def_version. Schedule pins do not change

Version 16: submit keys (SubmitOnce)

• Adds cyclone_runs.submit_key (TEXT, nullable). SubmitOnce and SubmitOnceTx write their key to it. Every other path writes NULL
• Adds the partial unique index idx_runs_submit_key ON cyclone_runs (workflow, submit_key) WHERE submit_key IS NOT NULL. It covers runs in all states. A key stays in use until Cyclone purges its run
• The index build reads all of cyclone_runs while the migration holds its lock. A large deployment can build the column and the index out of band first. The migration comment gives the statements. The comment tells you to check that the index is valid
• Rollout: no worker drain is necessary. A binary built before this migration does not read or write the column. A binary built with it uses the column only in SubmitOnce and SubmitOnceTx

Version 17: fan-out over a list (ForEach)

• Adds cyclone_runs.has_fan_out (BOOLEAN NOT NULL DEFAULT false). The engine reads it with the run row lock. A run with no fan-out node does no extra work
• Adds cyclone_steps.fan_out_parallel (INT) and cyclone_steps.fan_out_max_bytes (BIGINT), set on a fan-out node only. They are the WithMaxParallel limit and the byte limit of the collected output
• Adds cyclone_steps.item_of (TEXT) and cyclone_steps.item_index (INT), set on an item task only. An item task is an ordinary step row. Its node ID is the fan-out node ID plus the item index. The existing unique (run_id, node) constraint gives it one identity
• Adds the partial indexes idx_steps_fan_out and idx_steps_items. A step that is not part of a fan-out adds no entry to them. IF NOT EXISTS lets a large deployment build them with CREATE INDEX CONCURRENTLY first. The migration file has the statements
• Re-issues cyclone_v_steps to expose item_of and item_index
• Each column is nullable or has a constant default. No row is rewritten
• Rollout: apply it before you deploy a binary that includes it. The engine reads has_fan_out in every run. No worker drain is necessary, because an older worker ignores the columns. But an older worker does not know a fan-out node and can execute it as an ordinary step. Upgrade every worker before a workflow uses ForEach

Version 18: pause and resume for runs (run.Pause / run.Resume)

• Adds paused to cyclone_runs_state_check and to cyclone_steps_state_check. Each new CHECK accepts every old value. The migration therefore adds it NOT VALID and does not scan the table. To validate it out of band, execute ALTER TABLE ... VALIDATE CONSTRAINT
• Adds cyclone_runs.paused_at (TIMESTAMPTZ, nullable), the time of the pause. Resume moves deadline_at and each timed wait forward by the length of the pause. This is a catalog-only change
• Rebuilds idx_steps_unique_active and idx_steps_sequence with paused in their predicates. A paused step keeps its unique key and its place in its sequence. Each rebuild scans cyclone_steps one time while the migration holds its lock. Plan the time for a large table
• The claim plan does not change for a run that is not paused. A paused step is not in idx_steps_claim. The WithSequence gate of the claim query also counts a paused step as the head of its key
• Rollout: no worker drain is required to apply it, because an old worker never writes paused. Deploy the new build to every worker before you pause a run. An old worker does not know the paused state. It can start the dependent steps of a paused run. It cannot complete a paused run or fail it. An old producer or CLI reads a paused run as terminal. Its Wait returns. Its Signal and Cancel return ErrAlreadyTerminal
• Down: the down migration refuses while a run or a step is paused. Resume or cancel paused runs first

Version 19: child workflow runs (workflow.Child)

• Adds cyclone_runs.parent_run_id (BIGINT) and cyclone_runs.parent_node_id (TEXT), nullable. The engine writes them in the insert of a child run, in the transaction that inserts its graph. They are NULL for a run with no parent
• Adds cyclone_steps.child_workflow (TEXT, nullable), the workflow that a child node starts. It is NULL for any other step
• Adds cyclone_steps.child_needs_result (BOOLEAN, nullable). It is true for a ResultWorkflow.Child node, which needs the result of its child run. It is false for a Workflow.Child node. It is NULL for any other step
• Adds the partial unique index idx_runs_parent ON cyclone_runs (parent_run_id, parent_node_id) WHERE parent_run_id IS NOT NULL. It keeps one child run for each parent node
• Adds the partial index idx_steps_waiting_child ON cyclone_steps (run_id) WHERE state = 'waiting_child'. The wait scanner reads it
• Adds waiting_child to cyclone_steps_state_check. The new CHECK accepts every old value. The migration therefore adds it NOT VALID
• Re-issues cyclone_v_runs with parent_run_id and parent_node_id appended
• Each column is nullable and has no default. No row is rewritten. Each index build scans its table while the migration holds its lock. The indexes start empty. A large deployment can build the columns and the indexes out of band first. The migration comment gives the statements. The comment tells you to check that the indexes are valid
• The reserved column cyclone_steps.parent_node from version 1 stays reserved. The parent link is on cyclone_runs
• Rollout: apply it before you deploy a binary built with it. The run reads of that binary name the new columns. No worker drain is necessary. Upgrade every worker, every client that purges runs, and the CLI before you submit a workflow that uses Child. An older worker discards a child node. An older purge does not keep a child run for its parent. An older Cancel does not reach child runs
• Down: the down migration refuses while a step is waiting_child. Cancel the parent runs of the waiting child nodes first. Child runs stay, as runs with no parent

Cyclone applies migrations in file order. tern numbers each one by its position in the sorted set. tern rejects a gap and a duplicate.

migrations/ in the module is the canonical list. Check what your database has applied with SELECT * FROM cyclone_schema_version.

cyclone.LatestSchemaVersion is the version that a full Migrate of your build records. cyclone.SchemaVersionTable is the table name. A tool that reads the database but does not migrate it can compare the stored version with the constant. A lower stored version shows a pending migration. A higher stored version shows that a newer build migrated the database.

Running Migrations

On startup (only when the release permits mixed versions)

cyclone.Migrate(ctx, pool)

Cyclone skips a version that is already applied. A pending migration can take DDL locks or scan existing data. A startup migration is not automatically low-latency. It is not automatically safe with old workers.

Alternative: as a pre-deploy step

Build a migration-only binary or add a migrate subcommand:

if os.Args[1] == "migrate" {
    if err := cyclone.Migrate(ctx, pool); err != nil {
        log.Fatal(err)
    }
    return
}

Call this on a direct or session-mode connection before you roll out new worker instances. Use that path when:

• Migrations must finish before old workers stop reading the old schema.
• You want an explicit migration step in the deployment pipeline.

Downgrade Policy

There is no automatic downgrade. Cyclone applies migrations forward only.

To roll back a migration manually:

1. Identify the DDL that the migration applied (check the embedded SQL files in the Cyclone source).
2. Write a compensating DDL statement and apply it manually (for example, DROP VIEW cyclone_v_kinds — cyclone_v_* are views, not tables).
3. Lower the schema version: UPDATE cyclone_schema_version SET version = N-1.

Pre-1.0: there is no compatibility guarantee. A downgrade can need manual work.

Schema Fingerprinting in Tests

cyclonetest.New creates a fresh, isolated schema for each test. Each schema gets the full migrations from scratch. Tests do not share schema state.

env := cyclonetest.New(t)
// DSN includes search_path=cyclone_test_<random>
// All cyclone_* tables exist in this schema only.

The fresh schema keeps tests isolated even when they execute in parallel.

Production Rollout Notes

Cyclone normally applies migrations before workers start. The initial migration creates the Cyclone schema in one transaction. For a later upgrade, review the embedded SQL before the production rollout.

For a rolling deploy, use mixed worker versions only when the release notes say the migration is backward-compatible. Migration 11 is not. Drain the v0.1.0 workers before you apply it. Apply migrations on a direct or session-mode connection. The tern migration lock is session-scoped.

If a migration fails part way through, only that migration rolls back. Each migration uses its own transaction. Call Migrate again to retry the failed migration.

Related

• Installation and Migrations
• Upgrades and Versioning
• Schema