Upgrades and Versioning

How Cyclone handles versioning, what to expect during upgrades, and how to safely migrate in-flight work.

Current Pre-1.0 Status

Cyclone is pre-1.0. The public API is stabilizing, but it can change between minor versions. Before you commit Cyclone to a production system, review the changelog for the release you pin.

Release notes record breaking changes in pre-1.0. A non-trivial change includes a migration guide.

API Compatibility Expectations

Pre-1.0 (current): best-effort compatibility. Symbols can be renamed. Options can change signatures. New required parameters can be added. Pin a specific minor version (for example, v0.3.x) and review the diff before you upgrade.

Post-1.0 (planned): semantic versioning. Minor versions stay backward-compatible. Breaking changes occur only in major versions. The source marks deprecated symbols with // Deprecated:. Those symbols stay for at least one minor version before removal.

For now, treat the API as stable inside a patch version. Treat a minor version as a possible break.

Migration Compatibility

Cyclone migrations are:

• Forward-only. Every migration contains down SQL for development, but Migrate only advances production schemas. Do not assume that a future pre-1.0 migration is additive or lock-free.
• Tracked. cyclone_schema_version records applied versions. cyclone.Migrate skips migrations that are already applied.
• Safe to call again. A call to Migrate on an already-migrated schema is safe. The call is a no-op.

There is no automatic downgrade path. To roll back a migration, write manual DDL and apply it. Cyclone does not guarantee a safe downgrade path between pre-1.0 minor versions.

Production upgrade procedure:

1. Review the release notes for new migrations.
2. Check whether the release requires old workers to drain before migration. Version 11's lease-generation fence does.
3. Call cyclone.Migrate on a direct or session-mode Postgres connection (never transaction-mode PgBouncer).
4. Verify cyclone_schema_version contains the expected version row.
5. Deploy only binaries compatible with that schema. Do not assume reverting a binary is safe after migrating.

Version 11 rollout

Migration 11 adds cyclone_steps.lease_generation. New workers use it to fence a stale execution after a lease reclaim. A v0.1.0 worker does not include that generation in its completion CAS. That worker can bypass the fence. Stop every v0.1.0 worker and drain it fully. Then apply migration 11. Then start the upgraded workers. Do not use a mixed-version rolling deployment for this migration.

Version 14 rollout

Migration 14 adds cyclone_steps.reclaims, the counter that WithMaxReclaims bounds. It needs no drain. Workers built before it ignore the column. During a rolling deployment, the recovery sweeps of old workers reclaim without counting. Their operator retries do not reset the count. Only upgraded clients enforce a bound. Set the same WithMaxReclaims on every upgraded client that calls Start.

Version 15 rollout

Migration 15 adds cyclone_runs.result_node, the result node of a DefineResult workflow. Apply it before you deploy a binary built with it. The run insert of that binary names the column for every workflow, also for a workflow from Define. It needs no drain. A binary built before it does not name the column. Its run inserts leave the column NULL. Its workers claim, complete, and retry runs that have a result node. A result handle returns ErrNoOutput for a run with no result node. During a rolling deployment, a run that an old binary submits under the same workflow name has no result node.

If you change a workflow from Define to DefineResult, its def_version changes, because the result node is part of the fingerprint. Workflows with no result keep their def_version. A workflow from DefineResult has no schedule method, thus no schedule pin changes.

Version 17 rollout

Migration 17 adds the fan-out columns and indexes for ForEach. Apply it before you deploy a binary built with it. The engine of that binary reads the new columns in every run, also in a run with no fan-out node. It needs no drain. A binary built before it ignores the new columns. But its workers do not know a fan-out node and can execute one as an ordinary step. Upgrade every worker before you submit a workflow that uses ForEach.

Version 18 rollout

Migration 18 adds the paused state for runs and steps, and cyclone_runs.paused_at. It needs no drain, because a binary built before it never writes paused. It rebuilds idx_steps_unique_active and idx_steps_sequence. Each rebuild scans cyclone_steps while the migration holds its lock. Apply it, then deploy the new build to every worker, producer, and CLI before you pause a run. A process from an older build does not know the paused state. See Pause and Resume Runs.

Version 19 rollout

Migration 19 adds the parent link of a child run (cyclone_runs.parent_run_id and parent_node_id), cyclone_steps.child_workflow and child_needs_result, the waiting_child step state, and two partial indexes. Apply it before you deploy a binary built with it. The run reads of that binary name the new columns. It needs no drain, because a binary built before it writes no child node. Deploy the new build to every worker, every client that purges runs, and the CLI before you submit a workflow that uses Child. An older worker makes a child node available and discards it with no handler. An older purge can delete a child run that a failed parent needs on a retry. An older Cancel does not cancel the child runs of a run. See Run a Child Workflow.

Behavior Changes

Start Refuses a Client With No Handlers

Client.Start returns an error that matches ErrProducerOnly when the client cannot execute work. It refuses before it claims work or starts a goroutine. Start refuses these clients:

• A client from cyclone.New with no handlers. This is the producer-only mode.
• A client with an empty registry, also an empty registry given to NewClient. Before this change, such a client started. Its workers then discarded each step that they claimed, because no handler was registered.
• A client whose registry holds a RegisterKind declaration. Start refused this client before. Now the error also matches ErrProducerOnly.

A nil registry gives the same error as before.

A client with no handlers cannot execute only the background scanners now. Those scanners are the recovery sweep, the schedule scanner, the wait scanner, the deadline scanner, and the retention sweeper. Start them from a worker client that has handlers, or call PurgeTerminalRuns for retention.

cyclonetest.New registers an internal handler of a reserved kind. A test with no handler of its own still runs the scanners.

Step Kind Compatibility

Cyclone stores step kinds as strings in cyclone_steps.kind. Dispatch sends a claimed step to the handler registered for that kind.

A rename of a step kind is not safe while in-flight runs still name the old kind. The worker finds no handler, and the step fails.

Use WithVersion(n) for safe evolution:

// Original step — in-flight runs still reference kind "fetch"
var FetchStep = cyclone.NewStep("fetch", oldFetchFn)

// New version — new runs use kind "fetch@v2"
var FetchStepV2 = cyclone.NewStep("fetch", newFetchFn, cyclone.WithVersion(2))

Register both handlers in the mux. Old runs finish on the old handler. New runs use the new handler. When every old run is terminal, you can stop registering the old handler.

See Renaming and Versioning Steps for the full procedure.

Workflow Schedule Pinning

A workflow schedule records the definition fingerprint (def_version) of the DAG from its registration. The process that claims a fire rebuilds the DAG from its own registered builder. It refuses the fire when the fingerprint differs. Adjacent fires during a mixed-version deployment cannot get different graphs. ScheduleWorkflow re-pins on every registration. The upgrade path is to deploy the new build. Its startup ScheduleWorkflow call rewrites the pin.

The fingerprint covers topology only. It includes node ids, kinds, the dependency edge set, and wait declarations. It also includes OnDepFailure policies and each node's ordered fan-in bindings and assembly mode. The fingerprint excludes literal input values, timeouts, queues, priorities, and retry budgets. After(...) order does not matter.

Ordered bindings were added to the fingerprint after a gap. Join2(a, z) and Join2(z, a) have the same node ids, kinds, and edges. The edge set is sorted, so the two hashes were identical. A build that swapped a join's inputs passed the pin check and fired. The consumer then got its two arguments in the wrong order. The same gap applied to the slice order of FromAll. It also applied to a switch between Join2 and FromAll over the same sources.

Only a workflow with a join or FromAll changes its fingerprint. A workflow with no ordered bindings keeps the def_version it already has. It needs no new registration.

Rolling this change out

ScheduleWorkflow re-pins on every registration. The first upgraded process rewrites def_version for every join schedule while the rest of the fleet is still on the old build. From that moment, the old processes rebuild a different fingerprint and refuse the fire.

What happens next depends on the old build:

• The old build already treats a refusal as a deployment-topology condition. Cyclone leaves the slot alone. The refusal does not touch the poison counter. The first upgraded process fires the slot. Nothing is lost.
• The old build predates that behavior. A refusal increments consecutive_failures. The refusal holds the scanner lease for pollMax. After ten claims, Cyclone skips the slot and loses the fire. Early in a rollout, old processes outnumber new ones. That skip is then likely.

Release the refusal handling and deploy it fully before the release that changes the fingerprint. A release that changes the fingerprint is safe once every running process refuses without counting.

To upgrade across both changes at once, use one of two paths. Drain the old workers before the first upgraded process starts. Or pause the affected workflow schedules for the rollout.

Input/Output Compatibility

Cyclone stores step inputs as JSONB in cyclone_steps.input. An in-flight run carries the serialized input from the enqueue time.

Safe changes:

• Add a new field with a zero value or an explicit default. Old rows decode the field as the zero value.

Unsafe changes:

• Rename a field. Old rows carry the old name. The new field gets the zero value.
• Remove a required field. Old rows carry data that the new handler ignores. This change can be acceptable.
• Change the type of a field. Old rows then fail to decode.

For a breaking input change, version the input type and use WithVersion(n) on the step:

type FetchInputV2 struct {
    URL     string
    Headers map[string]string  // new field
    Timeout int                // renamed from MaxSeconds
}

var FetchStepV2 = cyclone.NewStep("fetch", handleFetchV2, cyclone.WithVersion(2))

See Changing Step Inputs for the full treatment.

Upgrade Checklist

When upgrading Cyclone to a new version:

• [ ] Read the release notes and changelog.
• [ ] Update go.mod: go get github.com/wafer-labs/cyclone@vX.Y.Z
• [ ] Run go mod tidy.
• [ ] Execute the test suite: go test ./... (or make test).
• [ ] Review any new migrations in the embedded SQL files.
• [ ] Follow any migration-specific drain or order requirement, then call cyclone.Migrate on a direct or session-mode connection.
• [ ] Monitor: watch cyclone_v_kinds and cyclone_v_queues for unusual patterns after rollout.

For upgrades whose release notes explicitly permit mixed worker versions:

1. Deploy the new binary alongside the old one (new handler registered in mux for new step version).
2. Let in-flight runs drain on the old binary.
3. Scale down the old binary.

Deprecation Policy

Pre-1.0: the source marks deprecated symbols with // Deprecated:. The next minor version can remove them. Check the release notes before you upgrade.

Post-1.0: Cyclone keeps a deprecated symbol for at least one minor version (for example, deprecated in v1.3 and removed no earlier than v1.4). The deprecation note includes a migration path.

To find deprecated symbols in your codebase after an upgrade:

go vet ./... 2>&1 | grep "deprecated"
# or
staticcheck ./... | grep "SA1019"