Mental Model

This page shows the core abstractions in Cyclone. It shows how those abstractions fit together.

Runs, Steps, and Actions

A run is one execution. Cyclone stores the run in cyclone_runs. The run holds one or more steps or actions. Each step or action is a row in cyclone_steps.

• A step is a typed handler (Context, I) → (O, error). Cyclone stores output O as JSONB. Downstream steps and callers can read it after completion.
• An action is a typed handler (Context, I) → error. An action has no stored output. Use an action for a side effect such as mail, a ledger write, or an HTTP call.

A step and an action are Go functions with a name and options. The only difference is whether output is stored.

Workflows as Static DAGs

A workflow is func(*Builder, In) error. You wrap that function with cyclone.Define. workflow.Submit is the method form of cyclone.Submit. One call of cyclone.Submit or workflow.Submit calls the builder one time. The builder calls step.Do or cyclone.Do to add a step. The builder calls action.Do or cyclone.Run to add an action. Each node declares its input. cyclone.Lit supplies a literal. cyclone.From supplies a data dependency on an upstream step. The result is a directed acyclic graph.

cyclone.DefineResult wraps a workflow with a typed result. The build function has the form func(*Builder, In) (*Future[Out], error). The returned future selects the result node. The cyclone_runs row stores the ID of that node. The run handle reads the output of that node.

Cyclone writes that graph to Postgres in one transaction. The transaction inserts the cyclone_runs row. The transaction inserts every cyclone_steps row and every cyclone_deps row. Workers then advance the run. When a step completes, its downstream steps become available.

Submit fixes the topology. You cannot add or remove nodes while a run is in progress. The graph has no dynamic branch. The graph has no loop.

One kind of node has an amount of work that a runtime value sets. That node is the fan-out node of ForEach. When its upstream step returns a list, the engine adds one item task for each item. The node itself is fixed at submit. See Fan Out Over a List.

A child node starts a second workflow as a separate run, the child run. Cyclone calls the builder of the child workflow when the input of the node exists. A runtime value can choose the graph of the child run. The child node itself is fixed at submit. See Run a Child Workflow.

Standalone Jobs as One-Node Runs

step.Enqueue is the method form of cyclone.Enqueue. action.Enqueue is the method form of cyclone.EnqueueAction. Each call creates a run with one step. That step has no dependency. Cyclone inserts the step as available. The step then moves to running. The step then moves to completed or discarded. The step uses the same lease recovery, retries, and observability hooks as a workflow step.

"Workflow" and "standalone job" are the same execution model. One-node runs are the short form.

State Lives in Postgres

Durability comes from Postgres ACID. cyclone_steps is the source of truth for the steps that exist. cyclone_steps stores the state of each step. cyclone_steps stores the input and the output of each step. cyclone_steps stores the time when each lease expires.

Workers are stateless. A worker reads a step from Postgres. The worker claims the step. The worker calls the handler. The worker writes the result back. A worker process can die at any point. The system recovers without data loss.

Cyclone has no in-memory queue. Cyclone has no external broker. The database is the queue.

Workers Claim Rows

Workers poll cyclone_steps for available steps with SELECT ... FOR UPDATE SKIP LOCKED. SKIP LOCKED means a worker never waits on a locked row. The worker skips that row and takes the next row. Workers that claim the same queue do not block each other.

Each claim is a compare-and-swap. The UPDATE includes WHERE state = 'available'. If two workers race on the same row, one wins. The other updates zero rows and moves on.

Leases Make Crashes Recoverable

When a worker claims a step, the worker writes:

• lease_expiry = now() + leaseTTL (default 30 seconds)
• lease_owner = clientID

Each claimed step gets its own heartbeat goroutine. The heartbeat renews lease_expiry every leaseTTL/3 while the handler executes. If the worker process crashes, heartbeats stop. Then lease_expiry is earlier than now().

Every Client.Start loop does a recovery sweep. The sweep finds steps in state running with lease_expiry < now(). The sweep reclaims those steps and sets the state back to available. Another worker can then claim the step. A reclaim does not increment the attempt count. A reclaim does not add backoff. A crash is not a handler failure. The step is claimable immediately. A reclaim does not charge an attempt. Without a bound, a handler that crashes its worker on every call repeats with no end. WithMaxReclaims bounds the reclaims. Past that bound, the sweep dead-letters the step.

Consequences:

• A long handler does not hold a database connection.
• The lease TTL is crash-detection time, not a handler time limit. A handler can continue for hours. The handler keeps its lease on the default TTL. A shorter TTL makes the steps of a dead worker available sooner.
• Crash recovery is automatic for most cases.
• Clock skew is not a concern. Each comparison uses the database clock.

Transactions Make Effects Atomic

A handler receives ctx.CompletionTx(). ctx.CompletionTx() is a pgx transaction. Cyclone commits that transaction as the completion transaction of the step. A business write on ctx.CompletionTx() commits with the step transition to completed.

The business write and the step completion commit together, or neither commits. If the worker crashes before the commit, the transaction rolls back. The sweep then reclaims the step. The worker calls the handler again from the start.

For a Postgres side effect, ctx.CompletionTx() gives exactly-once effects from at-least-once calls. You do not need external idempotency infrastructure.

var Charge = cyclone.NewAction("charge", func(ctx cyclone.Context, in ChargeInput) error {
    _, err := ctx.CompletionTx().Exec(ctx,
        `INSERT INTO ledger (run_id, node, amount)
         VALUES ($1, $2, $3)
         ON CONFLICT (run_id, node) DO NOTHING`,
        ctx.RunID(), ctx.Node(), in.Amount)
    return err
})

For a side effect outside Postgres, use (ctx.RunID(), ctx.Node()) as the idempotency key. HTTP, email, and a third-party API are outside Postgres. The key is stable across retries and reclaims for the same node. See Idempotency.