Engine Design

This page tells why Cyclone uses static DAGs, leases, and CAS guards. It also tells why Cyclone uses the ctx.CompletionTx() wedge. It lists the operational trade-offs.

Why Postgres

Cyclone stores state in the same Postgres database as the application data. That choice makes two properties possible:

1. A caller can enqueue a job in the same transaction as the business data for that job. There is no dual-write window.
2. ctx.CompletionTx() gives exactly-once effects. A business change commits in the same transaction as the step completion. There is no separate commit protocol.

The capacity of one Postgres instance limits the throughput.

Why Static DAGs

A Cyclone workflow is a static directed acyclic graph. You declare the graph in Go. Submit fixes which nodes exist and which node depends on which node. Cyclone writes the whole graph to Postgres in one transaction. Workers then advance the run node by node. Workers do not call the build function again to restore state.

A static DAG gives these properties:

• There is no replay, so there is no determinism requirement. Handlers are ordinary functions.
• The graph is serializable data in cyclone_deps. You can inspect the graph.
• Cyclone checks the topology at submit time. Cyclone does not check the topology again at claim time.

The topology cannot change after submit. Cyclone does not support a conditional branch. Cyclone does not support a long sleep. A fan-out over a list from a step is one declared node. During the run, the engine makes item tasks for that node. The engine inserts those tasks in the transaction that completes the upstream step. See Fan Out Over a List. A child node is also one declared node. The child node starts a separate run of a second workflow and waits for the result. See Run a Child Workflow.

Why No Workflow Replay

Replay requires deterministic handler code. Each new call of the workflow function must give the same command sequence. That rule excludes a random number. That rule excludes a wall clock. That rule excludes a side effect. Replay also needs careful history versioning.

Cyclone calls the build function once at submit time. Workers call each step as an ordinary function. The state machine in Postgres is the source of truth. A history log is not the source of truth.

That model is easier to explain. That model is easier to debug. That model does not constrain the handler code.

The Claiming Algorithm

The worker loop does these steps on each poll. The worker loop also does these steps on each NOTIFY.

1. The worker connects to Postgres.
2. The worker does this query:

SELECT ... FROM cyclone_steps WHERE state = 'available' AND queue = $queue AND scheduled_at <= now() FOR UPDATE SKIP LOCKED LIMIT 1

3. SKIP LOCKED skips a row that another worker already locked. Workers on that queue do not block each other.
4. If the query finds a row, the worker does a CAS UPDATE. The UPDATE sets the state to running. The UPDATE sets lease_expiry and lease_owner.
5. The CAS guard includes WHERE state = 'available' AND .... Two workers can race on the same row. Only one UPDATE changes that row.
6. The worker starts a heartbeat goroutine for this step.
7. The worker calls the handler.
8. On completion, the worker opens the completion transaction. The worker commits the output and the state change.

CAS State Transitions

Every state transition uses a compare-and-swap guard:

-- claimable selects one 'available' row in this worker's queue ($3),
-- ordered by priority, under FOR UPDATE SKIP LOCKED.
UPDATE cyclone_steps s
SET state = 'running',
    lease_owner = $1,
    lease_expiry = now() + ($2::bigint * interval '1 microsecond'),
    heartbeat_at = now(),
    started_at = now(),
    lease_generation = lease_generation + 1
FROM claimable c
WHERE s.id = c.id

The worker passes a TTL, not a deadline. Postgres computes lease_expiry and started_at. One database clock serves the whole lease model.

A transition from the worker also compares the lease_generation from the claim. The update changes zero rows when the row has moved on. The same is true when the step was reclaimed and claimed again. Cyclone treats that result as a benign stale transition. Cyclone rolls back a stale CompletionTx together with its business writes.

These guards protect the engine from these races:

• Two workers race on the same step.
• A late completion from a crashed handler arrives after the recovery sweep.
• A cancel races with a completion.

Leases and Heartbeats

A pinned database connection does not hold a step. Cyclone uses a lease. When a worker claims a step, the worker writes:

• lease_expiry = now() + leaseTTL
• lease_owner = clientID
• lease_generation = lease_generation + 1

One heartbeat goroutine for each claimed step renews that lease every leaseTTL/3. The renewal continues while the handler executes.

UPDATE cyclone_steps
 SET lease_expiry = now() + ($1::bigint * interval '1 microsecond'),
 heartbeat_at = now()
 WHERE id = $2
 AND state = 'running'
 AND lease_owner = $3
 AND lease_generation = $4
 AND (timeout IS NULL OR started_at + timeout > now());

The last guard makes WithTimeout on that step durable. After started_at + timeout, the heartbeat does not renew the lease. The recovery sweep selects a step on lease_expiry < now() or on started_at + timeout <= now(). The sweep takes the step even when the lease is still fresh.

If the worker crashes:

• The heartbeats stop with the process.
• The lease is not renewed. Then lease_expiry passes.
• The recovery sweep inside every Client.Start loop finds running steps with lease_expiry < now(). The sweep sets those steps back to available. The sweep clears the lease. The sweep does not change scheduled_at. A worker can claim the step immediately.

A handler that continues for one hour holds no Postgres connection in the function body. The handler needs a connection only for ctx.CompletionTx() at completion.

The lease TTL is crash-detection time, not a handler time limit. Renewal continues for the whole attempt. The length of the handler is not part of the TTL.

Crash recovery is automatic. The next sweep on any live client reclaims the steps of a crashed worker.

Clock skew is not a concern. The claim, the renewal, and the sweep all read the database now().

A crash is not a failed attempt. A reclaim does not increment the attempt count. A reclaim does not add backoff. The same attempt starts again under a fresh lease. The sweep counts each reclaim in cyclone_steps.reclaims. WithMaxReclaims bounds that count. A handler can stop its worker on every call. The sweep then dead-letters the step with error kind reclaim_limit. The step does not repeat without a limit.

ctx.CompletionTx(): The Exactly-Once-Effects Wedge

A step handler can write a business change in the completion transaction of the step.

var Charge = cyclone.NewAction("charge", func(ctx cyclone.Context, in ChargeInput) error {
    tx := ctx.CompletionTx()
    _, err := tx.Exec(ctx,
        `INSERT INTO ledger (run_id, node, amount)
         VALUES ($1, $2, $3)
         ON CONFLICT (run_id, node) DO NOTHING`,
        ctx.RunID(), ctx.Node(), in.Amount)
    return err
})

When the handler returns, Cyclone commits the completion transaction. One of these results follows:

• The INSERT INTO ledger and the step transition to completed both commit.
• Or neither of them commits. If the worker crashes before the commit, both changes roll back. The lease of the step lapses. The recovery sweep reclaims the step. The step starts again with a new ctx.CompletionTx().

For a side effect that you can write in Postgres, this gives exactly-once effects. The calls of the handler are still at least once. You do not need an external idempotency token. ON CONFLICT DO NOTHING on (run_id, node) removes a duplicate retry in the database.

Cyclone owns the completion transaction. That transaction uses the same Postgres database as the application data.

LISTEN/NOTIFY and the Polling Floor

Workers sleep between polls. After the creating transaction commits, Cyclone sends a Postgres NOTIFY on cyclone_available. Cyclone sends that NOTIFY when a new job is enqueued or a step becomes available. The NOTIFY is never inside the creating transaction. The JSON payload names the queue. A full global notification queue in Postgres makes that transaction fail at commit. An in-transaction NOTIFY can reject a state change that is otherwise valid.

Workers LISTEN on that one channel. Each worker forwards only the wakeups for the queue that it serves. A second channel, cyclone_run_done, carries completion wakeups for Wait. A separate channel for each queue would put every queue name into the Postgres channel namespace. A new queue would then force a new LISTEN.

In the common case, enqueue-to-claim latency is sub-millisecond. A short poll interval is not required for that latency. The poll loop is the correctness floor. Publication is best-effort. The next poll recovers a failed notification. The next poll also recovers a dropped notification. Cyclone loses no work when LISTEN/NOTIFY is unavailable for a time.

A caller-owned transaction sends no notification. SubmitTx, EnqueueTx, and the other caller-owned transaction calls are in that group. Cyclone does not observe the caller commit. A poll finds that work within one poll interval. The caller can record the transaction with TrackWakeups. The caller then calls Client.PublishWakeups after the commit. Work from EnqueueCompletion or SubmitCompletion uses the completion transaction of the step. After that commit, the worker publishes that wakeup together with the completion wakeup. See Transactional Enqueueing.

WithNotify(false) disables the listener of this client and the Wait LISTEN path. The client then claims by poll only. Poll-only claim is useful behind a transaction-mode PgBouncer, or where LISTEN is not permitted. WithNotify(false) does not stop publication. The client still posts wakeups that wake listeners in other processes. WithNotifyCoalesce folds many wakeups into fewer NOTIFY calls when producers are busy. See Client Options and Queue Sizing. Those pages list the ways a wakeup can be lost. Those pages also say what recovers a lost wakeup.

Output Resolution

Cyclone stores each completed output as JSONB in cyclone_steps.output. When a worker claims a downstream step, Cyclone reads the upstream output by (run_id, node). An in-process LRU cache skips a repeated read of the same upstream output. The cache holds at most 10,000 entries and 64 MiB of JSON. Those limits apply for the life of the client. Cyclone does not cache one output that is larger than the byte budget.

Each client has its own cache. Processes do not share that cache.

Failure Propagation

When a step reaches discarded, Cyclone resolves the run under the run row lock. That work is in the same transaction as the discard. Cyclone applies the OnDepFailure policy of each dependent. Fail-fast cancellation of sibling steps occurs inside advanceRun, in that transaction.

The discard is fatal in either of these cases:

• A node downstream of the discard declares FailRun. FailRun is the default.
• The discarded node is a leaf and has no dependents.

Cyclone then does these actions:

1. Cyclone marks the run failed.
2. Cyclone cancels every non-terminal step in the run. That cancel is a second UPDATE, not the same UPDATE. The discard transaction does both UPDATE statements.
3. A sibling that is still in the handler learns about the cancel on its next heartbeat. The CAS guard ignores the late completion.

A run can fail-fast only one time. The UPDATE from running to failed changes one row. A later discard finds the run already terminal and does not change the run.

The discard is contained when every downstream node declares OnDepFailure(CancelBranch). Cyclone then cancels only that branch. The run stays in state running. The run completes when no non-terminal step remains. Fail-fast is the default. Fail-fast is not the only result.

Fail-fast stops a wait on a step that cannot complete after a sibling fails permanently. CancelBranch limits that effect to one branch when a wide DAG can accept a partial result.

Pool Budget

NewClient checks the pool size against this rule:

MaxConns >= txSlots + 6

The +6 is a shared reserve for these tasks:

• The claim loop, on short connections, for polling and claiming.
• The recovery sweep.
• The schedule scanner.
• The wait scanner and the deadline scanner.
• The LISTEN/NOTIFY listener, which holds one connection for the life of the client.
• Spare capacity for separate GetRun, ListSteps, and Cancel calls.

txSlots limits how many completion transactions can be open together. The default is the worker count. With those defaults, a client needs at least workers + 6 pool connections. Lower WithTxSlots when fewer handlers need ctx.CompletionTx() at the same time.

Schema Isolation and Migrations

tern applies the migrations from embedded SQL files. Each migration is safe to apply more than once. A call to Migrate at every process start is safe. Cyclone skips a migration that is already applied. The table cyclone_schema_version records each applied version.

fillfactor = 90 on cyclone_steps and cyclone_runs leaves free space in each page. An eligible UPDATE can then be a HOT update. HOT means a heap-only tuple update. A change to an indexed column still updates the index. A change to a partial-index predicate still updates the index. Many changes to state are in that group. Many changes to lease_expiry are in that group.

Trade-Offs

Property	Cyclone behavior
Throughput	One Postgres instance sets the limit
Dynamic workflows	Submit fixes the topology
Worker language	Go workers only
Transactional enqueue	Postgres transactions support it
Exactly-once Postgres effects	ctx.CompletionTx() supports them
Operational complexity	Postgres is the one stateful dependency

Related

• Mental Model
• Lifecycle of a Run
• Idempotency
• Postgres Architecture