At-Least-Once Execution and Idempotency

Cyclone calls each handler at least once. A handler must accept more than one call. That contract lets Cyclone continue after a worker crash, a network partition, or a lease expiry. Cyclone does not lose the work.

At-Least-Once Execution

A worker claims a step under a row lock and calls the handler. A separate transaction marks the step terminal. The worker can die between the claim and that commit. The process can be killed, the host can reboot, or the network can drop. Then the lease expires. The recovery sweep makes the step claimable again. Another worker calls the handler again.

The worker can call a handler more than once when:

• the worker crashes after the side effect and before the completion transaction commits;
• a lease expires under load and the recovery sweep reclaims the step;
• a non-cooperative handler continues past its WithTimeout. A second attempt then starts.

In each case, the external side effects of the first call can already exist. Cyclone cannot un-send an email or un-charge a card. The handler must be ready for that.

Exactly-Once Effects

A side effect on ctx.CompletionTx() commits in the completion transaction of the step. The business write and the transition to completed commit together, or neither commits. A crash before the commit rolls back the business write and the step. The next call starts from a clean state.

var Charge = cyclone.NewAction("charge", func(ctx cyclone.Context, in ChargeInput) error {
    tx := ctx.CompletionTx()
    _, err := tx.Exec(ctx, `
        INSERT INTO ledger (run_id, node, amount)
        VALUES ($1, $2, $3)
        ON CONFLICT (run_id, node) DO NOTHING`,
        ctx.RunID(), ctx.Node(), in.Amount)
    return err
})

For a side effect that you can write in Postgres, ctx.CompletionTx() gives exactly-once effects. The handler calls are still at least once. A side effect outside Postgres needs an idempotency key. An HTTP call, mail, or a charge on a third-party API is outside Postgres.

Idempotency Keys: (runID, node)

Use (runID, node) when you must remove a duplicate effect. One case is your own write that does not use ctx.CompletionTx(). Another case is a downstream API that accepts an idempotency token.

(runID, node)

Do not use (runID, node, attempt). The attempt number changes on a retry and on a lease reclaim. A key with the attempt makes each later call look new. That key does not remove duplicates. (runID, node) stays the same for the life of the node. A retry, a timeout, and a lease reclaim all call that same node. Each of those calls carries the same key.

var SendReceipt = cyclone.NewAction("send_receipt", func(ctx cyclone.Context, in Receipt) error {
    key := ctx.RunID() + ":" + ctx.Node()
    return mailer.Send(ctx, in.Email, in.Body, stripe.IdempotencyKey(key))
})

The handler context gives ctx.RunID(), ctx.Node(), and ctx.Attempt(). Use RunID and Node for the idempotency key. Use Attempt only for logging and metrics.

External API Calls

Some external APIs accept an idempotency key. Stripe, SendGrid, and Twilio are examples.

var SendEmail = cyclone.NewAction("send_email", func(ctx cyclone.Context, in EmailInput) error {
    idempotencyKey := ctx.RunID() + ":" + ctx.Node()
    return emailClient.Send(ctx, sendgrid.Request{
        IdempotencyKey: idempotencyKey,
        To:             in.To,
        Subject:        in.Subject,
        Body:           in.Body,
    })
})

For an API with no idempotency key, check the key and then do the action.

var ProvisionUser = cyclone.NewAction("provision", func(ctx cyclone.Context, in ProvisionInput) error {
    key := ctx.RunID() + ":" + ctx.Node()
    if alreadyProvisioned(ctx, key) {
        return nil // already done in a previous attempt
    }
    return provisionUser(ctx, in, key)
})

Database Side Effects Without ctx.CompletionTx()

You sometimes write to Postgres outside ctx.CompletionTx(). A separate connection pool is one example. A table that the completion transaction cannot reach is another example. Use ON CONFLICT DO NOTHING with the key (run_id, node).

var AuditLog = cyclone.NewAction("audit", func(ctx cyclone.Context, in AuditInput) error {
    _, err := appPool.Exec(ctx, `
        INSERT INTO audit_log (run_id, node, action, ts)
        VALUES ($1, $2, $3, now())
        ON CONFLICT (run_id, node) DO NOTHING`,
        ctx.RunID(), ctx.Node(), in.Action)
    return err
})

This write is less safe than ctx.CompletionTx(). The audit row stays when the handler returns success and the completion transaction rolls back. Prefer ctx.CompletionTx() when the write and the completion must commit together.

Transactional Completion

When the handler returns nil, Cyclone does these actions:

1. Cyclone marks the step completed in cyclone_steps.
2. Cyclone stores the output, when the step has output, in cyclone_steps.output.
3. Cyclone makes the downstream steps available.
4. If every step is done, Cyclone marks the run completed.

Cyclone does all of these writes in the same transaction as ctx.CompletionTx(). The commit is atomic. Every write commits, or every write rolls back.

Retries Preserve Node Identity

An operator can reopen a discarded node in place. The calls are Client.Retry, Client.RetryAll, and the cyclone retry CLI. The node keeps the same run_id and the same node. Cyclone increments the attempt. Cyclone does not clone the node. Cyclone does not make a new id. A retried job uses the same (runID, node) key as the earlier attempts. An ON CONFLICT (run_id, node) DO NOTHING write still removes duplicates after an operator requeue. An external idempotency token does the same. An automatic retry uses that same check.

Crash Matrix

Scenario	Business effect happened?	Step state after crash	What Cyclone does
Crash before side effect	No	running, then the lease expires	The recovery sweep reclaims the step. The worker calls the handler again.
Crash after side effect, before ctx.CompletionTx() commit	Yes, outside the transaction. No, on ctx.CompletionTx().	running, then the lease expires	The recovery sweep reclaims the step. The worker calls the handler again. ON CONFLICT or the idempotency key removes the duplicate.
Crash after ctx.CompletionTx() commit	Yes. The commit is atomic.	completed	No second call is required. The step is already terminal.
Lease expires (worker alive but slow)	Possible	running, then available after the reclaim	The recovery sweep reclaims the same attempt. The worker calls the handler again.
Handler panics	No. Cyclone recovered the panic.	Retryable error	Cyclone increments the attempt and applies backoff. The next available claim calls the handler again.

Idempotent Submits

The sections above make a handler safe for more than one call. A caller can also submit the same workflow more than once. An HTTP timeout is one example. workflow.Submit is the method form of Submit. Each call of Submit or workflow.Submit makes a new run. workflow.SubmitOnce takes a key. That call makes at most one run for each workflow name and key. A second call returns the existing run. See Submit a Workflow Once.

Summary

• Assume that the worker can call every handler more than once.
• For a Postgres side effect, write it on ctx.CompletionTx(). That write gives an exactly-once effect.
• For a side effect outside Postgres, remove duplicates with (runID, node). Do not put attempt in that key.
• Make each external call idempotent. You can use a token, ON CONFLICT DO NOTHING, or an upsert.
• Use ctx.Attempt() only for observability. Do not put ctx.Attempt() in an idempotency key.