Submit a Workflow Once

Use workflow.SubmitOnce when a caller can send the same request more than one time. An HTTP retry is an example. Each call gives the same key. Cyclone makes one run for the key. Each later call returns that run.

run, created, err := checkout.SubmitOnce(ctx, client, "order:"+orderID, order)
if err != nil {
    return err
}
if created {
    log.Printf("started run %s", run.ID())
} else {
    log.Printf("run %s already exists", run.ID())
}

created is true when this call inserted the run. It is false when a run with the key exists. In both cases, run is a handle for the run of the key.

An error does not always mean that no run exists. For example, the connection can fail after Postgres commits the run. Call SubmitOnce again with the same key. It returns the existing run or makes a new one.

Submit and SubmitTx do not change. They take no key. Each call makes a new run.

Workflows With a Typed Result

A workflow from cyclone.DefineResult has the same SubmitOnce and SubmitOnceTx methods. They return a *cyclone.ResultRun[Out]. A new run stores its result node, as with Submit. Result on the handle waits for the run and returns the result. This is true also when created is false.

run, _, err := pricing.SubmitOnce(ctx, client, "quote:"+quoteID, cart)
if err != nil {
    return err
}
total, err := run.Result(ctx)

A workflow from Define and a workflow from DefineResult with the same name use the same keys. The first submit makes the run. If that submit came from the Define workflow, the run has no result node. Then Result on the DefineResult handle returns cyclone.ErrNoOutput.

Submit in Your Transaction

workflow.SubmitOnceTx inserts the run in your transaction. The run exists only after the commit.

tx, err := pool.Begin(ctx)
if err != nil {
    return err
}
defer tx.Rollback(ctx)

if err := saveOrder(ctx, tx, order); err != nil {
    return err
}
run, created, err := checkout.SubmitOnceTx(ctx, client, tx, "order:"+orderID, order)
if err != nil {
    return err
}
if err := tx.Commit(ctx); err != nil {
    return err
}

• If the transaction rolls back, the key is free again.
• After your transaction inserts a new run, a submit with the same key in a different transaction waits. It continues when your transaction commits or rolls back. Keep the transaction short.
• A second call with the same key in the same transaction returns the run of the first call. created is false.
• A conflict does not abort the transaction. You can continue to use it.
• Commit before you read or wait on the handle.
• Use a READ COMMITTED transaction (the Postgres default). In REPEATABLE READ or SERIALIZABLE, a key that a different transaction commits after your snapshot gives a serialization error (SQLSTATE 40001).

Scope of a Key

Postgres keeps a key unique for each workflow name. The unique index idx_runs_submit_key on cyclone_runs enforces this, also for concurrent calls from different processes.

• Two workflows can use the same key. Each workflow name has its own keys.
• A key applies to runs in all states: running, completed, failed, and cancelled.
• A failed run keeps its key. A second SubmitOnce returns the failed run. created is false. To do the work again, call run.RetryAll, or use a new key.
• The workflow graph does not take part. After you deploy a new version of the workflow, the key still returns the run of the old version.

Same Key, Different Input

The run of the key must have the same input. Cyclone compares the encoded JSON of the two inputs as Postgres jsonb values. The order of object keys and white space do not matter.

If the inputs are different, SubmitOnce inserts nothing. It returns a *cyclone.SubmitConflictError. The error matches cyclone.ErrSubmitConflict. The error holds the ID of the existing run.

_, _, err := checkout.SubmitOnce(ctx, client, key, order)
var conflict *cyclone.SubmitConflictError
if errors.As(err, &conflict) {
    return fmt.Errorf("key %s belongs to run %s with a different order", key, conflict.RunID)
}

The submit options, for example WithRunDeadline, do not take part in the comparison. The existing run keeps the options of its first submit.

A new field in the input type can change the encoded JSON. Then a retry across that deploy gets a conflict for a run from before the deploy.

When a Key Is Free Again

PurgeTerminalRuns and the retention sweeper (WithRetentionSweep) delete terminal runs. When Cyclone deletes the run of a key, the key is free again. The next SubmitOnce with that key makes a new run.

A key prevents a duplicate only while Cyclone keeps the run. Keep these runs longer than the time in which a caller can send the request again. See Retention.

Key Rules

A key must not be empty. It must be valid UTF-8 without NUL, and at most 256 bytes. These are also the rules for WithUnique keys. A bad key returns an error before Cyclone writes to the database.

Make the key from the business request, for example an order ID or a webhook event ID. Do not use the HTTP request ID: each retry has a new request ID.

Recovery Workflows

Cyclone does not start a recovery workflow when a run fails. Your application finds the failed runs, for example with client.ListRuns. It submits a recovery workflow. Make the key from the failed run ID. Then the key prevents a duplicate recovery run:

recovery, created, err := undoCharge.SubmitOnce(ctx, client, "recover:"+string(failedID), failedID)

If two processes find the same failed run, they get the same recovery run. Keep the failed run until the recovery is complete. Set its retention so that the sweeper does not purge it first. Cyclone does not block Retry or RetryAll on the failed run. Your application must prevent them after the recovery starts.

SubmitOnce or WithUnique

	SubmitOnce	WithUnique
Applies to	Workflow runs	Standalone enqueues
Key is unique for each	Workflow name	Step kind
Runs that hold the key	All runs, until the purge	Active runs only
A second call with the key	Returns the existing run	Returns a *cyclone.DuplicateError

Related

• Submit DAG Workflows
• Unique Jobs
• Idempotency
• Methods and Package Functions