Retries

Cyclone retries an ordinary error. Use this for an API that fails intermittently. Use this for a temporary database failure. Use this for a provider that returns status 500.

Use Permanent for bad input. Five attempts on a missing email address still leave it missing.

Default Behavior

Every step gets 5 attempts by default. A failed attempt waits before the next attempt. The wait starts at a few seconds. The wait grows to about one hour. The wait includes jitter.

var DeliverWebhook = cyclone.NewAction("deliver_webhook",
    func(ctx cyclone.Context, in WebhookDelivery) error {
        resp, err := http.Post(in.URL, "application/json", bytes.NewReader(in.Body))
        if err != nil {
            return err
        }
        defer resp.Body.Close()

        if resp.StatusCode >= 500 {
            return fmt.Errorf("webhook receiver returned %d", resp.StatusCode)
        }
        if resp.StatusCode >= 400 {
            return cyclone.Permanent(fmt.Errorf("bad webhook request: %d", resp.StatusCode))
        }
        return nil
    })

Cyclone retries a network error. Cyclone retries status 500. Cyclone treats status 400 as permanent. The payload is probably wrong.

Attempts

ctx.Attempt() starts at 1. The value increases after a failed attempt, on the next execution of the handler.

if ctx.Attempt() > 1 {
    slog.Info("retrying webhook", "attempt", ctx.Attempt(), "url", in.URL)
}

Crash recovery is different. When a worker stops and the lease is reclaimed, Cyclone starts that same attempt again. The attempt count does not change. The worker stopped. That stop is not a failed attempt.

Limit Attempts

Set a default on the step:

var SendEmail = cyclone.NewAction("send_email", sendEmail,
    cyclone.WithMaxAttempts(5),
)

Or set a different limit on one enqueue:

_, err := cyclone.EnqueueAction(ctx, client, SendEmail, in,
    cyclone.WithMaxAttempts(3),
)

The enqueue option wins when both are set.

n must be at least 1. 1 means one execution. 1 means no retry. Cyclone rejects a value that is not positive. NewStep panics. NewAction panics. Inside Do or Run, the value is a builder error. Every enqueue function returns an error before an insert. Cyclone accepted this value before. A step with max_attempts = 0 was discarded on the first failure. The error kind was retryable. No retry had occurred.

Change Backoff

Use WithBackoff when the default curve is wrong for the system you call.

var SendSMS = cyclone.NewAction("send_sms", sendSMS,
    cyclone.WithBackoff(func(attempt int) time.Duration {
        return 30 * time.Second
    }),
)

The function receives the 1-based ordinal of the execution that failed. Return the wait before the next claim.

For a provider API, use a capped exponential backoff:

cyclone.WithBackoff(func(attempt int) time.Duration {
    d := time.Duration(1<<uint(attempt)) * time.Second
    if d > 5*time.Minute {
        return 5 * time.Minute
    }
    return d
})

Snooze — Retry Without Consuming an Attempt

Use cyclone.Snooze when the handler knows the exact delay. Do not use a retry backoff for that delay.

var CallRateLimitedAPI = cyclone.NewAction("call_api",
    func(ctx cyclone.Context, in APIInput) error {
        resp, err := callAPI(ctx, in)
        if err != nil {
            return err // retries with backoff
        }
        if resp.StatusCode == 429 {
            retryAfter := resp.RetryAfter()
            return cyclone.Snooze(retryAfter)
        }
        return nil
    })

Snooze schedules the step at now() + d. Snooze does not increment the attempt counter. The next claim starts the same attempt ordinal. Cyclone rejects a zero duration as permanent. Cyclone rejects a negative duration as permanent.

Use Snooze for these cases:

• A rate-limited upstream with a Retry-After header.
• A resource that is not ready yet, for example a cloud resource that is still in provisioning.
• Backpressure when the caller knows the recovery time.

Each snooze records a snoozed event. WithOnStepEvent shows that event. Subscribe shows that event.

Stop Retrying

Wrap an error with cyclone.Permanent when another attempt cannot help.

var ValidateInvoice = cyclone.NewStep("validate_invoice",
    func(ctx cyclone.Context, in Invoice) (ValidatedInvoice, error) {
        if in.Email == "" {
            return ValidatedInvoice{}, cyclone.Permanent(fmt.Errorf("email is required"))
        }
        if in.TotalCents <= 0 {
            return ValidatedInvoice{}, cyclone.Permanent(fmt.Errorf("total must be positive"))
        }
        return ValidatedInvoice{Invoice: in}, nil
    })

A permanent error moves the step to discarded. In a workflow, one discarded node fails the run. Cyclone cancels each sibling that is not finished. This does not happen when every node downstream of the discarded node declares OnDepFailure(CancelBranch). That policy keeps the failure in that branch.

Inspect Discarded Steps

The Go constant for this state is cyclone.StepExhausted. The database stores "discarded". The CLI shows "discarded".

steps, err := client.ListSteps(ctx, runID)
if err != nil {
    return err
}

for _, step := range steps {
    if step.State == cyclone.StepExhausted {
        fmt.Printf("%s failed after %d attempts: %s\n",
            step.Node, step.Attempts, step.Error)
    }
}

Retry After a Fix

When the problem was external, such as an SMTP outage, open the discarded node again.

err := client.Retry(ctx, runID, "send_email",
    cyclone.RetryAdditionalAttempts(3),
    cyclone.RetryReason("SMTP provider recovered"),
)

To retry every discarded node in a failed run:

err := client.RetryAll(ctx, runID,
    cyclone.RetryAdditionalAttempts(3),
    cyclone.RetryReason("operator retry after provider outage"),
)

The CLI has the same operations:

cyclone retry 12345 send_email
cyclone retry 12345 --all

Safety Rule

A retry can start a handler more than once. Make the effect idempotent.

For a Postgres write, use ctx.CompletionTx(). Use a unique key such as (run_id, node). For an external API, pass an idempotency key when the API supports one.

Next

• Exactly-Once Effects
• Failure Recovery
• Signals and Waits
• Unique Jobs