Cyclone

Appearance

Retries, Leases, and Recovery ​

This page describes a handler failure and a worker crash. It also describes the return to a healthy execution.

Why Retries Happen ​

A step can fail to complete for these reasons:

  1. The handler returned a non-nil error.
  2. The WithTimeout deadline of the handler passed.
  3. The worker process crashed, or the process was killed.
  4. The lease expired because the worker could not reach Postgres to renew it.
  5. Cyclone recovered a handler panic and treated the panic as an error.

Each cause has its own recovery path. The paths do not use the same rules.

Backoff Algorithm ​

When the handler returns a retryable error, and attempts remain, the step returns to available. The new scheduled_at is in the future:

text
scheduled_at = now() + backoff(attempt)

The default backoff function makes an exponential delay and adds jitter. The delay is delay(n) = min(1h, 1s * 2^(n-1)). Attempt 1 waits about 1 second. From attempt 13 on, the cap is 1 hour. 1s * 2^12 is 4096s. 4096s is already past that cap. The jitter is uniform on the range [delay/2, delay]. That range goes from half of the nominal delay to the full nominal delay. The range is not a band around the nominal delay. The jitter separates retries after one shared failure.

You can supply a different function:

go
cyclone.WithBackoff(func(attempt int) time.Duration {
    base := time.Duration(1<<uint(attempt)) * time.Second // 2s, 4s, 8s, ...
    if base > 5*time.Minute {
        base = 5 * time.Minute
    }
    return base
})

Permanent Failures ​

Permanent(err) marks an error as not retryable. The step moves to discarded at once. The step does not use more attempts.

go
if in.Email == "" {
    return cyclone.Permanent(fmt.Errorf("missing email"))
}

Use Permanent in these cases:

  • The input fails validation, and a later attempt would see the same input.
  • The response is HTTP 4xx from an API that uses 4xx for a client error.
  • The business error cannot be corrected by another attempt.

Do not use Permanent for a transient infrastructure error. A timeout, a refused connection, or an HTTP 5xx response is transient. Leave those errors retryable.

Lease Expiry ​

The lease is a claim for a limited time on a step in state running. While the worker process is alive, the heartbeat renews that claim. The duration of the handler does not stop the renewal. Renewal stops when the process dies or when the process loses Postgres.

  • The heartbeat stops.
  • lease_expiry passes, so now() > lease_expiry.
  • The recovery sweep finds state = 'running' AND lease_expiry < now().
  • The sweep reclaims the step. The sweep sets state to available. The sweep sets lease_owner to NULL. The sweep sets lease_expiry to NULL. The sweep does not change scheduled_at, so a worker can claim the step immediately.

A reclaim is crash recovery, not a handler failure. A reclaim does not increment the attempt count. A reclaim does not add backoff. The same attempt number starts again under a new lease, on the next poll.

This is the one recovery path that adds no delay of its own. A timeout is a failed execution. The step then pays backoff. A returned error is a failed execution. The step pays backoff for that error too. A reclaim does not pay backoff.

Heartbeat ​

Each claimed step gets its own heartbeat goroutine for the life of the attempt. That goroutine renews the lease of that one step every leaseTTL/3. The first delay has a random phase. Each renewal has ±10% jitter. Workers do not renew at the same time. The interval follows the TTL. WithPollInterval does not change that interval.

sql
UPDATE cyclone_steps
 SET lease_expiry = now() + ($1::bigint * interval '1 microsecond'),
 heartbeat_at = now()
 WHERE id = $2
 AND state = 'running'
 AND lease_owner = $3
 AND lease_generation = $4
 AND (timeout IS NULL OR started_at + timeout > now());

The renewal names the step by id. The renewal also requires the lease_generation from the claim. A worker renews nothing after another worker has reclaimed that step and claimed it. The started_at + timeout test stops renewal after the timeout of the step. When a renewal matches zero rows, the worker cancels the context of the handler.

The default leaseTTL is 30 seconds. The TTL limits how long the steps of a crashed worker stay stranded. Choose the TTL for the crash-recovery delay that you want. The length of the handler is not part of that choice. A handler of one hour keeps its lease on a 30-second TTL. A handler of one second does the same. The heartbeat continues to renew the lease while the process stays alive.

Worker Crash ​

SIGKILL, OOM, or loss of power can kill the worker process.

  1. Every goroutine dies at once. The process does no cleanup.
  2. Postgres closes every connection from this process. Postgres rolls back each open transaction.
  3. The heartbeat stops for every step that this process owns.
  4. After leaseTTL, the recovery sweep on any live client reclaims those steps.
  5. The steps start again with the same attempt number.

The recovery sweep repeats on the poll interval. Each pass takes at most 100 steps. A reclaimed step is claimable within leaseTTL + one sweep tick of the crash. A smaller TTL makes that wait shorter.

Bounding Crash Loops ​

A reclaim does not charge an attempt. WithMaxAttempts cannot stop a crash that repeats before the step completes. A handler can use all of the memory, hit a fatal runtime error, or call os.Exit. Each of those events stops the worker on every call of the handler. Every live client then reclaims the step again.

The sweep stores the reclaim count for the step in cyclone_steps.reclaims. StepStatus.Reclaims shows that count. WithMaxReclaims(n) puts a limit on the count:

go
client, _ := cyclone.NewClient(pool, mux,
    cyclone.WithMaxReclaims(3), // default 0: unbounded
)

The sweep can reclaim a step n times. On the next lease expiry, the sweep discards the step and does not reclaim it. The sweep sets error_kind to "reclaim_limit". The sweep does not charge an attempt. From that point, the step is a normal dead letter. The sweep emits discarded. The sweep calls the WithErrorHandler hook with cyclone.ErrReclaimLimitExceeded. The run fails, unless a dependent with OnDepFailure(CancelBranch) contains the discard.

The count stays in place across a normal retry. A step that crashes, then fails with an error, then crashes again, has two reclaims. An operator Retry or RetryAll sets the count back to 0. The sweep that finds the expired lease uses the limit of its own client. Set the same limit on every client that calls Start.

Every expired lease counts, not only a dead process. A worker that cannot reach Postgres also spends one reclaim when the lease ends. Set n higher than the reclaim count that a healthy step can meet. A deploy or a database failover can cause those reclaims.

Handler Panic ​

The worker goroutine catches a panic in a handler with recover(). Cyclone treats the panic as a retryable error. Cyclone increments the attempt count. The step returns to available. Cyclone applies the backoff. Cyclone stores the panic value in cyclone_steps.error.

Timeout ​

Cyclone applies WithTimeout on two clocks. The handler context carries a deadline from the worker clock. The handler can read ctx.Done() and return ctx.Err(), which is context.DeadlineExceeded. That error is retryable. Cyclone increments the attempt count.

The durable test uses the database clock, started_at + timeout. After that time, the heartbeat does not renew the lease. The recovery sweep then selects the step because the timeout has passed. The sweep does not wait for the lease to expire. The sweep moves the step to available. The sweep applies backoff. When no attempts remain, the sweep moves the step to discarded. If the handler does not stop, the durable timeout is about one sweep tick after the limit. The lease TTL does not change that timeout.

Cyclone does not stop the goroutine. Only the step state moves forward. When that handler returns, its completion does not match a running row of the same lease generation. The CAS guard changes zero rows. Cyclone drops that completion.

Manual Retry ​

An operator can open a discarded step again:

go
client.Retry(ctx, runID, "send_email",
    cyclone.RetryAdditionalAttempts(5),
    cyclone.RetryReason("SMTP restored"),
)

This call sets state to 'available'. This call raises max_attempts to at least attempt + RetryAdditionalAttempts. A step that already used 5 of 5 attempts then gets 5 more attempts. This call also sets reclaims to 0. A step that WithMaxReclaims moved to the dead letter gets its full reclaim budget again. The worker calls the handler again on the next attempt number.

client.RetryAll(ctx, runID) opens every discarded node in a failed run again.

Failure Propagation in DAGs ​

Cyclone does these actions when a discarded step has a downstream node on the default FailRun policy. Cyclone does the same actions when the discarded node is a leaf and has no dependents.

  1. The run moves to failed.
  2. Cyclone sets every non-terminal step in the run to cancelled.

The discard transaction does both UPDATE statements. The transaction holds the run row lock. In that same transaction, fail-fast cancellation of sibling steps occurs inside advanceRun.

This fail-fast stops the run from waiting in a partial state. The operator can then inspect the discarded step. The operator can call RetryAll to reopen the work that did not complete.

After client.RetryAll(ctx, runID):

The other path applies when every downstream node declares OnDepFailure(CancelBranch). Cyclone then contains the discard. Cyclone cancels only that branch. The other branches continue. The run can still complete.

Cyclone stores each run in Postgres.