Retention

How to design a retention policy and manage Cyclone's storage footprint.

Retention Policy Design

The core trade-off:

Shorter retention	Longer retention
Less disk usage	More history for debugging
Faster vacuum (fewer live rows)	Auditable job history
Less data for post-mortem	Longer recovery window for missed re-runs

Start with retention that matches your debugging needs and tighten it once you understand your operational patterns.

A reasonable starting point:

• completed runs: keep 7 days (enough to debug recent issues)
• failed / cancelled runs: keep 30 days (post-mortem investigation)
• Short jobs with no output: keep 24 hours

Compliance and Audit Trade-Offs

If you are subject to audit requirements:

• Keep failed and discarded runs longer than completed ones. Failed runs carry error messages and input payloads that compliance review needs.
• You can export job history to a separate audit table with ctx.CompletionTx() before the purge.
• In a regulated environment, do not purge failed runs automatically. Require an operator review.

Automated Sweeps

WithRetentionSweep starts a background sweep inside Client.Start on a configurable interval. It is not PurgeTerminalRuns. The sweep honors each step's WithRetention with COALESCE(retention, DefaultAge). PurgeTerminalRuns applies one flat age cutoff to every run.

client, _ := cyclone.NewClient(pool, mux,
    cyclone.WithWorkers(8),
    cyclone.WithRetentionSweep(cyclone.RetentionConfig{
        DefaultAge: 30 * 24 * time.Hour,
        Interval:   10 * time.Minute,
        BatchSize:  1000,
    }),
)

The sweeper only deletes runs where:

1. The run is in a terminal state (completed, failed, cancelled).
2. now() > run.finished_at + max(step.retention across all steps in run).

A paused run is not terminal. Neither the sweeper nor PurgeTerminalRuns deletes it, however long it stays paused.

Neither the sweeper nor PurgeTerminalRuns deletes a terminal child run until its parent run completes or is deleted. A running, paused, or failed parent run can still need its result. A retry of the parent run reuses that result. A cancelled parent run starts again when a retry of its own parent reopens it. A parent run that you never retry keeps its child runs until you purge the parent run.

A step with no WithRetention uses RetentionConfig.DefaultAge. If DefaultAge is zero, the automated sweeper is disabled.

Manual Purge

deleted, err := client.PurgeTerminalRuns(ctx, 7*24*time.Hour, 1000)
if err != nil {
    log.Printf("purge error: %v", err)
}

Cyclone measures the age argument against finished_at. PurgeTerminalRuns repeats bounded deletes internally until no eligible rows remain, then returns the total deleted count. Each batch is its own transaction. The call as a whole is not atomic. Keep the batch size small (500-2000) so a transaction does not stay open and delay vacuum.

Submit Keys

A run from workflow.SubmitOnce keeps its key until Cyclone deletes the run. PurgeTerminalRuns and the sweeper free the key. After that, a submit with the same key makes a new run. Keep these runs longer than the time in which a caller can send the request again. See Submit a Workflow Once.

Per-Step Retention

Set retention at the step definition level to apply to all enqueues:

var QuickStep = cyclone.NewStep("quick_job", fn,
    cyclone.WithRetention(24 * time.Hour),
)

Or override per-enqueue:

cyclone.Enqueue(ctx, client, QuickStep, in,
    cyclone.WithRetention(72 * time.Hour),
)

The purge sweeper uses max(retention across all steps in a run). If a run has two steps, one with 24h retention and one with 72h, the run is eligible for purge after 72h.

Output Retention

Cyclone stores step outputs in cyclone_steps.output. When Cyclone purges a run, it deletes every step row of that run, including the outputs. After the purge, run.Result, run.Output, and cyclone.OutputNode return ErrRunNotFound.

If you need to preserve outputs:

1. Read the outputs and store them before the retention purge. Use a scheduled job that queries cyclone_v_runs for runs that expire soon.
2. Or write them to your application tables through ctx.CompletionTx() during step completion, before the step's retention expires.

Monitoring Retention Health

-- Runs eligible for purge now:
SELECT count(*) as eligible_for_purge
FROM cyclone_runs r
WHERE r.state IN ('completed', 'failed', 'cancelled')
  AND NOT EXISTS (
      SELECT 1 FROM cyclone_steps s
      -- COALESCE, not "retention IS NULL": a step with no per-step retention is
      -- eligible after the sweeper's DefaultAge, not never. Substitute your
      -- configured DefaultAge for the interval below.
      WHERE s.run_id = r.id AND r.finished_at + COALESCE(s.retention, interval '24 hours') > now()
  );

-- Disk usage by table:
SELECT relname, pg_size_pretty(pg_total_relation_size(relid)) as total_size
FROM pg_stat_user_tables
WHERE relname LIKE 'cyclone_%'
ORDER BY pg_total_relation_size(relid) DESC;

Related

• Retention and Purging
• Database Maintenance