Cyclone

Appearance

Benchmarks ​

How to run and interpret Cyclone's benchmark suite.

How to Run Benchmarks ​

The benchmark build tag gates the benchmarks. The default go test does not start them. Without CYCLONETEST_DSN, the harness starts a disposable Postgres with Testcontainers. Set CYCLONETEST_DSN to measure against a real database.

sh
# Optional: measure against a real Postgres instead of a container
export CYCLONETEST_DSN='postgres://user:pass@localhost:5432/mydb?sslmode=disable'

# Throughput and soak profiles
go test -tags benchmark -run '^$' -bench BenchmarkSustainedThroughput -benchtime 1x .
go test -tags benchmark -run '^$' -bench BenchmarkLongRunningSoak -benchtime 1x .

# Graph-shape and maintenance profiles
go test -tags benchmark -run '^$' -bench 'BenchmarkDAG|BenchmarkRetentionSweep|BenchmarkClaimWithDeepBacklog' -benchtime 1x .

# Run with CPU profiling
go test -tags benchmark -run '^$' -bench BenchmarkSustainedThroughput -cpuprofile=cpu.pprof .
go tool pprof cpu.pprof

The defaults fit a smoke test of a few seconds. Environment knobs scale them to production-sized profiles. See the header comments in bench_test.go and bench_dag_test.go for the full list.

Benchmark Profiles ​

Short jobs ​

BenchmarkSustainedThroughput measures end-to-end throughput of a minimal job at a target enqueue rate: enqueue → claim → complete, with a no-op handler.

This tests engine overhead at maximum throughput: DB round trips, the claim/complete cycle, and LISTEN/NOTIFY latency.

What to measure:

  • ops/sec (1 / (ns/op × 1e-9))
  • Latency from enqueue to completion (wall time per op in the benchmark)
  • DB connection utilization during the benchmark

Typical bottlenecks:

  • DB latency (network between app and Postgres)
  • Connection pool saturation (increase MaxConns or reduce workers)
  • Autovacuum lag (dead tuples building up; check pg_stat_user_tables)

Long-running soak ​

BenchmarkLongRunningSoak measures behavior over a sustained period with handlers that sleep for several seconds.

What to measure:

  • Heartbeat renewal rate (a steady rate is the healthy result)
  • Memory stability (goroutine count and connection count do not grow)
  • Lease expiry rate (0 under normal load)

Large DAG advancement ​

BenchmarkDAGAdvancementWide, BenchmarkDAGAdvancementDeep, and BenchmarkDAGAdvancementFanIn take wide fan-out, deep chain, and high fan-in graphs to completion. Run advancement re-evaluates graph state on every transition. Total advancement work grows with node count times transition count. That growth is the quadratic risk for large workflows.

What to measure:

  • Whole-run wall time as CYCLONE_BENCH_DAG_NODES grows (watch for super-linear scaling)
  • Submit latency and the bulk insert of cyclone_steps rows
  • Transaction size (lock contention is a fault)

Retention and claim scans ​

BenchmarkRetentionSweep seeds a cohort of aged terminal runs and measures PurgeTerminalRuns draining them in batches. BenchmarkClaimWithDeepBacklog measures claim throughput. The steps table also holds a large cohort of future-scheduled rows. That is the skewed-distribution case. The claim scan must stay index-selective. To capture query plans at these table sizes, start EXPLAIN (ANALYZE, BUFFERS) against the seeded schema while a benchmark is paused. Or execute the claim SQL and the retention SQL from queries.sql by hand.

Interpreting Results ​

text
BenchmarkSustainedThroughput-8   1000   1234567 ns/op
  • 1000: number of iterations.
  • 1234567 ns/op: 1.23ms per operation (one enqueue+complete cycle).
  • Throughput: 1 / 0.00123s ≈ 813 ops/sec with 8 workers.

Reading the right benchmark for your workload:

Your workloadRelevant benchmark
High-volume short jobsBenchmarkSustainedThroughput
Multi-step workflowsBenchmarkLargeDAGSubmit for submit; BenchmarkDAGAdvancementWide, BenchmarkDAGAdvancementDeep, and BenchmarkDAGAdvancementFanIn for execution
Long-running handlersBenchmarkLongRunningSoak
Transactional enqueueBenchmarkSustainedThroughput (enqueue → complete)
Retention and claim scansBenchmarkRetentionSweep and BenchmarkClaimWithDeepBacklog

Environment Variables ​

VariablePurpose
CYCLONETEST_DSNPostgres DSN; omit to use Testcontainers
CYCLONE_BENCH_DURATIONShort-job window (default 5s)
CYCLONE_BENCH_WORKERSShort-job worker count (default 8)
CYCLONE_BENCH_RATETarget jobs/sec (default 50)
CYCLONE_SOAK_HANDLERSConcurrent long handlers (default 20)
CYCLONE_SOAK_BODYSynthetic body duration (default 2s)
CYCLONE_SOAK_LEASESoak lease TTL (default 1s)
CYCLONE_BENCH_DAG_NODESNodes per DAG shape (default 150)
CYCLONE_BENCH_RETENTION_RUNSTerminal runs to purge (default 2000)
CYCLONE_BENCH_BACKLOG_ROWSFuture-scheduled rows polluting the claim scan (default 5000)
CYCLONE_BENCH_CLAIMABLEClaimable steps drained through the backlog (default 300)

Hardware and Database Notes ​

Co-location puts Postgres on the same host as the benchmarks. That removes network latency and inflates throughput versus production. Execute the benchmarks in an environment that matches the production network topology.

Postgres on NVMe SSD: cyclone_steps is write-heavy. Spinning disk limits throughput compared to SSD.

Connection latency is the main determinant of short-job throughput. A 0.5ms RTT to Postgres allows about 2000 claim+complete cycles/sec with 1 worker. A 5ms RTT allows about 200. Use more workers to hide that latency.

DB configuration for benchmarks:

  • shared_buffers = 256MB minimum
  • synchronous_commit = on (keep for accuracy; turn off only to explore theoretical limits)
  • max_connections must accommodate all benchmark workers plus pool reserve

Cyclone stores each run in Postgres.