Static DAGs and Type Safety

The workflow builder checks data flow at compile time. The builder uses Go generics.

Why Static DAGs

Cyclone uses static directed acyclic graphs. Your Go code sets the topology at submit time. The topology is which nodes exist and which node depends on which node. Cyclone writes that topology to Postgres. Workers advance the run node by node. Workers do not call the build function again.

That choice has three consequences:

1. There is no replay requirement. Workers are ordinary Go functions. Workers do not need to be deterministic across runs. Cyclone permits side effects. Cyclone encourages a side effect through ctx.CompletionTx().
2. The graph is data. Cyclone stores the topology in cyclone_deps. You can inspect the topology. You can show the topology in a diagram. You can use the topology as an audit trail. You do not call the build function again for that.
3. The compiler checks the data flow between steps. Go generics do that check.

Futures: The Typed Connection Between Steps

step.Do is the method form of cyclone.Do. cyclone.Do(b, step, input) returns *Future[O]. O is the output type of the step. A *Future[O] is both of these:

• The future declares that the output of this step becomes available.
• A downstream step uses the future to declare a data dependency.

// Do returns *Future[FetchOutput]
fetchFuture := cyclone.Do(b, FetchStep, cyclone.Lit(FetchInput{URL: in.URL}))

// From(fetchFuture) is Input[FetchOutput], only accepted where FetchOutput is expected
cyclone.Run(b, SaveAction, cyclone.From(fetchFuture))
// ↑ SaveAction's input type must be FetchOutput for this to compile

The *Future[O] is only a build-time token. The future does not hold the output value at submit time. The output value exists only after the step completes in a worker.

Typed Inputs at Compile Time

cyclone.From[I](f *Future[I]) returns Input[I]. Go infers the type parameter I from the type of the future. The downstream step must accept exactly I as its input. You can also pass the future where an input of type I is required. That binding is the same as cyclone.From.

var SaveAction = cyclone.NewAction("save", func(ctx cyclone.Context, in FetchOutput) error {
    // in is FetchOutput ✓
})

// Compile-time check: FetchStep outputs FetchOutput; SaveAction inputs FetchOutput ✓
cyclone.Run(b, SaveAction, cyclone.From(fetchFuture))

// Compile error: ParseAction expects ParseInput, not FetchOutput
cyclone.Run(b, ParseAction, cyclone.From(fetchFuture))
// ↑ error: cannot use *Future[FetchOutput] as Input[ParseInput]

Wrong-type wiring is a build error, not a runtime error.

Fan-In with Tuple Types

cyclone.Join2[A, B any](a *Future[A], b *Future[B]) returns Input[Tuple2[A, B]]. Join2 merges two futures. The downstream step receives a Tuple2 with fields .A and .B.

type MergeInput = cyclone.Tuple2[SalesData, CostsData]

var MergeStep = cyclone.NewStep("merge", func(ctx cyclone.Context, in MergeInput) (Report, error) {
    return combineReport(in.A, in.B), nil
})

salesFuture := cyclone.Do(b, SalesStep, cyclone.Lit(salesIn))  // *Future[SalesData]
costsFuture := cyclone.Do(b, CostsStep, cyclone.Lit(costsIn))  // *Future[CostsData]

// Join2 creates Input[Tuple2[SalesData, CostsData]]
// MergeStep's input type must be Tuple2[SalesData, CostsData]
cyclone.Do(b, MergeStep, cyclone.Join2(salesFuture, costsFuture))

Join3 produces Tuple3[A,B,C]. Join4 produces Tuple4[A,B,C,D].

Ordering-Only Dependencies with After

Use After when a step must wait for the side effects of another node. Use After when the step does not need that output value.

// Run returns *Node, not *Future, because actions have no output.
setupNode := cyclone.Run(b, SetupAction, cyclone.Lit(struct{}{}))

// workStep runs after setup; its input is a literal, not from setup
cyclone.Do(b, WorkStep, cyclone.Lit(WorkInput{...}), cyclone.After(setupNode))

After accepts a *Node or a *Future[O]. After adds a dependency edge and does not add data flow.

Wait Nodes Are Still Static

A signal does not make the graph dynamic. A timer does not make the graph dynamic. A signal or a timer adds a parked node at submit time. That node completes later, when a signal arrives or a duration elapses.

paid := cyclone.WaitSignal[Receipt](b, "paid", cyclone.After(chargeNode))
cyclone.Run(b, FulfillOrder, cyclone.From(paid))

The node already exists in Postgres. Client.Signal(ctx, runID, "paid", payload) only completes the node.

Fan-Out Over a Runtime List

ForEach is the one node whose amount of work comes from a runtime value. The builder declares that node at submit, as it declares any other node. The declaration gives the upstream future, the item step, and the output type.

items := discover.Do(b, cyclone.Lit(in)) // *Future[[]Item]
results := process.ForEach(b, items)     // *Future[[]Result]
report.Do(b, results)

The node is static. Submit fixes its place in the graph. Submit fixes its kind. Submit fixes its edges. Those values are part of def_version. Only the item tasks are made during the run. The engine inserts one item task for each item. The engine does that insert in the transaction that completes the upstream node. The compiler still checks the types. process must take an Item. report must take a []Result.

A fan-out node does one step for each item. A fan-out node does not add a branch, a loop, or another node. See Fan Out Over a List.

Node Identity

Each Do call creates a node with a stable ID. Each Run call creates a node with a stable ID. Cyclone uses a node ID for these purposes:

• Retry targeting, as in client.Retry(ctx, runID, "save").
• Output reading with cyclone.OutputNode, and the result node that a DefineResult run stores.
• A test assertion, as in env.RequireStepState(runID, "save", cyclone.StepCompleted).
• An idempotency key in a handler, from ctx.Node().

An automatic ID comes from the step kind plus an ordinal. The first ID is the kind, as in "fetch". The next ID adds an ordinal, as in "fetch-1". You set a fixed ID with cyclone.ID("id").

cyclone.Do(b, FetchStep, cyclone.Lit(in), cyclone.ID("primary-fetch"))

The Workflow Result

A cyclone.DefineResult build function returns a *cyclone.Future[Out]. The compiler checks that the future has the result type of the workflow. At submit, Cyclone checks that the future is not nil. Cyclone also checks that the future comes from the same call of the build function. Each future holds a private token of its build. Cyclone does not store the token. A future from a different submit fails. That failure also occurs when the node ID is the same.

The run row stores the ID of the result node in the same transaction as the graph. A result handle reads that stored node. The handle does not call the build function again. A change to the build function does not change the result of a run that exists. The result node is part of the definition fingerprint.

Compile-Time Errors You Will See

Wrong input type for From

cannot use *cyclone.Future[FetchOutput] as value of type cyclone.Input[ParseInput]

Set the input type of the downstream step to the output type of the upstream step.

Nil future passed to From

The build function can compile and then panic at submit time. The panic occurs when a *Future variable was never assigned. Use cyclone.ID("id") to pin the node. Then check that the build function behaves as you expect.

Non-existent node in After

A *Node from a different builder refers to an unknown node. A build function from a different workflow is one example. Check this case in tests.

Dynamic Workflow Limitations

A Cyclone static DAG does not support these patterns:

• Loops. You cannot call Do in a loop that uses runtime data. You can call Do in a loop that uses only the build-time data in In.
• Conditional branches. You cannot add a node from a condition on a runtime result. Cyclone inserts every node at submit time.
• Dynamic sub-graphs. ForEach does one step for each item of a runtime list. A run cannot add other nodes. A child node starts a second workflow as a child run. Cyclone calls the child builder when the input exists. See Run a Child Workflow.
• Runtime routing. A completed step cannot choose which new nodes to create next.

A static DAG does not give dynamic control of the graph. A static DAG gives a simple model. A static DAG gives type safety. A static DAG has no determinism requirement.

Related

• Submit DAG Workflows
• Signals and Waits
• Engine Design