Producer/Worker Split

Sometimes the service that submits work is not the service that does the work. An API handler enqueues a standalone job. A separate worker fleet does that job. The API service has no reason to import the handler bodies. It has no reason to import their dependencies or their secrets. The API service cannot always compile those packages.

cyclone.New(pool) with no handlers gives a producer-only client. The client enqueues, submits, and schedules typed steps, actions, and workflows. It can do that in batches and in a caller transaction. It needs no registration. Start returns ErrProducerOnly.

Declare the Kind on the Producer

The producer imports a small package of kind declarations. The package has no handler logic:

// package jobs: shared kind declarations, safe for any producer to import.
var FetchReport = cyclone.RegisterKind[FetchInput, ReportURL]("fetch_report", cyclone.WithVersion(2))

RegisterKind[I, O] returns the same *cyclone.Step[I, O] that NewStep does. It works with Do, Enqueue, and Submit. The checks on the types happen at compile time. A mismatched type is a build error, as for a step that has a handler.

Build the producer client and enqueue:

producer, err := cyclone.New(pool)
if err != nil {
    return err
}
// Shut down before the pool closes. This is necessary with WithNotifyCoalesce.
defer producer.Shutdown(context.Background())

run, err := jobs.FetchReport.Enqueue(ctx, producer, FetchInput{ID: id})

A producer-only client does not look up kinds in a registry. It accepts the typed definitions that you give it. It still validates the graph, the inputs, the options, and the payload sizes. A producer cannot see which kinds and queues the workers serve. Confirm that a worker serves each kind and each queue that the producer uses.

For a strict kind check on the producer, register the declarations: cyclone.New(pool, cyclone.WithHandlers(jobs.FetchReport)). Then a workflow or enqueue that uses a kind you did not declare fails before the insert. This client also refuses Start.

Implement the Handler on the Worker

The worker imports the real handler. It registers that handler under the same kind name and version:

fetch := cyclone.NewStep("fetch_report", func(ctx cyclone.Context, in FetchInput) (ReportURL, error) {
    // ... real work, real dependencies ...
    return url, nil
}, cyclone.WithVersion(2))

worker, err := cyclone.New(pool, cyclone.WithHandlers(fetch))
if err != nil {
    return err
}
defer worker.Shutdown(context.Background())
return worker.Start(ctx) // blocks, claiming and running work

Both clients point at the same database. The producer inserts runs. The worker claims the runs and completes them. The split does not change the semantics of a run. It changes only where the handler code lives.

A workflow schedule fires only in a worker that called Schedule on the workflow at startup. A producer can create the schedule row. The producer does not fire the schedule.

A child run starts only in a worker that registered its workflow: cyclone.New(pool, cyclone.WithHandlers(fetch, Fulfillment)). A producer can submit a workflow with a child node. The producer does not start the child run. A worker that did not register the workflow leaves the child node waiting.

Start Refuses on a Producer Client

A producer cannot do work. Start returns an error. It does not start workers that discard every run:

err := producer.Start(ctx)
errors.Is(err, cyclone.ErrProducerOnly) // true

Start refuses before it claims work or starts a goroutine. WithWorkers does not change this. Start also refuses an empty registry. Start refuses a registry that holds a RegisterKind declaration. Both errors match ErrProducerOnly.

If one binary must submit work and do the work, give it the real NewStep handlers. Do not put a RegisterKind declaration into a client that you start.

Read Results in the Producer

A handle uses only the database. The producer can read the result of a run that it inserted:

report, err := run.Result(ctx)

// In a different request, from a stored ID:
report, err = jobs.FetchReport.Open(producer, savedID).Result(ctx)

A producer does not call Start. It has no listener. Its Result and Wait calls poll the run once a second.

Enqueue From a Worker's Handler

A worker often must start work that it does not do. A model-call worker hands a result to a reducer in another fleet. Give the worker a producer-only client. Enqueue through the completion transaction of the handler with that client:

// In the worker binary: a client for the kinds that other fleets serve.
producer, err := cyclone.New(pool, cyclone.WithNotify(false))
if err != nil {
    return err
}
defer producer.Shutdown(context.Background())

call := cyclone.NewStep("model.call", func(ctx cyclone.Context, in CallInput) (CallOutput, error) {
    out, err := callModel(ctx, in)
    if err != nil {
        return CallOutput{}, err
    }
    _, err = cyclone.EnqueueCompletion(ctx, producer, jobs.ReduceSession,
        ReduceInput{SessionID: in.SessionID},
        cyclone.WithSequence(in.SessionID),
    )
    return out, err
}, cyclone.WithQueue("model"))

The producer validates the payload limits and the options. The worker registry does not need jobs.ReduceSession. Cyclone inserts the run in the completion transaction of the step. The run commits only with the step. The worker that does the handler publishes the wakeup after that commit. See Transactional Enqueueing.

When to Use It

• The producer and the worker are separate deployables. They have different dependency trees.
• You do not want the API tier to import worker internals (secrets, heavy SDKs).
• You split an existing monolith. Replace the NewStep handlers of the producer with RegisterKind declarations of the same name. The submit path does not change.

See Transactional Enqueueing to commit a producer submit with a business write. See Renaming and Versioning Steps to keep the @vN kind names the same on the producer and the worker.