Changing Step Inputs

How to evolve the input structs for steps that have in-flight runs.

Why Changing JSON Structs Is Risky

Cyclone serializes step inputs as JSON at enqueue time. Cyclone stores that JSON in cyclone_steps.input. When a worker claims the step, it calls the handler. Cyclone then deserializes the stored JSON into the current input type.

If you change the input struct after enqueue and before the claim, deserialization can fail. It can also produce an unexpected zero value. This matters for:

• Deployments that last a long time and have many in-flight runs.
• Rolling deploys, where the old binary and the new binary are active together.
• Delayed jobs scheduled far in the future.

Additive Changes (Safe)

A new field with a zero-value default is safe. A new field with an explicit default is safe. Old rows do not contain the field. Go's encoding/json initializes a missing field to its zero value.

// Before:
type ImportInput struct {
    FileID string `json:"file_id"`
}

// After (safe): added MaxRows with zero value default
type ImportInput struct {
    FileID  string `json:"file_id"`
    MaxRows int    `json:"max_rows,omitempty"` // zero = no limit (same as before)
}

Old rows decode MaxRows as 0. The handler's zero-value behavior must match the old behavior.

Breaking Changes (Unsafe)

These changes break old in-flight rows:

• Rename a field. Old rows have the old key. The new field gets the zero value.
• Remove a field. Old rows carry data that the new handler ignores. This is usually safe for the handler. The handler drops that data. It does not report the drop.
• Change the type of a field. JSON can decode into the wrong type. Or the decode can fail entirely.

If you must make a breaking change, use WithVersion(n) to create a new step kind.

Versioned Input Structs

For a breaking change, create a new input type and version the step:

// Old step — keep registered for in-flight runs
type ImportInputV1 struct {
    FileID     string `json:"file_id"`
    MaxSeconds int    `json:"max_seconds"` // old timeout field
}
var ImportStepV1 = cyclone.NewStep("import", importV1, cyclone.WithVersion(1))

// New step — new runs use this
type ImportInputV2 struct {
    FileID  string        `json:"file_id"`
    Timeout time.Duration `json:"timeout"` // renamed and typed
}
var ImportStepV2 = cyclone.NewStep("import", importV2, cyclone.WithVersion(2))

Register both steps in the mux. In-flight runs continue with V1. New enqueues use V2.

env.Mux().Register(ImportStepV1) // for in-flight V1 runs
env.Mux().Register(ImportStepV2) // for new V2 runs

Defaulting Old Fields

For a simple case, set the default for the missing field inside the handler. Do not version the step for that case:

type ImportInput struct {
    FileID  string `json:"file_id"`
    MaxRows int    `json:"max_rows,omitempty"`
}

var ImportStep = cyclone.NewStep("import", func(ctx cyclone.Context, in ImportInput) (ImportOutput, error) {
    if in.MaxRows == 0 {
        in.MaxRows = 1_000_000 // default for old rows that have 0
    }
    // ...
})

This works when the zero value means only "not set". If zero is a valid explicit value, use a pointer:

type ImportInput struct {
    FileID  string `json:"file_id"`
    MaxRows *int   `json:"max_rows,omitempty"` // nil = use default
}

Rolling Deploys

During a rolling deploy, the old binary and the new binary are both active:

1. Deploy the new binary first with both old and new step versions registered.
2. New enqueues use the new version.
3. Old in-flight runs continue on the old binary or the new binary. The new binary still has the old handler.
4. After all old in-flight runs are terminal, remove the old handler registration.

Do not remove the old handler before all in-flight runs of that kind are terminal.

Cleaning Up Old Fields

Do this after all old runs are terminal. No row in cyclone_steps has the old kind in a non-terminal state:

-- Verify no in-flight old runs remain:
SELECT count(*) FROM cyclone_steps
WHERE kind = 'import@v1'
  AND state NOT IN ('completed', 'discarded', 'cancelled');
-- Should return 0

When that count is 0, remove the old handler from the mux. Delete the old input type.

Testing Old Payloads

Test that your new handler handles old JSON payloads correctly:

func TestOldPayloadCompatibility(t *testing.T) {
    // Simulate a row with the old JSON shape.
    oldJSON := `{"file_id": "abc123"}` // no max_rows field

    var in ImportInput
    require.NoError(t, json.Unmarshal([]byte(oldJSON), &in))

    require.Equal(t, "abc123", in.FileID)
    require.Equal(t, 0, in.MaxRows) // zero value; handler must handle this
}

Or use cyclonetest with a custom stub that injects the old payload shape.

Recommended Compatibility Policy

Change type	Safe?	Approach
Add field with zero-value default	Yes	Add the field. Handle zero in the handler
Add field with explicit default	Yes	Add a pointer field, or add defaulting logic
Remove unused field	Usually	The handler ignores unknown JSON keys
Rename field	No	Use WithVersion(n)
Change field type	No	Use WithVersion(n)
Remove required field	No	Use WithVersion(n)

For a breaking change, always use WithVersion(n). The old step kind stays in the database for the life of the old runs. If you remove it before those runs drain, workers report unhandled-kind errors.

Related

• Renaming and Versioning Steps
• Engine Design