Workflows

A workflow is a versioned graph of steps (nodes) connected by edges (channels and messages).

If Pipelines is about authoring one job, workflows are about composing many jobs into a system.

Lifecycle: draft → publish → new version

Draft: editable while you iterate.
Validate: checks schema and binding rules before you can publish.
Publish: makes the version immutable (the version you deploy).
New version: to change a published workflow, clone it into a new draft and publish again.

This “immutable published artifact” rule is the same idea used for jobs.

Steps: binding to building blocks

Each step is one node in the graph. A step becomes runnable when it is bound to a building block:

Blueprint: a reusable module from the Catalog (connectors, adapters, patterns).
Workflow module: an imported workflow that exposes an in/out interface (a reusable subgraph).

Steps have in and out bindings. Those bindings name the channels the step consumes/produces.

Edges: channels and transport kinds

Channel edges are the data plane: events flow along a named channel from producer steps to consumer steps.

Each edge also has a transport kind that controls how those events move (for example, fast in-memory vs durable spooling). The Deployment Manager can “stitch in” the required transport adapters when you choose a transport kind; you usually do not need to add extra steps manually.

If you pick a durable transport, the plan may include additional adapter jobs (for example “writer/reader” pairs). That is expected.

Rule of thumb:

Use worker channels for low-latency hops within a worker group.
Use file store or object store transports at stage boundaries where you need durability, replay/backfill, or blast-radius isolation.

Message edges: trigger and coordination

Message edges are the trigger plane: they wire “wake up” signals between steps. They are not for bulk event transport.

If you want the deeper mechanics, see Deployments Core Concepts and Messages.

Starting points (Catalog workflows)

The Catalog includes end-to-end workflows you can use as starting points (for example the built-in “Quickstart: File Store Roundtrip” workflow).

Typical flow:

Open Catalog and filter to Workflows.
Select a workflow version and click Plan & register.
Provide a deployment name, select a worker group, and complete any wizard fields if the workflow provides them.