Deployments Core Concepts

Deployments is easiest to learn when you separate data flow from control flow.

This page introduces a minimal set of primitives that match the current product:

Graph (what’s wired to what),
Data (events moving over channels),
Trigger (signals/messages that wake things up),
State (KV, contexts, secrets),
Control (the Deployment Manager that plans/applies/reconciles).

Graph: steps and edges

Workflow

A workflow is a versioned graph made of steps and edges. It is authored as a draft, then published as an immutable version.

Step (node)

A step (node) is one box in the workflow graph. Each step:

has a stable id,
may be bound to a building block (a blueprint or workflow module),
declares inputs/outputs (in / out bindings),
and can optionally declare branching (dispatch) for multi-port outputs.

Edges: channel edges vs message edges

Workflows have two different kinds of “wiring”:

Channel edges (data flow): events move from step A to step B over a named channel.
Message edges (trigger flow): step A emits a tagged internal message, and step B runs when that tag is observed.

Message edges are for “wake up” signaling; they are not how bulk data moves.

Data: events, channels, transports

Event

An event is the payload that flows along channel edges (what most people think of as “the data”).

Channel

A channel is a named logical link (for example channel:orders_raw) referenced by step inputs/outputs.

Transport kind

A transport defines how events are delivered along a channel edge. Two important starting points:

Worker channel (fast, in-memory hop between jobs on workers).
Durable spool (for example file-store) used as a landing zone between stages.

Durable spooling is essential for pipelines that need replay/backfill or blast-radius isolation between stages.

Ports and dispatch

Steps can have multi-port outputs (a map of named outputs). When a step has multiple outputs, its dispatch mode defines how events are routed:

clone duplicates each event to all output ports,
round-robin partitions events across ports,
conditional routes events based on a field value.

Trigger: signals, messages, request/reply

Message bus (internal coordination)

LyftData uses an internal message bus for coordination and signaling. It is separate from channel-based data flow.

Message edges (trigger wiring)

A message edge compiles into “tag emitted” → “input triggers on tag received”. Sometimes the system inserts a relay step to convert “data events on a channel” into a tagged trigger message.

Request/reply (“serviceable”) messages

Some internal messages are request/reply (RPC-like): the server asks a worker to do something (or report something) and expects a reply. Public docs usually don’t require the details, but it’s helpful context when debugging control-plane ↔ worker behavior.

Emit hook (data → message bridge)

Jobs can emit sidecar messages (for checkpoints/backlog/progress) after a successful primary write, without changing their primary output. This is one of the bridges between the data plane and the control plane.

State: KV, contexts, secrets

Worker KV (memory/cache/work queues)

The Worker KV store is a lightweight key/value and work-queue abstraction used for:

checkpoints and cursors,
backlog queues with leases (at-least-once work processing),
dedupe/idempotency tracking,
and coordination state (for example, leader-election style locks).

Think of it as “memory + coordination state” that survives across runs and replicas.

Contexts, variables, secrets

Deployable systems need configuration and credentials. Contexts/variables let you parameterize workflows across environments; secrets let you safely deliver credentials to workers.

Control: the Deployment Manager

The Deployment Manager is the control plane that turns workflows into running jobs on workers.

At a high level it:

Plans changes (expands a workflow into concrete jobs/channels and shows the diff),
Applies the plan (stages and deploys jobs),
Places steps onto worker groups and sets replicas,
and Reconciles drift back to desired state.

Starter artifacts: jobs, templates, blueprints, workflows

Library job: a single-job starter (useful when you want “one job that does X”).
Template: a parameterized job starter that generates a draft job from a snippet.
Blueprint: a reusable workflow module (building block) used inside workflows.
Workflow: an end-to-end DAG composed of blueprints (deployable via the Deployment Manager).

Read Workflows for how edges/transports and publishing work in practice.
Read Deployment Manager for plan/apply/reconcile.
Read Catalog for how workflows/blueprints/templates and library jobs show up in the UI.