Advanced workflows: stateful multi-step Jobs
This page shows how to build a Workflow whose Jobs remember state between events or runs. Start with Workflows, then use the Worker KV input, Worker KV output, Worker KV action, and Worker KV Get action references. Choose an edge from Transports and Durability when data must cross a Worker boundary.
Where state lives
Section titled “Where state lives”Worker KV is a keyed store on the Worker that runs the Job. A namespace groups keys, such as demo:orders. A key such as entity-001 holds one JSON or raw value. A ttl-ms on set or set-if-absent expires that key after the configured number of milliseconds.
A set, also called a workset, is a queue of members. Each member has an identifier and a payload. pop-batch leases members for processing. The input returns a batch handle; an ack-batch output removes the leased members after your Job finishes.
LyftData 3.0 has no external key-value state store. Worker KV is the only keyed state, and it is local to one Worker rather than shared between Workers. Redis is a messaging connector: use Redis Streams to hand events from one Job or system to another. Redis is not a Worker KV backend.
Worker KV has four limits. The Server holds them as worker_kv settings and sends them to its Workers:
| Setting | Default | What it limits |
|---|---|---|
namespaces_limit |
100 | Namespaces on one Worker. |
set_cardinality_limit |
250,000 | Members in one workset. |
outstanding_leases_limit |
10,000 | Leases that are out at the same time. |
sweeper_interval |
60 seconds | How often expired keys and expired leases are swept. |
The store itself is a SQLite database file on the Worker. On an external Worker it is kv-store-db/kv-store.db under the Worker’s output directory.
Worker KV operations
Section titled “Worker KV operations”Use the input for reads and the output or action for mutations. Write the operation directly under worker-kv, as the examples on this page do; do not wrap it in an operation: key.
| Surface | Operation | What it does | Main settings |
|---|---|---|---|
| Input | get |
Fetches one value from a namespace. | namespace, get.key; optional value-format (json or raw-string), trigger, retry |
| Input | pop-batch |
Leases members from a workset. | namespace, set, limit (default 1), lease-ttl-ms, ack-mode (ack-batch or ack-each) |
| Input | iterate |
Scans keys with a prefix. | namespace, prefix, page-size |
| Output/action | set |
Stores or replaces a value. | namespace, key (or action key-field), value-field, optional ttl-ms |
| Output/action | set-if-absent |
Stores a value only when the key is missing. | namespace, key (or key-field), value-field, optional ttl-ms |
| Output/action | delete |
Removes a key. | namespace, key (or action key-field) |
| Output/action | add-to-set |
Adds a member and payload to a workset. | namespace, set, member-field, value-field; output also accepts lease-ttl-ms |
| Output/action | remove-from-set |
Removes a member from a workset. | namespace, set, member-field |
| Output/action | ack-batch |
Acknowledges a leased batch. | namespace, set, batch-handle-field |
| Action | worker-kv-get |
Reads one key and writes it into the event. | namespace, key or key-field, output-field (default /kv_value) |
| Action | worker-kv |
Runs one mutation, optionally under a condition. |
namespace, optional condition, and exactly one operation written directly under the action (set, set-if-absent, delete, add-to-set, remove-from-set, ack-batch); key operations accept key-field |
value-field, member-field, key-field, and output-field are event selectors. A selector can be a JSON pointer or dotted path, such as /payload or payload. The action form can use a dynamic key from the event; the output form uses a literal key.
Read, decide, update
Section titled “Read, decide, update”This pattern reads the previous value, makes a decision, and writes the new value with an expiry. Keep all four Jobs on the same Worker so their Worker Channels and Worker KV namespace are local to one Worker.
flowchart LR accTitle: Read, decide, update with Worker KV accDescr: An HTTP intake normalizes an event, a lookup reads prior state, a script tags changed events, and an update stores the new state before forwarding it. intake["HTTP intake"] -->|normalized| lookup["Lookup"] lookup -->|prior value| decide["Decide"] decide -->|changed| update["Update"] update --> output["Workflow output"]
The intake Job accepts a JSON event such as {"id": "Entity-001", "status": "ready"} over HTTP and normalizes the key. The other three Jobs each read one Worker Channel and write the next.
name: state-intakedescription: "Normalize an HTTP event for the stateful workflow."input: http-server: address: 127.0.0.1:8000 path: /events only-body: true json: trueactions: - script: overwrite: true let: id: "lower(trim(id))"output: worker-channel: worker-channel-name: state-intake-to-lookupname: state-lookupdescription: "Read the prior value for the event key."input: worker-channel: worker-channel-name: state-intake-to-lookupactions: - worker-kv-get: namespace: tenant:demo key-field: /id output-field: /prioroutput: worker-channel: worker-channel-name: state-lookup-to-decidename: state-decidedescription: "Mark the event as changed when its status differs from the stored one."input: worker-channel: worker-channel-name: state-lookup-to-decideactions: - script: let: changed: "prior ~= status" # true the first time, when prior is emptyoutput: worker-channel: worker-channel-name: state-decide-to-updatename: state-updatedescription: "Store changed state for one day and forward the decision."input: worker-channel: worker-channel-name: state-decide-to-updateactions: - worker-kv: condition: "changed == true" namespace: tenant:demo set: key-field: /id value-field: /status ttl-ms: 86400000 # one dayoutput: worker-channel: worker-channel-name: stateful-outputPublish each Job above as a blueprint version 1 under the names state-intake, state-lookup, state-decide, and state-update. Give each blueprint one worker-channel input or output; the planner rewrites that single channel to the step’s in or out. Confirm the four names appear in the Workflow palette, then import this manifest. Keep the step IDs stable when you change placement or replicas.
{ "workflow": "stateful-read-decide-update", "version": 1, "summary": "Read prior state, decide, and update it with a TTL.", "imports": [ "blueprints/state-intake@1 as intake", "blueprints/state-lookup@1 as lookup", "blueprints/state-decide@1 as decide", "blueprints/state-update@1 as update" ], "steps": [ { "id": "intake", "uses": "intake", "out": "channel:state-intake-to-lookup" }, { "id": "lookup", "uses": "lookup", "in": "channel:state-intake-to-lookup", "out": "channel:state-lookup-to-decide" }, { "id": "decide", "uses": "decide", "in": "channel:state-lookup-to-decide", "out": "channel:state-decide-to-update" }, { "id": "update", "uses": "update", "in": "channel:state-decide-to-update", "out": "channel:stateful-output", "replicas": 1 } ]}Splitting the work into steps lets you change and observe intake, lookup, decision, and update one at a time. All four steps must be placed on the same Worker, because both the Worker Channels and the Worker KV namespace are local to it. When you do not need separate steps, the same three actions fit in one Job’s actions list.
A work queue with lease and acknowledge
Section titled “A work queue with lease and acknowledge”A producer adds members to the backlog workset. A consumer leases up to ten members, processes them, and acknowledges the returned batch handle. This adapts the shipped quickstart-worker-kv-backlog-roundtrip assets.
flowchart LR
accTitle: Worker KV work queue lease cycle
accDescr: A producer adds a member to the workset, a consumer leases it, acknowledges it after processing, and an expired lease returns it to the workset.
producer["Producer"] -->|add-to-set| workset[("backlog workset")]
workset -->|pop-batch lease| consumer["Consumer"]
consumer -->|acknowledge| done["Removed"]
consumer -. "lease expires" .-> workset
name: quickstart-worker-kv-backlog-producerdescription: "Enqueue work items."input: echo: trigger: interval: duration: 2s event: '{"id":"entity-001","payload":{"message":"hello worker-kv"}}' json: trueactions: []output: worker-kv: namespace: demo:quickstart-kv add-to-set: set: backlog member-field: /id value-field: /payloadname: quickstart-worker-kv-backlog-consumerdescription: "Lease, process, and acknowledge work."input: worker-kv: trigger: interval: duration: 2s namespace: demo:quickstart-kv pop-batch: set: backlog limit: 10 lease-ttl-ms: 60000 ack-mode: ack-batchactions: []output: worker-kv: namespace: demo:quickstart-kv ack-batch: set: backlog batch-handle-field: /batch_handleack-batch acknowledges the batch handle returned by pop-batch in one operation. ack-each acknowledges members individually, which is useful when a batch can partially succeed; the input defaults to ack-batch.
The matching Workflow uses the shipped blueprint names:
{ "workflow": "quickstart-worker-kv-backlog-roundtrip", "version": 1, "summary": "Enqueue, lease, and acknowledge workset members.", "imports": [ "blueprints/quickstart-worker-kv-backlog-producer@1 as producer", "blueprints/quickstart-worker-kv-backlog-consumer@1 as consumer" ], "steps": [ { "id": "producer", "uses": "producer" }, { "id": "consumer", "uses": "consumer" } ]}Deploying this manifest runs the shipped blueprints. The YAML above is a simplified adaptation with a fixed entity-001 id; publish it under new blueprint names if the sample must emit that fixed id.
If the consumer does not acknowledge before the lease expires, the member is eligible to be leased again. Make processing idempotent with the member identifier and choose a lease long enough for the work.
Handing off to another system with Redis Streams
Section titled “Handing off to another system with Redis Streams”An output Job appends events to a Redis Stream, and an input Job in another Workflow, or another system, reads the stream.
name: stream-outputinput: worker-channel: worker-channel-name: redis-outoutput: redis: mode: stream address: redis://redis.example.com:6379/0 stream: orders.pending payload-field: payloadname: stream-inputinput: redis: mode: stream address: redis://redis.example.com:6379/0 streams: [orders.pending] start-id: $ payload-field: payloadoutput: worker-channel: worker-channel-name: redis-inThis gives you a hand-off through a broker, with the stream position kept by Redis. It does not give two Workers shared keyed state: for a key lookup, use Worker KV on the Worker that owns the state.
When something fails
Section titled “When something fails”| Situation | What happens to Worker KV | What to do |
|---|---|---|
| Job restart | The Job restarts on its Worker, so the Worker-local store remains available. | Re-read the key or workset and make processing idempotent. |
| Worker restart | Worker KV is a database file on the Worker’s disk, so the Worker reopens it when it starts again. | Check the namespace after the restart and let the Job continue. |
| Lease expiry | The sweeper makes an unacknowledged member eligible for another lease. | Process it again safely and acknowledge the current batch handle. |
| Job moves to another Worker | Worker KV is local to the original Worker, so the new Worker cannot read that state. | Keep the stateful steps on the original Worker group or rebuild state from the source. |
Limits
Section titled “Limits”- Worker KV is per Worker. It is not a shared store across Workers.
- Worker KV limits namespaces, workset members, and outstanding leases (see the table above). LyftData 3.0 does not set a size limit for a single value.
- Worker KV contents stay in the database file on the Worker that wrote them. A replacement Worker, or another Worker that a Job is placed on, starts without them. Keep the state rebuildable from your source, and pin stateful steps to the Worker group that holds it.
- A Worker Channel also joins Jobs only on the same Worker; it is not a cross-Worker state path.