Skip to content

Advanced workflows: stateful multi-step Jobs

This page shows how to build a Workflow whose Jobs remember state between events or runs. Start with Workflows, then use the Worker KV input, Worker KV output, Worker KV action, and Worker KV Get action references. Choose an edge from Transports and Durability when data must cross a Worker boundary.

Worker KV is a keyed store on the Worker that runs the Job. A namespace groups keys, such as demo:orders. A key such as entity-001 holds one JSON or raw value. A ttl-ms on set or set-if-absent expires that key after the configured number of milliseconds.

A set, also called a workset, is a queue of members. Each member has an identifier and a payload. pop-batch leases members for processing. The input returns a batch handle; an ack-batch output removes the leased members after your Job finishes.

LyftData 3.0 has no external key-value state store. Worker KV is the only keyed state, and it is local to one Worker rather than shared between Workers. Redis is a messaging connector: use Redis Streams to hand events from one Job or system to another. Redis is not a Worker KV backend.

Worker KV has four limits. The Server holds them as worker_kv settings and sends them to its Workers:

Setting Default What it limits
namespaces_limit 100 Namespaces on one Worker.
set_cardinality_limit 250,000 Members in one workset.
outstanding_leases_limit 10,000 Leases that are out at the same time.
sweeper_interval 60 seconds How often expired keys and expired leases are swept.

The store itself is a SQLite database file on the Worker. On an external Worker it is kv-store-db/kv-store.db under the Worker’s output directory.

Use the input for reads and the output or action for mutations. Write the operation directly under worker-kv, as the examples on this page do; do not wrap it in an operation: key.

Surface Operation What it does Main settings
Input get Fetches one value from a namespace. namespace, get.key; optional value-format (json or raw-string), trigger, retry
Input pop-batch Leases members from a workset. namespace, set, limit (default 1), lease-ttl-ms, ack-mode (ack-batch or ack-each)
Input iterate Scans keys with a prefix. namespace, prefix, page-size
Output/action set Stores or replaces a value. namespace, key (or action key-field), value-field, optional ttl-ms
Output/action set-if-absent Stores a value only when the key is missing. namespace, key (or key-field), value-field, optional ttl-ms
Output/action delete Removes a key. namespace, key (or action key-field)
Output/action add-to-set Adds a member and payload to a workset. namespace, set, member-field, value-field; output also accepts lease-ttl-ms
Output/action remove-from-set Removes a member from a workset. namespace, set, member-field
Output/action ack-batch Acknowledges a leased batch. namespace, set, batch-handle-field
Action worker-kv-get Reads one key and writes it into the event. namespace, key or key-field, output-field (default /kv_value)
Action worker-kv Runs one mutation, optionally under a condition. namespace, optional condition, and exactly one operation written directly under the action (set, set-if-absent, delete, add-to-set, remove-from-set, ack-batch); key operations accept key-field

value-field, member-field, key-field, and output-field are event selectors. A selector can be a JSON pointer or dotted path, such as /payload or payload. The action form can use a dynamic key from the event; the output form uses a literal key.

This pattern reads the previous value, makes a decision, and writes the new value with an expiry. Keep all four Jobs on the same Worker so their Worker Channels and Worker KV namespace are local to one Worker.

flowchart LR
  accTitle: Read, decide, update with Worker KV
  accDescr: An HTTP intake normalizes an event, a lookup reads prior state, a script tags changed events, and an update stores the new state before forwarding it.
  intake["HTTP intake"] -->|normalized| lookup["Lookup"]
  lookup -->|prior value| decide["Decide"]
  decide -->|changed| update["Update"]
  update --> output["Workflow output"]

The intake Job accepts a JSON event such as {"id": "Entity-001", "status": "ready"} over HTTP and normalizes the key. The other three Jobs each read one Worker Channel and write the next.

state-intake.yaml
name: state-intake
description: "Normalize an HTTP event for the stateful workflow."
input:
http-server:
address: 127.0.0.1:8000
path: /events
only-body: true
json: true
actions:
- script:
overwrite: true
let:
id: "lower(trim(id))"
output:
worker-channel:
worker-channel-name: state-intake-to-lookup
state-lookup.yaml
name: state-lookup
description: "Read the prior value for the event key."
input:
worker-channel:
worker-channel-name: state-intake-to-lookup
actions:
- worker-kv-get:
namespace: tenant:demo
key-field: /id
output-field: /prior
output:
worker-channel:
worker-channel-name: state-lookup-to-decide
state-decide.yaml
name: state-decide
description: "Mark the event as changed when its status differs from the stored one."
input:
worker-channel:
worker-channel-name: state-lookup-to-decide
actions:
- script:
let:
changed: "prior ~= status" # true the first time, when prior is empty
output:
worker-channel:
worker-channel-name: state-decide-to-update
state-update.yaml
name: state-update
description: "Store changed state for one day and forward the decision."
input:
worker-channel:
worker-channel-name: state-decide-to-update
actions:
- worker-kv:
condition: "changed == true"
namespace: tenant:demo
set:
key-field: /id
value-field: /status
ttl-ms: 86400000 # one day
output:
worker-channel:
worker-channel-name: stateful-output

Publish each Job above as a blueprint version 1 under the names state-intake, state-lookup, state-decide, and state-update. Give each blueprint one worker-channel input or output; the planner rewrites that single channel to the step’s in or out. Confirm the four names appear in the Workflow palette, then import this manifest. Keep the step IDs stable when you change placement or replicas.

{
"workflow": "stateful-read-decide-update",
"version": 1,
"summary": "Read prior state, decide, and update it with a TTL.",
"imports": [
"blueprints/state-intake@1 as intake",
"blueprints/state-lookup@1 as lookup",
"blueprints/state-decide@1 as decide",
"blueprints/state-update@1 as update"
],
"steps": [
{ "id": "intake", "uses": "intake", "out": "channel:state-intake-to-lookup" },
{ "id": "lookup", "uses": "lookup", "in": "channel:state-intake-to-lookup", "out": "channel:state-lookup-to-decide" },
{ "id": "decide", "uses": "decide", "in": "channel:state-lookup-to-decide", "out": "channel:state-decide-to-update" },
{ "id": "update", "uses": "update", "in": "channel:state-decide-to-update", "out": "channel:stateful-output", "replicas": 1 }
]
}

Splitting the work into steps lets you change and observe intake, lookup, decision, and update one at a time. All four steps must be placed on the same Worker, because both the Worker Channels and the Worker KV namespace are local to it. When you do not need separate steps, the same three actions fit in one Job’s actions list.

A producer adds members to the backlog workset. A consumer leases up to ten members, processes them, and acknowledges the returned batch handle. This adapts the shipped quickstart-worker-kv-backlog-roundtrip assets.

flowchart LR
  accTitle: Worker KV work queue lease cycle
  accDescr: A producer adds a member to the workset, a consumer leases it, acknowledges it after processing, and an expired lease returns it to the workset.
  producer["Producer"] -->|add-to-set| workset[("backlog workset")]
  workset -->|pop-batch lease| consumer["Consumer"]
  consumer -->|acknowledge| done["Removed"]
  consumer -. "lease expires" .-> workset
producer.yaml
name: quickstart-worker-kv-backlog-producer
description: "Enqueue work items."
input:
echo:
trigger:
interval:
duration: 2s
event: '{"id":"entity-001","payload":{"message":"hello worker-kv"}}'
json: true
actions: []
output:
worker-kv:
namespace: demo:quickstart-kv
add-to-set:
set: backlog
member-field: /id
value-field: /payload
consumer.yaml
name: quickstart-worker-kv-backlog-consumer
description: "Lease, process, and acknowledge work."
input:
worker-kv:
trigger:
interval:
duration: 2s
namespace: demo:quickstart-kv
pop-batch:
set: backlog
limit: 10
lease-ttl-ms: 60000
ack-mode: ack-batch
actions: []
output:
worker-kv:
namespace: demo:quickstart-kv
ack-batch:
set: backlog
batch-handle-field: /batch_handle

ack-batch acknowledges the batch handle returned by pop-batch in one operation. ack-each acknowledges members individually, which is useful when a batch can partially succeed; the input defaults to ack-batch.

The matching Workflow uses the shipped blueprint names:

{
"workflow": "quickstart-worker-kv-backlog-roundtrip",
"version": 1,
"summary": "Enqueue, lease, and acknowledge workset members.",
"imports": [
"blueprints/quickstart-worker-kv-backlog-producer@1 as producer",
"blueprints/quickstart-worker-kv-backlog-consumer@1 as consumer"
],
"steps": [
{ "id": "producer", "uses": "producer" },
{ "id": "consumer", "uses": "consumer" }
]
}

Deploying this manifest runs the shipped blueprints. The YAML above is a simplified adaptation with a fixed entity-001 id; publish it under new blueprint names if the sample must emit that fixed id.

If the consumer does not acknowledge before the lease expires, the member is eligible to be leased again. Make processing idempotent with the member identifier and choose a lease long enough for the work.

Handing off to another system with Redis Streams

Section titled “Handing off to another system with Redis Streams”

An output Job appends events to a Redis Stream, and an input Job in another Workflow, or another system, reads the stream.

stream-output.yaml
name: stream-output
input:
worker-channel:
worker-channel-name: redis-out
output:
redis:
mode: stream
address: redis://redis.example.com:6379/0
stream: orders.pending
payload-field: payload
stream-input.yaml
name: stream-input
input:
redis:
mode: stream
address: redis://redis.example.com:6379/0
streams: [orders.pending]
start-id: $
payload-field: payload
output:
worker-channel:
worker-channel-name: redis-in

This gives you a hand-off through a broker, with the stream position kept by Redis. It does not give two Workers shared keyed state: for a key lookup, use Worker KV on the Worker that owns the state.

Situation What happens to Worker KV What to do
Job restart The Job restarts on its Worker, so the Worker-local store remains available. Re-read the key or workset and make processing idempotent.
Worker restart Worker KV is a database file on the Worker’s disk, so the Worker reopens it when it starts again. Check the namespace after the restart and let the Job continue.
Lease expiry The sweeper makes an unacknowledged member eligible for another lease. Process it again safely and acknowledge the current batch handle.
Job moves to another Worker Worker KV is local to the original Worker, so the new Worker cannot read that state. Keep the stateful steps on the original Worker group or rebuild state from the source.
  • Worker KV is per Worker. It is not a shared store across Workers.
  • Worker KV limits namespaces, workset members, and outstanding leases (see the table above). LyftData 3.0 does not set a size limit for a single value.
  • Worker KV contents stay in the database file on the Worker that wrote them. A replacement Worker, or another Worker that a Job is placed on, starts without them. Keep the state rebuildable from your source, and pin stateful steps to the Worker group that holds it.
  • A Worker Channel also joins Jobs only on the same Worker; it is not a cross-Worker state path.