Skip to content

Advanced architectures: Worker-to-Worker data flows

Use this page when one Workflow must move data between Workers. You choose a boundary for each hop, place the jobs that can reach that boundary, and keep ownership clear when you add replicas. You should already know Workflows, Transports and Durability, Placement and Scaling, and Routing and Fan-out. Workflow authoring and deployment are available in the licensed editions described in Workflows.

Edge Crosses Workers? Survives a Worker restart? Can be replayed? What you must provide
Worker Channel No; both Jobs are on one Worker No; it is bounded in-memory state No external replay Co-located Jobs and a compatible channel name
file-store edge Only when the path is genuinely shared Stored files remain; local paths remain in that Worker’s failure domain Yes, by listing retained files path and roll settings; co-locate a local path
S3 object-store edge Yes, when every Worker can reach the store Yes, subject to provider retention and access Yes, by listing retained objects bucket, base prefix, endpoint/region, and role or test credentials
broker reached by an output Job and an input Job (Kafka, MQTT, Redis Streams) Yes, when both Jobs can reach the broker Broker-specific Broker-specific offsets, cursors, or session rules connector fields, credentials, topic/stream, acknowledgement, and ownership policy

LyftData 3.0 has edge adapters for two durable transport kinds, file-store and s3; worker-channel is the in-memory default. An object-store edge therefore means S3 or an S3-compatible store. Kafka, MQTT, Redis Streams, Google Cloud Storage, and Azure Blob Storage are connectors that Jobs use: to cross Workers through one of them, end one step with an output Job and start the next with an input Job.

Use this shape when sources are spread across sites or teams and a core Worker group owns delivery. Put a durable edge between collection and delivery, then add replicas only where the input contract assigns work to one consumer.

flowchart LR
  accTitle: Enterprise collection and delivery
  accDescr: Collection Workers write events to an S3 prefix. A core delivery Worker reads the retained prefix and sends events to a destination.
  c1["Collection Worker"] -->|"S3 edge"| store[("S3 bucket\nevents.raw/")]
  c2["Collection Worker"] -->|"S3 edge"| store
  store --> d["Core delivery Worker"]
  d --> sink["Destination Job"]

Declare the transport on the producing step’s edges map. The planner expands the transport kind to its adapter blueprint; the adapter definitions contain the generated writer and reader Jobs. Use a role ARN in a deployed environment, and keep inline access keys to local testing.

Workflow manifest

This import is the pass-through placeholder used by the shipped S3 example. Publish your collector as a blueprint with the real input and exactly one worker-channel output, then use that blueprint instead of task. The planner rewrites that one output to the channel named in out. Publish a delivery blueprint with the matching worker-channel input before importing it as well.

{
"workflow": "enterprise-worker-flow",
"version": 1,
"summary": "Collect at source Workers and deliver from a core group.",
"imports": [
"blueprints/enterprise-collector@1 as collector",
"blueprints/core-delivery@1 as delivery"
],
"steps": [
{
"id": "collect",
"uses": "collector",
"out": "channel:events.raw",
"edges": {
"channel:events.raw": {
"transport": {
"kind": "s3",
"bucket": "events-raw",
"base_prefix": "events.raw/",
"region": "us-east-1",
"endpoint": "https://example.com",
"role_arn": "arn:aws:iam::123456789012:role/lyftdata-events",
"poll_interval": "10s"
}
}
}
},
{ "id": "deliver", "uses": "delivery", "in": "channel:events.raw", "replicas": 1 }
]
}

The step IDs are placement anchors. out, in, bucket, and base_prefix bind the named channel and durable location. Place collect near sources and deliver in the core group. In this configuration two replicas of the S3 reader would both list the same prefix, so keep the reader at one replica.

The expanded adapter contains a s3-worker-channel-writer with a worker-channel input and S3 output, and a s3-worker-channel-reader with an S3 list-and-download input and worker-channel output. Treat those generated Jobs as implementation details of the edge.

Failure What happens to in-flight data How you recover
Collection Worker restarts Events still in its in-memory input can be lost; committed objects remain Restart collection from the source and let the reader list retained objects
Core reader restarts Stored objects remain; an unfinished read depends on reader state Restart the reader and inspect its fingerprint state before replay
Delivery Worker leaves Retained objects remain available to an eligible reader Restore one eligible Worker and keep the reader at one replica

Use this shape when local Workers can reach sources but the cloud group owns the destination. Choose an S3 edge when a shared bucket is the boundary, or use an output Job and input Job around a broker when the boundary is a broker. Keep file-store for one Worker or a genuinely shared path.

flowchart LR
  accTitle: Hybrid on-premises to cloud flow
  accDescr: On-premises collection writes to an S3 bucket. Cloud delivery reads the retained prefix and sends it onward.
  src["On-prem source"] --> local["Local collection and shaping"]
  local -->|"S3 edge"| bucket[("Shared S3 bucket\nhybrid/")]
  bucket --> cloud["Cloud Worker group"]
  cloud --> dest["Cloud delivery Job"]

Workflow manifest

This import is the pass-through placeholder used by the shipped S3 example. Publish your collector as a blueprint with the real input and exactly one worker-channel output, then use that blueprint instead of task. The planner rewrites that one output to the channel named in out. Publish a delivery blueprint with the matching worker-channel input before importing it as well.

{
"workflow": "hybrid-worker-flow",
"version": 1,
"summary": "Shape locally and deliver from the cloud group.",
"imports": [
"blueprints/hybrid-collector@1 as collector",
"blueprints/cloud-delivery@1 as delivery"
],
"steps": [
{
"id": "local-shape",
"uses": "collector",
"out": "channel:hybrid.events",
"edges": {
"channel:hybrid.events": {
"transport": {
"kind": "s3",
"bucket": "hybrid-events",
"base_prefix": "hybrid/",
"region": "us-east-1",
"endpoint": "https://example.com",
"role_arn": "arn:aws:iam::123456789012:role/lyftdata-hybrid"
}
}
}
},
{ "id": "cloud-deliver", "uses": "delivery", "in": "channel:hybrid.events", "replicas": 1 }
]
}

For a file-store edge, use kind: "file-store", a shared path such as fs://var/lyftdata/edges/hybrid, and roll with max_rows and max_age. A local path stays in one Worker’s failure domain.

Job definitions for the broker variant

name: shaped-events-to-kafka
input:
worker-channel:
worker-channel-name: shaped-events
output:
kafka:
bootstrap-servers: "{{KAFKA_BROKER}}"
topic: events.raw
payload: json
acks: all
name: kafka-delivery
context:
KAFKA_BROKER: "{{KAFKA_BROKER}}"
KAFKA_GROUP: hybrid-delivery
input:
kafka:
bootstrap-servers: "{{KAFKA_BROKER}}"
group-id: "{{KAFKA_GROUP}}"
topics:
- events.raw
payload: json
output:
worker-channel:
worker-channel-name: delivery

Resolve KAFKA_* values from the Credential Manager or deployment context. Kafka is an Enterprise connector. Use a separate routing Job for a dead-letter topic.

Failure What happens to in-flight data How you recover
Boundary link is down S3 objects or broker records remain under their retention rules Restore connectivity, then resume the reader or consumer group
Shared path is mounted locally by mistake The cloud Worker sees no local files Move the boundary to genuine shared storage or a broker
Cloud delivery Worker restarts Retained objects or broker records remain; in-memory work may not Restart with the same group, prefix, and ownership settings

Use MQTT when devices already publish telemetry to an edge broker. A central collector subscribes once, then a clone fan-out sends one copy to alerting and one to archive.

flowchart LR
  accTitle: IoT MQTT collection and fan-out
  accDescr: Devices publish to MQTT. A collector receives telemetry and clones it to alert and archive Jobs.
  devices["Devices"] --> mqtt[("MQTT broker\nsensors/#")]
  mqtt --> collector["mqtt-collect"]
  collector --> fan{{"clone"}}
  fan --> alert["alert"]
  fan --> archive["archive"]

Keep the MQTT subscriber in the central group and put the fan-out after it. MQTT is an Enterprise connector. Add an MQTT dead-letter output only when the architecture republishes to an MQTT topic; this flow does not, so configure it only for that variant.

Workflow manifest

{
"workflow": "iot-centre-flow",
"version": 1,
"summary": "Collect MQTT telemetry and fan out to alert and archive.",
"imports": [
"blueprints/mqtt-collector@1 as mqtt_collector",
"builtins/blueprints/builtin-fanout@2 as fanout",
"blueprints/alert@1 as alert",
"blueprints/archive@1 as archive"
],
"steps": [
{
"id": "mqtt-collect",
"uses": "mqtt_collector",
"out": "channel:telemetry"
},
{ "id": "route", "uses": "fanout", "in": "channel:telemetry", "dispatch": { "mode": "clone" }, "out": { "alert": "channel:alerts", "archive": "channel:archive" } },
{ "id": "alert", "uses": "alert", "in": "channel:alerts" },
{ "id": "archive", "uses": "archive", "in": "channel:archive" }
]
}

Publish mqtt-collector, alert, and archive as blueprint version 1 before importing them. Give the collector one worker-channel output and each downstream Job one worker-channel input; the planner rewrites that single channel to telemetry, alerts, or archive. The alert and archive keys on the fan-out step’s out map are dispatch port names, not Job port names.

Job definitions

name: mqtt-collector
input:
mqtt:
mode: client
endpoint: "mqtt://example.com:1883"
keep-alive-secs: 30
clean-session: true
topics: ["sensors/#"]
payload-mode: auto
qos: 0
request-channel-capacity: 10
inflight: 100
protocol: v3
output:
worker-channel:
worker-channel-name: telemetry
name: alert
input:
worker-channel:
worker-channel-name: alerts
output:
print:
output: stdout
name: archive
input:
worker-channel:
worker-channel-name: archive
output:
file-store:
path: /var/lyftdata/archive
object-name:
name: telemetry
guid-prefix: /
disable-object-name-guid: false

A clone dispatch sends a copy of every event to both branches.

Failure What happens to in-flight data How you recover
MQTT connection drops Broker session and QoS behavior determine redelivery; collector memory is bounded Reconnect with the same topic and session settings
Alert branch fails Archive branch can continue independently; the failed branch may retry and duplicate Repair the alert Job and deduplicate downstream
Collector Worker restarts Events held only in its Worker Channel are lost Rely on broker retention and session behavior, then replay where supported
  • A Worker Channel never crosses Workers. It is bounded in memory and loses in-flight events on a Worker restart.
  • Worker KV is local to one Worker and is not shared between Workers.
  • A spare Worker is capacity, not automatic failover. Placement, access, ownership, and recovery still matter.
  • In the S3 edge shown here, two reader replicas would both list the same prefix. Keep the reader at one replica.
  • Drain an S3 or file-store edge before you tear down or re-create its deployment. A new deployment reads a new prefix, and objects left under the old one are neither delivered nor deleted. See Tear down.
  • Kafka, MQTT, and Redis Streams are connectors that Jobs use. They are not Workflow edge kinds and not kinds of Worker.
  • The Kafka output has no dead-letter setting.
  • MQTT dead-letter settings apply to MQTT publishing only.
  • Fan-out branches deliver independently. A retry can duplicate one branch while another succeeds.

Start with Workflows, then choose Transports and Durability and Placement and Scaling. For routing, see Routing and Fan-out. Practice a durable local boundary with Tutorial: File-store Roundtrip.