Advanced architectures: Worker-to-Worker data flows
Use this page when one Workflow must move data between Workers. You choose a boundary for each hop, place the jobs that can reach that boundary, and keep ownership clear when you add replicas. You should already know Workflows, Transports and Durability, Placement and Scaling, and Routing and Fan-out. Workflow authoring and deployment are available in the licensed editions described in Workflows.
Choosing the edge for each hop
Section titled “Choosing the edge for each hop”| Edge | Crosses Workers? | Survives a Worker restart? | Can be replayed? | What you must provide |
|---|---|---|---|---|
| Worker Channel | No; both Jobs are on one Worker | No; it is bounded in-memory state | No external replay | Co-located Jobs and a compatible channel name |
| file-store edge | Only when the path is genuinely shared | Stored files remain; local paths remain in that Worker’s failure domain | Yes, by listing retained files | path and roll settings; co-locate a local path |
| S3 object-store edge | Yes, when every Worker can reach the store | Yes, subject to provider retention and access | Yes, by listing retained objects | bucket, base prefix, endpoint/region, and role or test credentials |
| broker reached by an output Job and an input Job (Kafka, MQTT, Redis Streams) | Yes, when both Jobs can reach the broker | Broker-specific | Broker-specific offsets, cursors, or session rules | connector fields, credentials, topic/stream, acknowledgement, and ownership policy |
LyftData 3.0 has edge adapters for two durable transport kinds, file-store and s3; worker-channel is the in-memory default. An object-store edge therefore means S3 or an S3-compatible store. Kafka, MQTT, Redis Streams, Google Cloud Storage, and Azure Blob Storage are connectors that Jobs use: to cross Workers through one of them, end one step with an output Job and start the next with an input Job.
Large enterprise, many Workers
Section titled “Large enterprise, many Workers”Use this shape when sources are spread across sites or teams and a core Worker group owns delivery. Put a durable edge between collection and delivery, then add replicas only where the input contract assigns work to one consumer.
flowchart LR
accTitle: Enterprise collection and delivery
accDescr: Collection Workers write events to an S3 prefix. A core delivery Worker reads the retained prefix and sends events to a destination.
c1["Collection Worker"] -->|"S3 edge"| store[("S3 bucket\nevents.raw/")]
c2["Collection Worker"] -->|"S3 edge"| store
store --> d["Core delivery Worker"]
d --> sink["Destination Job"]
Declare the transport on the producing step’s edges map. The planner expands the transport kind to its adapter blueprint; the adapter definitions contain the generated writer and reader Jobs. Use a role ARN in a deployed environment, and keep inline access keys to local testing.
Workflow manifest
This import is the pass-through placeholder used by the shipped S3 example. Publish your collector as a blueprint with the real input and exactly one worker-channel output, then use that blueprint instead of task. The planner rewrites that one output to the channel named in out. Publish a delivery blueprint with the matching worker-channel input before importing it as well.
{ "workflow": "enterprise-worker-flow", "version": 1, "summary": "Collect at source Workers and deliver from a core group.", "imports": [ "blueprints/enterprise-collector@1 as collector", "blueprints/core-delivery@1 as delivery" ], "steps": [ { "id": "collect", "uses": "collector", "out": "channel:events.raw", "edges": { "channel:events.raw": { "transport": { "kind": "s3", "bucket": "events-raw", "base_prefix": "events.raw/", "region": "us-east-1", "endpoint": "https://example.com", "role_arn": "arn:aws:iam::123456789012:role/lyftdata-events", "poll_interval": "10s" } } } }, { "id": "deliver", "uses": "delivery", "in": "channel:events.raw", "replicas": 1 } ]}The step IDs are placement anchors. out, in, bucket, and base_prefix bind the named channel and durable location. Place collect near sources and deliver in the core group. In this configuration two replicas of the S3 reader would both list the same prefix, so keep the reader at one replica.
The expanded adapter contains a s3-worker-channel-writer with a worker-channel input and S3 output, and a s3-worker-channel-reader with an S3 list-and-download input and worker-channel output. Treat those generated Jobs as implementation details of the edge.
| Failure | What happens to in-flight data | How you recover |
|---|---|---|
| Collection Worker restarts | Events still in its in-memory input can be lost; committed objects remain | Restart collection from the source and let the reader list retained objects |
| Core reader restarts | Stored objects remain; an unfinished read depends on reader state | Restart the reader and inspect its fingerprint state before replay |
| Delivery Worker leaves | Retained objects remain available to an eligible reader | Restore one eligible Worker and keep the reader at one replica |
Hybrid, on-premises to cloud or VPC
Section titled “Hybrid, on-premises to cloud or VPC”Use this shape when local Workers can reach sources but the cloud group owns the destination. Choose an S3 edge when a shared bucket is the boundary, or use an output Job and input Job around a broker when the boundary is a broker. Keep file-store for one Worker or a genuinely shared path.
flowchart LR
accTitle: Hybrid on-premises to cloud flow
accDescr: On-premises collection writes to an S3 bucket. Cloud delivery reads the retained prefix and sends it onward.
src["On-prem source"] --> local["Local collection and shaping"]
local -->|"S3 edge"| bucket[("Shared S3 bucket\nhybrid/")]
bucket --> cloud["Cloud Worker group"]
cloud --> dest["Cloud delivery Job"]
Workflow manifest
This import is the pass-through placeholder used by the shipped S3 example. Publish your collector as a blueprint with the real input and exactly one worker-channel output, then use that blueprint instead of task. The planner rewrites that one output to the channel named in out. Publish a delivery blueprint with the matching worker-channel input before importing it as well.
{ "workflow": "hybrid-worker-flow", "version": 1, "summary": "Shape locally and deliver from the cloud group.", "imports": [ "blueprints/hybrid-collector@1 as collector", "blueprints/cloud-delivery@1 as delivery" ], "steps": [ { "id": "local-shape", "uses": "collector", "out": "channel:hybrid.events", "edges": { "channel:hybrid.events": { "transport": { "kind": "s3", "bucket": "hybrid-events", "base_prefix": "hybrid/", "region": "us-east-1", "endpoint": "https://example.com", "role_arn": "arn:aws:iam::123456789012:role/lyftdata-hybrid" } } } }, { "id": "cloud-deliver", "uses": "delivery", "in": "channel:hybrid.events", "replicas": 1 } ]}For a file-store edge, use kind: "file-store", a shared path such as fs://var/lyftdata/edges/hybrid, and roll with max_rows and max_age. A local path stays in one Worker’s failure domain.
Job definitions for the broker variant
name: shaped-events-to-kafkainput: worker-channel: worker-channel-name: shaped-eventsoutput: kafka: bootstrap-servers: "{{KAFKA_BROKER}}" topic: events.raw payload: json acks: allname: kafka-deliverycontext: KAFKA_BROKER: "{{KAFKA_BROKER}}" KAFKA_GROUP: hybrid-deliveryinput: kafka: bootstrap-servers: "{{KAFKA_BROKER}}" group-id: "{{KAFKA_GROUP}}" topics: - events.raw payload: jsonoutput: worker-channel: worker-channel-name: deliveryResolve KAFKA_* values from the Credential Manager or deployment context. Kafka is an Enterprise connector. Use a separate routing Job for a dead-letter topic.
| Failure | What happens to in-flight data | How you recover |
|---|---|---|
| Boundary link is down | S3 objects or broker records remain under their retention rules | Restore connectivity, then resume the reader or consumer group |
| Shared path is mounted locally by mistake | The cloud Worker sees no local files | Move the boundary to genuine shared storage or a broker |
| Cloud delivery Worker restarts | Retained objects or broker records remain; in-memory work may not | Restart with the same group, prefix, and ownership settings |
IoT or edge to centre
Section titled “IoT or edge to centre”Use MQTT when devices already publish telemetry to an edge broker. A central collector subscribes once, then a clone fan-out sends one copy to alerting and one to archive.
flowchart LR
accTitle: IoT MQTT collection and fan-out
accDescr: Devices publish to MQTT. A collector receives telemetry and clones it to alert and archive Jobs.
devices["Devices"] --> mqtt[("MQTT broker\nsensors/#")]
mqtt --> collector["mqtt-collect"]
collector --> fan{{"clone"}}
fan --> alert["alert"]
fan --> archive["archive"]
Keep the MQTT subscriber in the central group and put the fan-out after it. MQTT is an Enterprise connector. Add an MQTT dead-letter output only when the architecture republishes to an MQTT topic; this flow does not, so configure it only for that variant.
Workflow manifest
{ "workflow": "iot-centre-flow", "version": 1, "summary": "Collect MQTT telemetry and fan out to alert and archive.", "imports": [ "blueprints/mqtt-collector@1 as mqtt_collector", "builtins/blueprints/builtin-fanout@2 as fanout", "blueprints/alert@1 as alert", "blueprints/archive@1 as archive" ], "steps": [ { "id": "mqtt-collect", "uses": "mqtt_collector", "out": "channel:telemetry" }, { "id": "route", "uses": "fanout", "in": "channel:telemetry", "dispatch": { "mode": "clone" }, "out": { "alert": "channel:alerts", "archive": "channel:archive" } }, { "id": "alert", "uses": "alert", "in": "channel:alerts" }, { "id": "archive", "uses": "archive", "in": "channel:archive" } ]}Publish mqtt-collector, alert, and archive as blueprint version 1 before importing them. Give the collector one worker-channel output and each downstream Job one worker-channel input; the planner rewrites that single channel to telemetry, alerts, or archive. The alert and archive keys on the fan-out step’s out map are dispatch port names, not Job port names.
Job definitions
name: mqtt-collectorinput: mqtt: mode: client endpoint: "mqtt://example.com:1883" keep-alive-secs: 30 clean-session: true topics: ["sensors/#"] payload-mode: auto qos: 0 request-channel-capacity: 10 inflight: 100 protocol: v3output: worker-channel: worker-channel-name: telemetryname: alertinput: worker-channel: worker-channel-name: alertsoutput: print: output: stdoutname: archiveinput: worker-channel: worker-channel-name: archiveoutput: file-store: path: /var/lyftdata/archive object-name: name: telemetry guid-prefix: / disable-object-name-guid: falseA clone dispatch sends a copy of every event to both branches.
| Failure | What happens to in-flight data | How you recover |
|---|---|---|
| MQTT connection drops | Broker session and QoS behavior determine redelivery; collector memory is bounded | Reconnect with the same topic and session settings |
| Alert branch fails | Archive branch can continue independently; the failed branch may retry and duplicate | Repair the alert Job and deduplicate downstream |
| Collector Worker restarts | Events held only in its Worker Channel are lost | Rely on broker retention and session behavior, then replay where supported |
Limits
Section titled “Limits”- A Worker Channel never crosses Workers. It is bounded in memory and loses in-flight events on a Worker restart.
- Worker KV is local to one Worker and is not shared between Workers.
- A spare Worker is capacity, not automatic failover. Placement, access, ownership, and recovery still matter.
- In the S3 edge shown here, two reader replicas would both list the same prefix. Keep the reader at one replica.
- Drain an S3 or file-store edge before you tear down or re-create its deployment. A new deployment reads a new prefix, and objects left under the old one are neither delivered nor deleted. See Tear down.
- Kafka, MQTT, and Redis Streams are connectors that Jobs use. They are not Workflow edge kinds and not kinds of Worker.
- The Kafka output has no dead-letter setting.
- MQTT dead-letter settings apply to MQTT publishing only.
- Fan-out branches deliver independently. A retry can duplicate one branch while another succeeds.
Next steps
Section titled “Next steps”Start with Workflows, then choose Transports and Durability and Placement and Scaling. For routing, see Routing and Fan-out. Practice a durable local boundary with Tutorial: File-store Roundtrip.