Transports and Durability
A transport determines how events cross a data edge. Select it after dispatch and before placement: an edge that cannot cross the intended worker boundary invalidates the topology.
Decision table
Section titled “Decision table”| Transport | Locality | Durability/replay | Operator concerns |
|---|---|---|---|
| Worker channel | One worker process | In-memory; no external replay | Co-locate producer/consumer, bounded queue, restart loss |
| File-store | A local or deliberately shared path | Files can provide a replayable landing zone | Permissions, retention, atomic visibility, shared-mount truth |
| Object store (for example S3) | Workers with network and credential access | Durable objects with provider-specific read/list semantics | Credentials, prefixes, listing consistency, retention, cost |
| External broker | Workers with broker access | Broker-specific acknowledgement and replay | Consumer groups, partition ownership, ordering, retention, duplicates |
| Provider connector job | Defined by that source/sink connector | Provider-specific | Setup, rate limits, idempotency, receipts; not a generic edge transport |
The selected release’s planner/compiler determines which edge profiles it can materialize directly. A native input or output connector is not automatically an interchangeable workflow edge. Confirm the current editor palette, plan, and Pack readiness rather than inferring support from the DSL catalog.
Worker channels
Section titled “Worker channels”Worker channels are low-latency in-process queues. Both jobs must be placed on the same worker. They do not span worker processes, provide broker-style consumer groups, or survive as a durable replay log. Use them for a bounded same-worker topology whose recovery can restart from the original source.
File-store
Section titled “File-store”File-store creates a visible stage boundary and can support replay/backfill. If the path is local, keep writer, notifier/poller, and reader on the same worker group. Split placement only when both sides use the same genuine shared storage and permissions. A healthy reader on another host does not prove it can see the writer’s local path.
Object stores and brokers
Section titled “Object stores and brokers”Object storage can cross workers when every assigned worker has network, credential, bucket, and prefix access. Define object naming, completion/atomic visibility, retention, cursor/checkpoint, duplicate, and cleanup rules.
For a broker, document acknowledgement timing, partition/key mapping, consumer group ownership, retry/dead-letter behavior, retention, ordering scope, scale-in drain, and offset recovery. Do not replace this contract with a generic “durable” label.
Delivery questions for every edge
Section titled “Delivery questions for every edge”- Can the transport cross the selected worker boundary?
- What is acknowledged, and when?
- Where can an unacknowledged event be recovered?
- What can be duplicated, reordered, or lost during retry or restart?
- Who owns retention, cleanup, and capacity?
- Which credentials and network paths must every eligible worker have?
- What runtime metric and record count proves the edge?
See Delivery Semantics for the full boundary, then choose Placement and Scaling.