Reliability: retry, land, replay
Reliability in LyftData is chosen per output and per edge. An output can retry a failed attempt, while an edge can keep data in a file store, object store, or broker until the next step can read it. This page gives you the outage story, the settings to inspect, and the replay paths; the linked reference pages define each connector’s complete contract.
What happens when a destination is down
Section titled “What happens when a destination is down”Think about the same collection and delivery pair in three frames.
Without a durable edge
Section titled “Without a durable edge”Normal. The collection job hands events to the delivery job through a worker channel or another in-memory hand-off. The output builds a batch, sends it, and waits for the connector’s acknowledgement.
Destination unavailable. The output retries failures that its connector classifies as retryable. Events in the bounded channel and the current output batch wait in memory. A timeout only bounds that attempt; it does not tell you whether the destination accepted work before the connection ended. If the Worker restarts, in-memory events are lost. Recovery then depends on the source reading or resending them again.
Destination back. The output continues or stops according to its retry policy. If the source can resend, a later run can process the events again. The retry and resend paths can overlap, so make the destination idempotent when duplicates matter.
With a durable edge
Section titled “With a durable edge”Normal. On an ordinary file-store or S3 edge, collection hands events across an in-process Worker Channel to a writer Job that lands the object; the source can advance after that hand-off, before the object is durable. The opt-in durable file-store path holds the Kafka origin until the writer commits the object. Delivery reads the landing zone and completes its own destination effect.
Destination unavailable. Collection can keep landing data while delivery retries. The backlog is bounded by the storage capacity and retention you provide. A durable commit and consumer completion are separate; a stored object is not the same as a completed destination write.
Destination back. Delivery discovers the retained data and catches up from the landing zone. Ordinary file-store recovery continues listing and fingerprinting, and a replay_generation adds another object prefix. On the opt-in durable file-store path, an explicit replay starts another pass over selected object IDs and does not replace the unfinished pass. Delivery can repeat a delivered prefix when an acknowledgement was lost, so duplicates are possible.
flowchart LR accTitle: Durable edge during a destination outage accDescr: Collection accepts events into a landing zone while delivery retries, then reads the retained data when the destination returns. S[Source] --> C[Collection] C -->|landing-zone commit| L[(File store or S3 landing zone)] L --> D[Delivery output] D -->|retry while unavailable| X[Destination down] X -->|returns| D D --> E[Destination effect]
See Transports and Durability for placement, storage access, and the durable-edge operating procedure. See Delivery Semantics for acknowledgement and duplicate boundaries.
A retry repeats an attempt after a connector classifies the failure as retryable. It does not guarantee eventual success. The shared retry object uses these keys:
| Key | Meaning | Default or omission behavior |
|---|---|---|
count |
Number of retry attempts | If absent, retries continue indefinitely. |
pause |
Delay before retrying | Defaults to 500ms. |
Output retry accepts count and pause only. If count is absent, retryable failures continue; the connector decides which failures are retryable. The strategy, jitter, and related backoff fields belong to the call action’s retry configuration, not output retry. A connector-specific timeout can bound one attempt. A batch can hold events until its size, byte, or flush timeout is reached, so a failed attempt can cover the whole batch or a connector-defined subset. Check the output reference for the exact acknowledgement and failure rules.
For example, an HTTP output can use a bounded retry and a short batch flush:
output: http-post: url: https://example.com/events retry: count: 6 pause: 2s batch: mode: fixed fixed-size: 100 timeout: 100msThe fields and defaults above are component-specific. Compare HTTP POST, Splunk HEC, Kafka, MQTT, Elasticsearch, S3, and File Store. Kafka and MQTT are Enterprise-edition connectors, as their reference pages state.
Land data on disk or in a store
Section titled “Land data on disk or in a store”A file-store edge puts a visible stage boundary between two workflow steps. Use a local path only when the producer, notifier or poller, and reader share the worker group. Use a genuinely shared path when the steps run on different workers. An S3 edge crosses workers when every assigned worker has network, credential, bucket, and prefix access.
This shipped manifest declares the edge on the producing step’s edges map:
{ "workflow": "file-store-example", "version": 1, "display_name": "File Store Example", "summary": "Reference workflow demonstrating the file-store edge transport adapter.", "imports": ["blueprints/workflow-blueprint@1 as task"], "steps": [ { "id": "producer", "uses": "task", "out": "channel:fs-edge", "edges": { "channel:fs-edge": { "transport": { "kind": "file-store", "path": "fs://var/lyftdata/edges/fs-edge", "roll": {"max_rows": 100000, "max_age": "5m"} } } } }, {"id": "consumer", "uses": "task", "in": "channel:fs-edge"} ]}The producer writes rolled edge objects under the configured fs:// path. The consumer reads the edge objects; the file-store input can target a filesystem root and object names or prefixes. The example shows roll bounds of 100000 rows or 5m; it does not define a deletion or retention period. Your storage capacity, cleanup policy, and retention bound the backlog. The S3 workflow uses a bucket and base_prefix instead; provider listing, visibility, credentials, retention, and cost remain your responsibility. Read File Store input, File Store output, S3 input, S3 output, and the file-store roundtrip.
Replay
Section titled “Replay”Replay is a deliberate operation over retained data. Re-read a file-store or S3 landing zone with its input and object or prefix selection. Ordinary durable-edge recovery continues the unfinished consumer/pass; explicit replay starts another pass, including over completed objects. Protect retained objects until every live and replay obligation finishes.
A cursor reset changes where an input starts reading. It does not recreate deleted landing objects, undo a completed destination effect, or prove that a durable consumer completed. Before clearing runtime state, identify the worker, job, state, and resource, capture the current value, decide the replay range, and account for duplicate writes. The supported artifact commands are:
lyftdata runtime-artifacts list --worker-id <worker-id> --job-id <job-id> --state-id <state-id> --state-kind <state-kind>lyftdata runtime-artifacts get --worker-id <worker-id> --job-id <job-id> --state-id <state-id> --state-kind <state-kind> --resource-key <resource-key>lyftdata runtime-artifacts clear --worker-id <worker-id> --job-id <job-id> --state-id <state-id> --state-kind <state-kind> --resource-key <resource-key>For Kafka, offsets belong to Kafka’s consumer group. The input exposes group-id for coordinated consumption and cursor modes offset, timestamp, and custom, with per-topic/partition initial offsets. Resetting a LyftData cursor does not rewrite broker offsets or make a new consumer group; use the broker’s offset controls and the input’s configured cursor deliberately. See Kafka input.
Worker KV output-delivery worksets expose an Inspect surface with pending, leased, dead-letter, and KV views. Select the Dead-letter tab, then use Replay selected to return records to the pending output queue or Discard selected after review. The supported API operations are the deployment workset dead-letter/replay and dead-letter/discard endpoints. Leases expire back to the queue through the workset operation; output acknowledgement for a popped batch is ack-batch. The Worker KV input’s ack-mode can be ack-batch or ack-each. See Worker KV input and runtime evidence.
Dead-letter outputs
Section titled “Dead-letter outputs”LyftData 3.0 has no single dead-letter queue for a whole Job or edge. What exists is specific to a component:
| Where | What it does |
|---|---|
MQTT output, dead-letter |
Optional. After retries are exhausted, a failed publish goes to the MQTT topic you name. Set topic; qos defaults to 0 and accepts 0 or 1; retain defaults to false. |
| Kafka output | No dead-letter setting. Route rejects to their own topic yourself (below). |
| Worker KV output-delivery worksets | Failed members move to a dead-letter set that you can inspect, replay, or discard (see Replay). |
infer action, on-error |
Accepts fail, skip, or dlq:<name>. With dlq:<name> the action drops the failed event, logs a warning that names the policy, and increments a counter (infer_llm.dlq, infer_embedding.dlq, or infer_anomaly.dlq). It does not send the event to an output. |
To keep rejected events, give them their own destination. The shipped kafka-json-dlq-router library Job shows the pattern for Kafka: set the output topic in a field, overwrite it for rejects, and let the Kafka output read the topic from that field.
actions: - add: description: "Default output topic." output-fields: kafka_topic: events.accepted - add: description: "Send invalid JSON to the reject topic (invalid JSON arrives as _raw)." condition: "_raw != ''" output-fields: kafka_topic: events.rejected reject_reason: "invalid_json"output: kafka: bootstrap-servers: "{{KAFKA_BOOTSTRAP_SERVERS}}" topic-field: kafka_topic payload: json acks: allFor other destinations, branch rejects with a conditional fan-out and give the branch its own output. See Routing and Fan-out.
Where data can still be lost
Section titled “Where data can still be lost”| Situation | What is lost | How to avoid it |
|---|---|---|
| Worker restarts with events in a Worker Channel | In-memory events in the bounded hand-off | Use a file-store, S3, or broker edge when restart loss is unacceptable. |
| Events are batched before an output acknowledges | The in-memory batch, or an ambiguous prefix after a timeout | Bound batch size and timeout; use destination idempotency. |
| An output gives up after its retry count | The unaccepted attempt unless the source or landing zone can resend it | Choose a retry count deliberately and retain a replayable source or landing zone. |
| A landing zone fills or expires | Data that was never durably accepted, or retained data past cleanup | Monitor capacity and retain objects for the full recovery and replay window. |
| A destination acknowledges before it persists | Data the destination later loses | Require a destination acknowledgement that matches its persistence contract. |
| A deployment with a file-store or S3 edge is torn down, or retired and created again, before the edge drains | Objects still in the edge’s store. A new deployment reads a new prefix, and nothing reads or deletes the old one. | Before you tear down, stop the producers and leave the reader Job running with no errors. Wait until the consuming Job’s Events out stops increasing and its deployment-flow out rate stays at zero; a flat Events out while the reader is stopped or failing does not mean the edge is empty. See Tear down. |
Duplicates
Section titled “Duplicates”Retries can deliver twice. LyftData makes no blanket exactly-once claim. A timeout or lost response can leave the destination effect ambiguous, and whole-batch or whole-object retry can repeat a delivered prefix. Use stable business keys, destination-side deduplication, idempotency keys, transactional APIs, or reconciliation when duplicates are unacceptable. See Delivery Semantics.
Choosing for each hop
Section titled “Choosing for each hop”| Need | Choice | What you provide |
|---|---|---|
| Lowest-latency same-worker hand-off and source resend is acceptable | Worker channel | Co-location and a bounded recovery plan. |
| Replayable local or shared stage boundary | File-store edge | Path permissions, atomic visibility, capacity, cleanup, and retention. |
| Cross-worker durable objects | S3 edge | Network and credentials for every worker, bucket/prefix access, listing and retention rules. |
| Broker buffering, partitioning, and consumer-group replay | Kafka or another external broker | Broker acknowledgement, offsets, ownership, ordering, retention, and duplicate policy. |
| Rejected events must be kept | A reject topic or a conditional branch with its own output | A destination and an owner for the rejected data. |