Advanced correlation: state, windows and multi-source alerts
Correlation connects events that share an identity or arrive within a useful period. This page builds a two-source overlap check and a multi-source alert that fires once per key in a ten-minute window. Start with Workflows, review Routing and Fan-out, and keep the Worker KV and Worker KV Get references open.
The building blocks
Section titled “The building blocks”| Primitive | Role in correlation |
|---|---|
| Script | Derives a canonical key, normalises values, and shapes results. |
| Filter | Keeps valid records, accepted revisions, or alert transitions. |
| Worker KV Get | Reads one value by a static key or an event key-field. |
| Worker KV (set) | Stores a value; ttl-ms expires a window entry. set-if-absent makes a first-writer suppression key. |
| Fan Out | Clones one stream to result, alert, and audit branches. |
| Worker Channel | Connects Jobs on one Worker with named in and out channels. |
| Inputs | HTTP Server accepts JSON webhook events; Files reads JSON lines from a worker-visible path. |
| Outputs | HTTP Post sends alerts to an HTTP endpoint; Print writes a local development stream. |
LyftData 3.0 has no built-in join operator. The aggregate action groups events by group-by, computes configured metrics such as count, and can use an ISO 8601 time-field with a tumbling window; the patterns below use explicit keys so you control matching and suppression.
Pattern 1: Compare two sources: overlap and conflict
Section titled “Pattern 1: Compare two sources: overlap and conflict”Give both feeds the same key, such as entity_id. Keep two small values per key and source in Worker KV: the latest accepted revision under rev/<key>/<source>, and its normalised value under val/<key>/<source>. For each event, read your own stored revision and the other source’s stored value, compare, and store the new revision and value only when the revision is newer. A revision that is not newer than the stored one is ignored.
Source A:
{"entity_id":"entity-001","source":"A","revision":1,"value":20,"unit":"C"}{"entity_id":"entity-002","source":"A","revision":1,"value":7,"unit":"C"}Source B:
{"entity_id":"entity-001","source":"B","revision":1,"value":68,"unit":"F"}{"entity_id":"entity-001","source":"B","revision":2,"value":77,"unit":"F"}{"entity_id":"entity-001","source":"B","revision":1,"value":68,"unit":"F"}flowchart LR
accTitle: Two-source overlap and conflict
accDescr: Two source feeds are normalised to a shared key, compared with keyed Worker KV state, and cloned to result, notification, and ignored revision branches.
a["Source A"] --> n["Normalise key and units"]
b["Source B"] --> n
n --> c["Get own revision and other value"] --> s["Accept newer revision"]
s --> k["Set Worker KV"] --> f{{"Clone"}}
f --> r["Results filter: overlap"]
f --> q["Conflict filter: conflict"]
f --> i["Ignored filter: accepted == false"]
The following definitions use the Job shape documented for individual Jobs. Replace paths and endpoints with your values. Keep every comparison step on the same Worker group because the channels and KV namespace are local.
name: source-ainput: { files: { path: "/data/source-a/*.jsonl", payload-mode: json } }output: { worker-channel: { worker-channel-name: compare-in } }name: source-binput: { files: { path: "/data/source-b/*.jsonl", payload-mode: json } }output: { worker-channel: { worker-channel-name: compare-in } }name: compareinput: { worker-channel: { worker-channel-name: compare-in } }actions: - filter: { how: { expression: "entity_id ~= nil and source ~= nil and revision ~= nil and value ~= nil" } } - script: overwrite: true let: other_source: 'source == "A" and "B" or "A"' rev_key: '"rev/" .. entity_id .. "/" .. source' val_key: '"val/" .. entity_id .. "/" .. source' other_val_key: '"val/" .. entity_id .. "/" .. other_source' value_c: 'unit == "F" and (value - 32) * 5 / 9 or value' - worker-kv-get: { namespace: "demo:comparison", key-field: rev_key, output-field: stored_revision } - worker-kv-get: { namespace: "demo:comparison", key-field: other_val_key, output-field: other_value_c } - script: overwrite: true let: accepted: 'is_null(stored_revision) or revision > stored_revision' overlap: 'not is_null(other_value_c)' difference: 'overlap and abs(value_c - other_value_c) or nil' status: 'not overlap and "waiting" or difference > 0.5 and "conflict" or "overlap"' - worker-kv: { namespace: "demo:comparison", condition: accepted, set: { key-field: rev_key, value-field: revision } } - worker-kv: { namespace: "demo:comparison", condition: accepted, set: { key-field: val_key, value-field: value_c } }output: { worker-channel: { worker-channel-name: compare-out } }name: resultsinput: { worker-channel: { worker-channel-name: result } }actions: - filter: { how: { expression: 'accepted and status == "overlap"' } }output: { http-post: { url: "https://example.com/correlation/results" } }name: notificationsinput: { worker-channel: { worker-channel-name: notify } }actions: - filter: { how: { expression: 'accepted and status == "conflict"' } }output: { http-post: { url: "https://example.com/correlation/alerts" } }name: ignoredinput: { worker-channel: { worker-channel-name: ignored } }actions: - filter: { how: { expression: "accepted == false" } }output: { print: { output: stdout } }Publish collector, feed-health-record, correlate, alert-destination, and alert-audit as blueprint version 1 under the import names below, with one worker-channel input or output each, and confirm them in the Workflow palette. The collection-point Jobs remain separate HTTP producers; the central workflow owns the recorder, correlator, fan-out, and destinations.
{ "workflow": "source-comparison", "version": 1, "summary": "Compare two revisions per entity", "imports": [ "blueprints/source-a@1 as source_a", "blueprints/source-b@1 as source_b", "blueprints/compare@1 as compare", "builtins/blueprints/builtin-fanout@2 as fanout", "blueprints/results@1 as results", "blueprints/notifications@1 as notifications", "blueprints/ignored@1 as ignored" ], "steps": [ { "id": "source-a", "uses": "source_a", "out": "channel:compare-in" }, { "id": "source-b", "uses": "source_b", "out": "channel:compare-in" }, { "id": "compare", "uses": "compare", "in": "channel:compare-in", "out": "channel:compare-out" }, { "id": "route", "uses": "fanout", "in": "channel:compare-out", "dispatch": { "mode": "clone" }, "out": { "results": "channel:result", "notify": "channel:notify", "ignored": "channel:ignored" } }, { "id": "results", "uses": "results", "in": "channel:result" }, { "id": "notifications", "uses": "notifications", "in": "channel:notify" }, { "id": "ignored", "uses": "ignored", "in": "channel:ignored" } ]}Publish source-a, source-b, compare, results, notifications, and ignored as blueprint version 1 under those import names, with one worker-channel input or output each, and confirm them in the Workflow palette. The planner rewrites that single channel to the step’s in or out.
The manifest imports each published Job as a Blueprint and uses the built-in Fan Out blueprint with clone dispatch. Both source steps write to the same channel. Every branch receives every event; the filter at the head of each branch Job keeps only its own.
Expected output for the sample:
| Event | Stored? | Status | Branch that emits it |
|---|---|---|---|
A, entity-001, revision 1 (20 °C) |
Yes | waiting (nothing from B yet) |
None |
A, entity-002, revision 1 |
Yes | waiting |
None |
B, entity-001, revision 1 (68 °F = 20 °C) |
Yes | overlap, difference 0 |
Results |
B, entity-001, revision 2 (77 °F = 25 °C) |
Yes | conflict, difference 5 |
Notifications |
B, entity-001, revision 1 again |
No | conflict; not stored because revision 1 is not newer than 2; only Ignored emits it |
Ignored |
Pattern 2: Detect across sources with a window, and alert once
Section titled “Pattern 2: Detect across sources with a window, and alert once”Send each collection point’s matching events to a central collector. Collection points are separate Workers that send to the central collector’s HTTP endpoint at https://collector.example.com/events. The collector, correlator, route, and output steps run together on the central Worker so their channels and KV namespace are local.
The correlator stores the collection point’s name under seen/<key>/<source> with a ten-minute TTL. When an event arrives from the other collection point inside that window, it reads alerted/<key>; set-if-absent creates that key for the same ten minutes, so repeats stay quiet. The value read from the other point’s key goes into the alert, so the alert names both collection points. The window runs from the time each event is processed, not from its event_time. Fan out the alert to an alert destination and an audit copy.
{"user_id":"user-017","collection_point":"collection-point-1","event_time":"2026-11-11T08:00:00Z"}{"user_id":"user-017","collection_point":"collection-point-2","event_time":"2026-11-11T08:03:00Z"}{"user_id":"user-017","collection_point":"collection-point-2","event_time":"2026-11-11T08:04:00Z"}flowchart LR
accTitle: Multi-source window alert
accDescr: Separate collection point Workers post to a central collector, which correlates two points in a TTL window and clones one enriched alert to alert and audit outputs.
p1["Collection point 1 Worker"] --> co["Central collector Worker"]
p2["Collection point 2 Worker"] --> co
co --> g["Get other source and alert key"] --> w["Set source with TTL"] --> e["Enrich alert with both points"] --> d{{"Clone"}}
d --> al["Alert output"]
d --> au["Audit output"]
name: collection-point-1input: { http-server: { address: "0.0.0.0:18001", path: "/events", only-body: true, json: true } }output: { http-post: { url: "https://collector.example.com/events" } }name: collection-point-2input: { http-server: { address: "0.0.0.0:18002", path: "/events", only-body: true, json: true } }output: { http-post: { url: "https://collector.example.com/events" } }name: collectorinput: { http-server: { address: "0.0.0.0:18100", path: "/events", only-body: true, json: true } }output: { worker-channel: { worker-channel-name: correlate-in } }name: correlateinput: { worker-channel: { worker-channel-name: correlate-in } }actions: - filter: { how: { expression: "user_id ~= nil and collection_point ~= nil and event_time ~= nil" } } - script: overwrite: true let: own_key: '"seen/" .. user_id .. "/" .. collection_point' other_point: 'collection_point == "collection-point-1" and "collection-point-2" or "collection-point-1"' other_key: '"seen/" .. user_id .. "/" .. other_point' alert_key: '"alerted/" .. user_id' - worker-kv-get: { namespace: "demo:window", key-field: other_key, output-field: other_seen } - worker-kv-get: { namespace: "demo:window", key-field: alert_key, output-field: prior_alert } - worker-kv: { namespace: "demo:window", set: { key-field: own_key, value-field: collection_point, ttl-ms: 600000 } } - script: overwrite: true let: alert: 'not is_null(other_seen) and is_null(prior_alert)' alert_message: 'alert and ("Both collection points saw " .. user_id .. ": " .. collection_point .. " and " .. other_seen) or nil' - worker-kv: { namespace: "demo:window", condition: alert, set-if-absent: { key-field: alert_key, value-field: alert_message, ttl-ms: 600000 } } - filter: { how: { expression: "alert" } }output: { worker-channel: { worker-channel-name: alert-out } }name: alert-destinationinput: { worker-channel: { worker-channel-name: alert } }output: { http-post: { url: "https://example.com/correlation/alerts" } }name: alert-auditinput: { worker-channel: { worker-channel-name: audit } }output: { print: { output: stdout } }{ "workflow": "multi-source-window", "version": 1, "summary": "Alert once when two collection points see the same key inside a window", "imports": [ "blueprints/collector@1 as collector", "blueprints/feed-health-record@1 as feed_health_record", "blueprints/correlate@1 as correlate", "builtins/blueprints/builtin-fanout@2 as fanout", "blueprints/alert-destination@1 as alert_destination", "blueprints/alert-audit@1 as alert_audit" ], "steps": [ { "id": "collector", "uses": "collector", "out": "channel:collection-events" }, { "id": "record-route", "uses": "fanout", "in": "channel:collection-events", "dispatch": { "mode": "clone" }, "out": { "record": "channel:health-in", "correlate": "channel:correlate-in" } }, { "id": "record", "uses": "feed_health_record", "in": "channel:health-in", "out": "channel:health-out" }, { "id": "correlate", "uses": "correlate", "in": "channel:correlate-in", "out": "channel:alert-out" }, { "id": "route", "uses": "fanout", "in": "channel:alert-out", "dispatch": { "mode": "clone" }, "out": { "alert": "channel:alert", "audit": "channel:audit" } }, { "id": "alert-destination", "uses": "alert_destination", "in": "channel:alert" }, { "id": "alert-audit", "uses": "alert_audit", "in": "channel:audit" } ]}The two collection-point Jobs run on the Workers at each collection point; they are not steps of this Workflow. The Workflow is the central part: collector, feed-health recorder, correlator, fan-out, and the two outputs, all on the central Worker.
Expected output for the sample:
| Event | What the correlator finds | Result |
|---|---|---|
collection-point-1, 08:00 |
No entry for the other point | No alert; seen/user-017/collection-point-1 stored |
collection-point-2, 08:03 |
The other point’s entry, and no suppression key | One alert naming both points, one audit copy; suppression key stored |
collection-point-2, 08:04 |
The other point’s entry, and the suppression key | No alert |
Know when a feed goes quiet
Section titled “Know when a feed goes quiet”On the central Worker, record last_seen_ms/<source> as each event passes through the collector, in namespace demo:feed-health. Run feed-health-record and feed-health-check on that same Worker. A timestamp written on a collection-point Worker is not visible to the check. Feed health here means silence detection: this configuration does not maintain a volume counter. A scheduled check reads the timestamp, computes its age with now_milliseconds(), and emits feed_silent only when the age exceeds ten minutes.
name: feed-health-recordinput: { worker-channel: { worker-channel-name: health-in } }actions: - script: overwrite: true let: { last_key: '"last_seen_ms/" .. source', last_seen_ms: 'now_milliseconds()' } - worker-kv: { namespace: "demo:feed-health", set: { key-field: last_key, value-field: "last_seen_ms" } }output: { worker-channel: { worker-channel-name: health-out } }name: feed-health-checkinput: echo: trigger: interval: duration: 60s json: true event: '{"source":"collection-point-1"}'actions: - script: overwrite: true let: { last_key: '"last_seen_ms/" .. source', threshold_ms: '600000' } - worker-kv-get: { namespace: "demo:feed-health", key-field: last_key, output-field: last_seen_ms } - script: overwrite: true let: feed_silent: 'is_null(last_seen_ms) or now_milliseconds() - last_seen_ms > threshold_ms' alert_kind: 'feed_silent and "feed_silent" or nil' alert_message: 'feed_silent and ("Feed " .. source .. " has been silent for more than ten minutes") or nil' - filter: { how: { expression: "feed_silent" } }output: { http-post: { url: "https://example.com/correlation/feed-health" } }Publish feed-health-record and feed-health-check as blueprint version 1 and confirm both names in the Workflow palette. Run the check on a schedule by setting the echo input’s trigger.interval. A source that has never been seen also counts as silent:
{ "workflow": "scheduled-feed-health", "version": 1, "summary": "Check once a minute whether a feed has gone quiet", "imports": ["blueprints/feed-health-check@1 as feed_health_check"], "steps": [ { "id": "check", "uses": "feed_health_check" } ]}When something fails
Section titled “When something fails”| Situation | What happens | What you do |
|---|---|---|
| Worker restart | Events that were in a Worker Channel at that moment are lost. Worker KV is a database file on the Worker’s disk, so entries that have not expired are still there when the Worker starts again. | Resend the lost events from the source, and keep correlation state rebuildable. |
| Window state expires | The next event starts a new window and can alert again. | Choose a TTL that matches the business window and explain the repeat interval to operators. |
| Duplicate delivery after a retry | Clone does not deduplicate, and a retry can repeat an alert unless the key, revision, or suppression entry still matches. | Make destinations idempotent or retain a stable deduplication key. |
| Collector and collection point use different Workers | Worker Channels do not bridge Workers. | Use the HTTP edge shown above and keep KV steps together on the central Worker. |
Limits
Section titled “Limits”- Worker KV is per Worker. Every step that reads or writes one correlation state must run on the same Worker; pin them to one Worker group.
- A Worker Channel joins Jobs on the same Worker only.
- Kafka and MQTT are Enterprise-edition connectors used by Jobs, not Worker types.
- Worker KV limits namespaces, workset members, and outstanding leases, and sets no size limit for a single value; see Advanced workflows. Its contents stay on the Worker that wrote them, so keep windows short and state rebuildable.
- Retries can deliver an alert twice unless the destination deduplicates.
Next steps
Section titled “Next steps”Read Workflows, Routing and Fan-out, Transports and Durability, Placement and Scaling, and Triggers.