Skip to content
LyftData Docs

Tutorial: Application Log Router

This advanced tutorial turns two application-log shapes into one governed stream. Invalid records go to quarantine; valid records are cloned to a durable full archive and a critical-event evidence branch.

Outcome and topology

Section titled “Outcome and topology”

Two log shapes normalize and route to quarantine, durable archive, and critical evidence

gateway JSONL (12) -- normalize --\
                                   validate -- conditional valid/invalid
worker JSONL  (12) -- normalize --/              |            |
                                                  |            +-> quarantine (5)
                                                  +-> clone
                                                       archive  -> durable store (19)
                                                       critical -> filter -> evidence (6)

Fixture contract

Section titled “Fixture contract”
d07fba52217352f5cfae52fe8affbd4f7abbc5f20371ed2f75cc99bae04a5895  gateway.jsonl
7f0a18e74a23ce431d5386a17d8d993f860cfc50e287c27ae515d9e790d541cd  worker.jsonl
421e9e5050bafc3e55756059ec28adf57522098b521dc5ac2675d23e90c23a9b  expected-summary.json

All data is synthetic. Copy both JSONL files to fresh run-owned paths visible to the source worker.

Normalize the source shapes

Section titled “Normalize the source shapes”

Create two finite Files source paths. Gateway records already use source, severity, and message. Map worker fields as follows while preserving event_id and timestamp:

Worker field Canonical field
component source
level severity
text message

The canonical required fields are event_id, timestamp, source, severity, and message.

Validate and route

Section titled “Validate and route”
  1. Mark a record valid only when all five required fields are present and non-empty.
  2. Add a conditional routing step with valid and invalid ports. Send missing/unmatched values to invalid rather than dropping them.
  3. Send invalid to a run-owned quarantine file/evidence sink.
  4. Send valid to a clone step with archive and critical ports.
  5. Use a durable file-store edge and archive sink for archive.
  6. Put the predicate severity == "critical" on the critical branch and write it to a separate local evidence sink.
  7. Validate the graph and inspect both dispatch definitions in the manifest.

Placement and storage

Section titled “Placement and storage”

For the first proof, use one worker group and co-locate local paths, worker channels, file-store writer, and reader/evidence jobs. If you later split placement, move the durable boundary to a genuine shared mount or object store and prove every worker’s access. Do not treat the same path string on two hosts as shared storage.

Publish, plan, and apply

Section titled “Publish, plan, and apply”

Publish the reviewed workflow, create a run-owned deployment, and plan it. Review generated jobs/transports, one-group placement, replica intent, credentials, path permissions, warnings, and teardown scope. Record immutable identities, then apply the unchanged fresh plan.

Acceptance evidence

Section titled “Acceptance evidence”
Check Expected
Input records 24
Valid archive records 19
Quarantine records 5
Critical records 6
Unexplained loss 0
Unexpected duplicates 0

The critical IDs must be gw-002, gw-006, gw-010, wk-001, wk-005, and wk-009. Verify archive and quarantine are disjoint and together contain all 24 input IDs.

MCP automation path

Section titled “MCP automation path”

Follow Author and Deploy Workflows with MCP. Ask the assistant to report the canonical schema, both routing decisions, transport/placement/storage boundary, expected arithmetic, proposed writes, and cleanup identities. Keep preview in read-only; approve upsert, publish, create, plan, and apply separately in the write-tier session.

Optional external HTTP branch

Section titled “Optional external HTTP branch”

Replace the local critical evidence sink only after the base proof passes. Use a managed credential reference—not a raw header in the workflow—plus a run correlation field and destination namespace. After runtime success, query the destination and prove the six IDs. An HTTP status or worker log alone is not acceptance. Remove the exact external run-owned objects during teardown.

Failure and recovery

Section titled “Failure and recovery”
  • Arithmetic fails: preserve artifacts and reconcile every input ID across archive/quarantine before retrying.
  • Archive exists but reader is empty: check shared-storage truth, path permissions, completion visibility, and placement.
  • Critical count is wrong: inspect normalized severity values before the filter.
  • Duplicate records: check source fingerprints, source/reader replicas, retries, and sink idempotency.
  • Apply interrupted: inspect proposal/status/events before any retry; re-plan only when the current state is understood.

Teardown and second cycle

Section titled “Teardown and second cycle”

Preview cleanup, then remove exact recorded deployments, jobs, channels, groups, workflow drafts, local files, and optional external objects. Re-list all run-owned surfaces and require zero residue. Start a new cycle with a fresh prefix and paths; it must reproduce the same 24/19/5/6 arithmetic.