Skip to content
LyftData Docs

When the DSL Isn't Enough

The built-in actions cover most parsing, shaping, enrichment, and routing work. When they don’t, LyftData gives you a deliberate escalation ladder instead of a dead end. Work down this list and stop at the first rung that solves the problem:

  1. Compose built-in actions. Check the DSL index first; a sequence such as extract -> convert -> enrich often replaces custom code entirely.
  2. Write an in-process script. The script (Lua) and java-script actions transform events inside the job with no extra deployment surface.
  3. Call an external HTTP service you own. The call action sends the event to a service and continues with its JSON response.
  4. Run a container per call. The call action’s container runner executes an OCI image with the event as input and reads JSON back.

Each rung adds operational surface: a script adds code review, an HTTP service adds a deployment and a network dependency, and a container adds an image supply chain. Prefer the smallest rung that works.

Scripts first: in-process custom logic

Section titled “Scripts first: in-process custom logic”

Before reaching for an external service, check whether the logic fits in a script action. Scripts receive the event as a JSON object and return the transformed event, and they run inside the job — no network hop, no second deployment, no additional failure mode:

actions:
  - java-script:
      source: |
        function transform(event) {
          event.normalized = String(event.raw || "").trim().toLowerCase();
          return event;
        }

Reach past scripts when the work needs something a sandboxed script cannot do: a proprietary library, a model runtime, native code, heavy dependencies, or state that lives in another system.

call with the HTTP runner

Section titled “call with the HTTP runner”

The HTTP runner posts work to a service you operate and continues the pipeline with the response. The contract is JSON-in/JSON-out: the runner sends the request you configure, expects a JSON object (or an array of objects) back, and emits the result as the event(s) flowing downstream. There is no per-field templating of the response — your service owns the payload shape.

actions:
  - call:
      description: Score the event with the fraud service
      runner: http
      http:
        method: post
        url: https://scoring.internal.example/v1/score
        headers:
          Authorization: "${secret|scoring/api_token}"
          Content-Type: application/json
        expected-status: [200]
        retry-on-status: [502, 503]
        redact-headers:
          Authorization: true
      timeout: 15s
      retry:
        max-attempts: 3
        backoff:
          strategy: exponential
          base: 200ms
          max: 10s
          jitter: true
      idempotency-key: "{{ event.id }}"
      concurrency:
        max-in-flight: 5
        queue-depth: 10
      response:
        payload: object
        max-size-bytes: 5000000
      telemetry:
        team: fraud

Field notes:

  • expected-status lists the statuses treated as success (defaults to 2xx). retry-on-status lists statuses that should retry under the retry policy instead of failing immediately.
  • Secrets resolve through ${secret|scope/name} lookups against the deployment’s secrets store — the value is fetched at call time, not interpolated into the job definition. Never inline a credential literal.
  • redact-headers keeps named header values out of logs and telemetry. Redact every credential-bearing header.
  • idempotency-key deduplicates retried work on your side of the contract. Derive it from a stable event identity so a retried call cannot double-apply an effect in the downstream service.
  • concurrency bounds in-flight calls and queued work so a slow service backpressures the pipeline instead of exhausting it.
  • response.max-size-bytes fails the call before parsing an oversized body. Set it; an unbounded response is a resource risk.
  • Distributed-trace headers (traceparent/tracestate) are injected automatically when trace context is present.

A circuit-breaker block is accepted by the schema but not yet enforced at runtime; do not depend on it.

Design the service contract defensively

Section titled “Design the service contract defensively”

The webhook and deployment automation patterns proven in production LyftData estates follow rules worth copying:

  • Keep the service’s request contract closed: require exactly the fields you send and reject unknown fields, so drift fails loudly.
  • Never derive URLs, commands, file paths, or environment names from untrusted event text. The url should be fixed configuration; untrusted data belongs only in the request body.
  • Give privileged effects (deploys, file promotion, host mutations) their own narrowly-scoped service on a private or loopback address, and call it with a closed request schema, rather than granting the worker broad credentials.
  • Check your deployment’s egress policy: if worker egress is restricted, request a narrow exception for the exact host and port, never a blanket private-network allowance.

call with the container runner

Section titled “call with the container runner”

The container runner executes an OCI image per call, passing the event in and reading a JSON result back. Use it when the custom logic ships as an image — a model runtime, a binary tool, code in another language — and you don’t want a standing service.

actions:
  - call:
      description: Run the classifier image
      runner: container
      container:
        image: registry.example.com/classifier:1.4@sha256:8f0a…
        command: ["/app/classify"]
        args: ["--mode", "predict"]
        io:
          mode: stdio
        env:
          MODEL_PROFILE: default
        timeout: 60s
      timeout: 65s
      response:
        payload: object

Two IO modes exist:

  • stdio (default): the event JSON is written to the container’s stdin and the result JSON is read from stdout.

  • artifact: the event is written to input-path inside the container and the result is read from output-path, for tools that work with files:

    io:
      mode: artifact
      input-path: /in/event.json
      output-path: /out/result.json

Pin images by digest (image: …@sha256:…) so the executed code is exactly what was reviewed, and keep the worker’s container daemon access in mind when placing the job: the runner needs a worker host with a local Docker daemon.

Choosing between the runners

Section titled “Choosing between the runners”
Concern HTTP runner Container runner
Custom code lives in A service you deploy and operate An OCI image per call
Latency profile Network round-trip to a warm service Container start per call
State Service can hold state and connections Stateless per invocation
Best for Shared services, privileged effects behind a narrow API, high call rates Packaged tools, model runtimes, occasional or batch-shaped work
Maturity Production-ready Early; mounts/resources/security not yet enforced

Where to go next

Section titled “Where to go next”