Skip to content
LyftData Docs

HTTP Endpoints (Webhooks, API Polling, Delivery)

LyftData can interact with HTTP endpoints in three ways:

  • Receive webhooks with the http-server input.
  • Poll remote APIs on a schedule with the http-poll input.
  • Deliver events to downstream services with the http-post (or http-get) output.

This guide is operator-focused (recipes and common gotchas). For full schema details, use:

Receive webhooks with http-server

Use http-server when an upstream system pushes events to LyftData (webhooks, alerts, callbacks).

Key fields:

  • address (required) and optional path for routing.
  • tls to enable HTTPS.
  • source-ip-field to capture the caller IP.
  • custom-response and content-type to control the response body.
  • only-body, json, and ignore-linebreaks to control how request bodies become events.

Example: accept JSON webhooks over HTTPS

input:
  http-server:
    address: 0.0.0.0:8443
    path: /webhooks/ingest
    tls:
      cert: /etc/lyftdata/certs/server.crt
      key: /etc/lyftdata/certs/server.key
    source-ip-field: source_ip
    custom-response: '{"status":"accepted"}'
    content-type: application/json
    json: true
    ignore-linebreaks: true

Notes:

  • http-server is an ingestion surface. For authentication and rate limiting, place it behind your API gateway / reverse proxy.
  • If the webhook sends non-JSON payloads, set json: false and parse later with actions.

Poll APIs with http-poll

Use http-poll to run scheduled API requests and emit the responses as events.

Key fields:

  • url, method, headers, query, and optional body / form-urlencoded-body.
  • trigger.interval or trigger.cron for scheduling.
  • payload-mode (auto, json, raw, binary) to control how the body is interpreted.
  • ignore-line-breaks and document-mode to control event framing.
  • response.* to include status codes and headers in the emitted event.
  • retry and timeout for reliability.
  • pagination when the upstream API spans multiple pages.

Example: poll a JSON endpoint and capture status/headers

input:
  http-poll:
    url: https://status.example.com/api/incidents
    trigger:
      interval:
        duration: 60s
    method: get
    headers:
      Accept: application/json
    response:
      status-field: http_status
      headers-field: response_headers
      response-field: body
    payload-mode: json
    ignore-line-breaks: true

Authentication recipes

API keys and bearer tokens (auth.header-token)

input:
  http-poll:
    url: https://api.example.com/v1/events
    method: get
    auth:
      header-token:
        token: "${secret|api/token}"
    headers:
      Accept: application/json
    payload-mode: json
    ignore-line-breaks: true

This sets the Authorization header to Bearer <token> by default. Override auth-header or set prefix: "" when needed.

OAuth2 via Credential Manager (auth.oauth2-via-credential-manager)

input:
  http-poll:
    url: https://api.example.com/v1/events
    auth:
      oauth2-via-credential-manager:
        credential-id: "2b64d2a0-0b76-4cd0-9ef0-3f0ef3d2a9a1"
        credential-scope-mode: allowed
    payload-mode: json
    ignore-line-breaks: true

Use credential-scope-mode: custom with credential-scopes: [...] when the API requires explicit scopes.

AWS SigV4 signing (auth.aws-sig-v4)

input:
  http-poll:
    url: https://abc123.execute-api.us-east-1.amazonaws.com/prod/events
    auth:
      aws-sig-v4:
        access-key-id: "${secret|aws/access_key_id}"
        secret-access-key: "${secret|aws/secret_access_key}"
        region: us-east-1
        service: execute-api
    payload-mode: json
    ignore-line-breaks: true

Pagination patterns (including RSS/Atom)

http-poll supports multiple pagination strategies:

  • Cursor (pagination.strategy: cursor): extract cursors and/or next-page URLs from a JSON response and persist the cursor in Worker KV (so the next run can resume).
  • Link header (pagination.strategy: link-header): follow RFC8288 Link: <...>; rel="next" headers within a run (stateless; no checkpoint).
  • Query param (pagination.strategy: query-param): increment a numeric query parameter (for example ?page=2, ?paged=3) and persist the “next value to request” (defaults to runtime artifacts; Worker KV is an explicit opt-in).

Example: RSS/Atom polling with WordPress-style paged pagination

This example polls an RSS feed, paginates via ?paged=, parses XML into JSON, and expands RSS items into events.

name: rss-wordpress-news-poll
description: "Poll an RSS feed via http-poll and paginate via a page query-param."


input:
  http-poll:
    trigger:
      interval:
        duration: 300s
    url: https://wordpress.org/news/feed/
    method: get
    headers:
      Accept: application/rss+xml
    ignore-line-breaks: true
    document-mode: true
    payload-mode: raw


    pagination:
      strategy: query-param
      param: paged
      start: 2
      step: 1
      max-pages-per-run: 10
      # checkpoint defaults to runtime-artifacts.
      # terminal-status-codes defaults to [404].
      continue-if-body-regex: "<item>"


actions:
  - xml:
      input-field: _raw
      remove: true
  - expand:
      mode:
        events:
          skip-list:
            - "^/rss/channel/item/\\d+/"
          exclude-non-empty-arrays: false

Deliver events over HTTP (http-post)

Use http-post (or http-get) when you need to push events to a downstream HTTP service (SaaS APIs, collectors, webhooks, SIEM/HEC endpoints).

Key fields:

  • url supports ${} runtime expansions for dynamic routing.
  • headers to set auth and content type.
  • body selects the event, a field, or a literal/template string.
  • batch and retry control throughput and resiliency.

Example: batch and send events as a JSON array

output:
  http-post:
    url: https://collector.example.com/events/acme-retail
    method: post
    headers:
      Content-Type: application/json
      Authorization: Bearer ${secret|collector/token}
    body:
      event: {}
    retry:
      timeout: 30s
      retries: 5
    batch:
      mode: fixed
      fixed-size: 100
      timeout: 250ms
      wrap-as-json: true

Troubleshooting and common gotchas

  • 401/403: confirm token source ({{ }} context vs ${ } runtime expansions), required scopes, and header names/prefixes.
  • 429/rate limiting: reduce schedule frequency, narrow the requested time window, and tune retries; avoid infinite pagination by keeping max-pages-per-run low until you confirm stop conditions.
  • Unexpected event splitting: set ignore-linebreaks: true when the body should stay intact; use document-mode: true when downstream batching relies on document boundaries.
  • Non-JSON payloads: set payload-mode: raw and parse later (for example xml for RSS/Atom). For JSON arrays, prefer events-field instead of post-processing with regexes.

For templating patterns, see Variable Expansion and Context Management.