Skip to content
LyftData Docs

HTTP Poll

HTTP Poll (http-poll)

Section titled “HTTP Poll (http-poll)”

Run HTTP queries.

Http binary json raw

Minimal example

Section titled “Minimal example”
input:
  http-poll:
    url: ""
JSON 
{
  "input": {
    "http-poll": {
      "url": ""
    }
  }
}

Contents

Section titled “Contents”

A ✓ marks a field that accepts a context variable such as {{ VARIABLE }}.

Authentication

Section titled “Authentication”
Authentication
Field Type Required Description
auth Auth HTTP authentication / signing strategy.
Allowed values: basic, header-token, facebook-app-secret, aws-sig-v4, oauth2-via-credential-manager

Behavior

Section titled “Behavior”
Behavior
Field Type Required Description
json ✓ boolean (bool) Use for when data is text or already JSON.
Default: false
payload-mode ✓ Payload Mode Select how the response body should be interpreted (auto, json, raw, binary).
Default: auto
Allowed values: auto, json, raw, binary
ignore-line-breaks ✓ boolean (bool) Do not treat separate lines as distinct events.
Default: false

Pagination

Section titled “Pagination”
Pagination
Field Type Required Description
pagination Pagination Optional pagination handler configuration.
Allowed values: cursor, link-header, query-param

Processing

Section titled “Processing”
Processing
Field Type Required Description
batch Batch Mark each group of input events in a distinct way.
events-field field (string) extract events from this field of the response, assumed to be an array. Body is assumed to be JSON and we ignore line breaks.
Examples: data_field
document-mode ✓ boolean (bool) collect all the output together as a single document (will affect batching on the output).
Default: false

Reliability

Section titled “Reliability”
Reliability
Field Type Required Description
retry Retry How to retry operation if it fails.
timeout time-interval (string) Timeout for the request (e.g. 500ms, 2s, etc. - the default is 30 seconds).
Examples: 500ms, 2h

Request

Section titled “Request”
Request
Field Type Required Description
url url (string) ✅ The URL for this request.
Examples: https://example.com/path
headers map (string) Headers to send with the query.
query map (string) Query parameters to send with the query.
body multiline-text (string) The payload to send.
form-urlencoded-body map (string) Payload as key-value pairs posted as form-urlencoded.
method Method HTTP method to use for the query - default is GET.
Default: get
Allowed values: get, post, put, patch, delete, options, head

Response

Section titled “Response”
Response
Field Type Required Description
response Response Fields to include in returned event.

Security

Section titled “Security”
Security
Field Type Required Description
insecure ✓ boolean (bool) Ignore TLS certificate validation errors (This is unsafe to use).
Default: false

Trigger

Section titled “Trigger”
Trigger
Field Type Required Description
trigger Trigger When and how to run this input. If omitted, runs exactly once.
Allowed values: message, cron, interval

Schema

Section titled “Schema”

Auth Options

Section titled “Auth Options”
Option Name Type Description
basic Basic object
header-token Header Token object
facebook-app-secret Facebook App Secret object
aws-sig-v4 Aws Sig V4 object
oauth2-via-credential-manager Oauth2 Via Credential Manager object

Pagination - Query Param - Checkpoint Options

Section titled “Pagination - Query Param - Checkpoint Options”
Option Name Type Description
runtime-artifacts Runtime Artifacts map Store pagination state in the runtime artefact store.
worker-kv Worker Kv object Store pagination state in Worker KV (explicit opt-in).

Pagination Options

Section titled “Pagination Options”
Option Name Type Description
cursor Cursor object
link-header Link Header object
query-param Query Param object

Trigger - Cron - Window - Start Options

Section titled “Trigger - Cron - Window - Start Options”
Option Name Type Description
start-time Start Time object
tracked Tracked string Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger - Interval - Window - Start Options

Section titled “Trigger - Interval - Window - Start Options”
Option Name Type Description
start-time Start Time object
tracked Tracked string Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger Options

Section titled “Trigger Options”
Option Name Type Description
message Message object
cron Cron object
interval Interval object

Auth - Basic Fields

Section titled “Auth - Basic Fields”
Field Type Required Description
username string ✅ Name of User.
password string ✅ Password.

Auth - Header Token Fields

Section titled “Auth - Header Token Fields”
Field Type Required Description
token ✓ string ✅ Token value (literal or context/secret/credential reference).
auth-header string Header used to carry the token.
Default: Authorization
prefix string Prefix prepended to the token (use an empty string to send the token as-is).
Default: Bearer

Auth - Facebook App Secret Fields

Section titled “Auth - Facebook App Secret Fields”
Field Type Required Description
access-token ✓ string ✅ Access token (usually a user or page token). Accepts literal values or context/secret references ({{ ctx.facebook.token }}).
app-secret ✓ string ✅ Facebook App Secret used to compute the HMAC proof (literal or context/secret reference).
proof-param string Query parameter name used for the proof.
Default: appsecret_proof
auth-header string Header used to carry the access token.
Default: Authorization

Auth - Aws Sig V4 Fields

Section titled “Auth - Aws Sig V4 Fields”
Field Type Required Description
access-key-id ✓ string ✅ AWS access key id (literal or context/secret/credential reference).
secret-access-key ✓ string ✅ AWS secret access key (literal or context/secret/credential reference).
session-token ✓ string Optional AWS session token for temporary credentials (literal or context reference).
region ✓ string ✅ AWS region used to compute the signature (e.g. us-east-1).
service ✓ string AWS service name (defaults to execute-api when omitted).

Auth - Oauth2 Via Credential Manager Fields

Section titled “Auth - Oauth2 Via Credential Manager Fields”
Field Type Required Description
credential-id ✓ credential-id (string) ✅ Credential manager id used to hydrate the OAuth2 token.
Examples: 2b64d2a0-0b76-4cd0-9ef0-3f0ef3d2a9a1
credential-scope-mode Credential Scope Mode ✅ How to choose the scopes bound to this job when using the credential.
Allowed values: allowed, custom
credential-scopes string[] Custom scopes to bind to this job when credential-scope-mode is custom.
token-field string Credential field name used as the token (e.g. access_token).
Default: access_token
auth-header string Header used to carry the token.
Default: Authorization
prefix string Prefix prepended to the token (use an empty string to send the token as-is).
Default: Bearer

Pagination - Cursor Fields

Section titled “Pagination - Cursor Fields”
Field Type Required Description
namespace ✓ string ✅ Worker KV namespace in tenant:namespace form used to persist cursors.
cursor-key ✓ string ✅ Key used to store the latest cursor for this poller.
cursor-query-param string Query parameter appended for cursor pagination.
Default: after
cursor-pointer json-pointer (string) JSON pointer to extract the cursor value.
Default: /paging/cursors/after
next-pointer json-pointer (string) JSON pointer to extract the next page URL.
Default: /paging/next
max-pages-per-run number Maximum number of pages fetched per run.
Default: 5

Pagination - Link Header Fields

Section titled “Pagination - Link Header Fields”
Field Type Required Description
header string Header to parse for pagination links.
Default: Link
rel string Link relation to follow (typically next).
Default: next
max-pages-per-run number Maximum number of pages fetched per run.
Default: 5

Pagination - Query Param - Checkpoint - Worker Kv Fields

Section titled “Pagination - Query Param - Checkpoint - Worker Kv Fields”
Field Type Required Description
namespace ✓ string ✅ Worker KV namespace in tenant:namespace form.
key ✓ string ✅ Key used to store pagination state.

Pagination - Query Param Fields

Section titled “Pagination - Query Param Fields”
Field Type Required Description
param string ✅ Query parameter name used for pagination.
start number First paged value to request after the base URL is fetched.
Default: 2
step number Increment applied after each successfully fetched page.
Default: 1
max-pages-per-run number Maximum number of pages fetched per run.
Default: 5
checkpoint Checkpoint Where to persist pagination checkpoints (defaults to runtime artefacts).
Allowed values: runtime-artifacts, worker-kv
continue-if-body-regex string Continue pagination only while this regex matches the response body.
terminal-status-codes number[] (integer) Treat these HTTP status codes as a clean end-of-pagination (no error event, no payload emit).
Examples: 42, 1.2e-10

Batch Fields

Section titled “Batch Fields”
Field Type Required Description
uuid-field field (string) Field where generated uuid, the unique marker for the group, will be stored.
Examples: data_field
invocation-time-field field (string) Field where invocation time will be stored.
Examples: data_field
completion-time-field field (string) Field where completion (end of execution) time will be stored.
Examples: data_field
begin-marker-field field (string) Field used to mark first event in the group.
Examples: data_field
end-marker-field field (string) Field used to mark last event in the group.
Examples: data_field
line-count-field field (string) Field used to store the line count of the batch.
Examples: data_field
line-num-field field (string) Field used to store the line number of the batch.
Examples: data_field

Retry - Backoff Fields

Section titled “Retry - Backoff Fields”
Field Type Required Description
strategy Strategy Backoff strategy to use (default exponential).
Allowed values: exponential, linear, fixed
base string Base delay before retrying (e.g. “200ms”).
max string Maximum delay between retries.
jitter ✓ boolean (bool) Whether to add jitter to retry delays.
Default: false

Retry Fields

Section titled “Retry Fields”
Field Type Required Description
max-attempts number (integer) Maximum attempts before giving up.
Examples: 42, 1.2e-10
forever ✓ boolean (bool) Retry indefinitely until cancelled.
Default: false
backoff Backoff Backoff strategy configuration.

Response Fields

Section titled “Response Fields”
Field Type Required Description
json ✓ boolean (bool) Return response in JSON format as single line-delimited event.
Default: true
like raw, but guarantees that the JSON is on one line
status-field field (string) Field to store HTTP status code.
Examples: data_field
headers-field field (string) Field to store response headers.
Examples: data_field
response-field field (string) Field to store the body of the response.
Default: _raw
Examples: data_field

Trigger - Message Fields

Section titled “Trigger - Message Fields”
Field Type Required Description
limit ✓ number (integer) The number of times to run the input.
Examples: 42, 1.2e-10
filter-kind Filter Kind Specifies whether the message originated from the “system” or by the “user”.
Allowed values: system, user, runtime-artifact-fetch, runtime-artifact-fetch-error, runtime-artifact-clear, runtime-artifact-clear-ack, runtime-artifact-fetch-reply, runtime-artifact-update, runtime-artifact-update-ack
filter-source Filter Source[] Specifies what process generated the message. Was it a “server”, “worker” or “job”?
Allowed values: job, worker, server
filter-worker string Specifies what worker to select.
filter-job string Specifies the name of the job that the message came from.
filter-type Filter Type[] Specifies that particular types of message ought to match.
Allowed values: worker-licensed, worker-unlicensed, variable, variable-deleted, begin-shutting-down-job, begin-shutting-down-server, begin-shutting-down-worker, broadcast-job-thread-state, broadcast-server-thread-state, broadcast-worker-thread-state, …
filter-tag string Specifies that messages matched ought to carry a tag with a particular value. This only matches against user-generated messages.

Trigger - Cron - Window - Start - Start Time Fields

Section titled “Trigger - Cron - Window - Start - Start Time Fields”
Field Type Required Description
start-time time-format (string) ✅ Allows the windowing to start at a specified time.
Hint: %Y-%m-%d %H:%M:%S%.3f %z
It should in the following format: 2019-07-10 18:45:00.000 +0200
highwatermark-file path (string) ✅ Specify file where timestamp would be stored in order to resume, for when Job has been restarted.
Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger - Cron - Window Fields

Section titled “Trigger - Cron - Window Fields”
Field Type Required Description
size duration (string) ✅ Window size.
offset duration (string) Window offset.
Default: 0s
start Start Specify file where timestamp would be stored in order to resume, for when Job has been restarted.
Allowed values: start-time, tracked

Trigger - Cron Fields

Section titled “Trigger - Cron Fields”
Field Type Required Description
cron cron-expression (string) ✅ The Cron pattern.
immediate ✓ boolean (bool) Run as soon as invoked, instead of waiting for the specified cron interval.
Default: false
random-offset duration (string) Sets a random offset to the schedule, then sticks to it.
Default: 0s
window Window Optional window definition when the schedule should only read a bounded range.

Trigger - Interval - Window - Start - Start Time Fields

Section titled “Trigger - Interval - Window - Start - Start Time Fields”
Field Type Required Description
start-time time-format (string) ✅ Allows the windowing to start at a specified time.
Hint: %Y-%m-%d %H:%M:%S%.3f %z
It should in the following format: 2019-07-10 18:45:00.000 +0200
highwatermark-file path (string) ✅ Specify file where timestamp would be stored in order to resume, for when Job has been restarted.
Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger - Interval - Window Fields

Section titled “Trigger - Interval - Window Fields”
Field Type Required Description
size duration (string) ✅ Window size.
offset duration (string) Window offset.
Default: 0s
start Start Specify file where timestamp would be stored in order to resume, for when Job has been restarted.
Allowed values: start-time, tracked

Trigger - Interval Fields

Section titled “Trigger - Interval Fields”
Field Type Required Description
duration duration (string) ✅ Duration to wait between events.
random-offset duration (string) Sets a random offset to the schedule, then sticks to it.
Default: 0s
window Window Optional window definition when the interval should only cover a bounded range.

Headers Table

Section titled “Headers Table”
Header Value
Header-Name value

Value format: templated-text.

Query Table

Section titled “Query Table”
Parameter Value
param value

Value format: templated-text.

Form Urlencoded Body Table

Section titled “Form Urlencoded Body Table”
Field Value
field value

Value format: templated-text.

Auth - Oauth2 Via Credential Manager - Credential Scope Mode Options

Section titled “Auth - Oauth2 Via Credential Manager - Credential Scope Mode Options”
Value Aliases Name Description
allowed allowed
custom custom

Payload Mode Options

Section titled “Payload Mode Options”
Value Aliases Name Description
auto auto
json json
raw raw
binary binary

Retry - Backoff - Strategy Options

Section titled “Retry - Backoff - Strategy Options”
Value Aliases Name Description
exponential exponential
linear linear
fixed fixed

Method Options

Section titled “Method Options”
Value Aliases Name Description
get get
post post
put put
patch patch
delete delete
options options
head head

Trigger - Message - Filter Kind Options

Section titled “Trigger - Message - Filter Kind Options”
Value Aliases Name Description
system system
user user
runtime-artifact-fetch runtime-artifact-fetch
runtime-artifact-fetch-error runtime-artifact-fetch-error
runtime-artifact-clear runtime-artifact-clear
runtime-artifact-clear-ack runtime-artifact-clear-ack
runtime-artifact-fetch-reply runtime-artifact-fetch-reply
runtime-artifact-update runtime-artifact-update
runtime-artifact-update-ack runtime-artifact-update-ack

Trigger - Message - Filter Source Options

Section titled “Trigger - Message - Filter Source Options”
Value Aliases Name Description
job job
worker worker
server server

Trigger - Message - Filter Type Options

Section titled “Trigger - Message - Filter Type Options”
Value Aliases Name Description
worker-licensed worker-licensed
worker-unlicensed worker-unlicensed
variable variable
variable-deleted variable-deleted
begin-shutting-down-job begin-shutting-down-job
begin-shutting-down-server begin-shutting-down-server
begin-shutting-down-worker begin-shutting-down-worker
broadcast-job-thread-state broadcast-job-thread-state
broadcast-server-thread-state broadcast-server-thread-state
broadcast-worker-thread-state broadcast-worker-thread-state
check-job-report-time check-job-report-time
check-worker-report-time check-worker-report-time
de-register-job-thread-dependency de-register-job-thread-dependency
de-register-server-thread-dependency de-register-server-thread-dependency
de-register-worker-thread-dependency de-register-worker-thread-dependency
deployed-job-active deployed-job-active
deployed-job-removed deployed-job-removed
deployed-job-should-be-running deployed-job-should-be-running
deployment-phase deployment-phase
durable-transport-effect-ack durable-transport-effect-ack
heart-beat heart-beat
initialise-internal-state initialise-internal-state
initialise-job-states initialise-job-states
job-batch-end job-batch-end
job-backlog-update job-backlog-update
job-analytics-batch job-analytics-batch
job-checkpoint-update job-checkpoint-update
job-deploy-ready job-deploy-ready
job-deployed job-deployed
job-document-end job-document-end
job-document-start job-document-start
job-errors job-errors
job-execution-anomaly job-execution-anomaly
job-execution-status job-execution-status
job-emit-custom job-emit-custom
job-finished job-finished
job-idle job-idle
job-initiated job-initiated
job-is-processing job-is-processing
job-logs job-logs
job-metrics job-metrics
job-notifications job-notifications
job-removing job-removing
job-removed job-removed
job-remove-failed job-remove-failed
job-remove-ready job-remove-ready
job-replaced job-replaced
job-required job-required
job-run-ended job-run-ended
job-runtime-error job-runtime-error
job-runtime-settings job-runtime-settings
job-run-started job-run-started
job-running-docker job-running-docker
job-running-script job-running-script
job-running-subprocess job-running-subprocess
job-running-system-d job-running-system-d
job-settings job-settings
job-step-statistics job-step-statistics
job-started job-started
job-staged job-staged
job-state-transition job-state-transition
job-shutting-down job-shutting-down
job-stopping job-stopping
job-stopped job-stopped
job-timed-out job-timed-out
job-suspicious-silence job-suspicious-silence
job-thread-state job-thread-state
job-trace job-trace
job-trace-requires-samples job-trace-requires-samples
job-updated job-updated
job-worker-comms-error job-worker-comms-error
job-unstaged job-unstaged
license-state-changed license-state-changed
license-validation-failed license-validation-failed
license-validation-ok license-validation-ok
license-volume-violation license-volume-violation
new-license new-license
override-job-coordinated-shutdown override-job-coordinated-shutdown
override-server-coordinated-shutdown override-server-coordinated-shutdown
override-worker-coordinated-shutdown override-worker-coordinated-shutdown
register-job-thread-dependency register-job-thread-dependency
register-server-thread-dependency register-server-thread-dependency
register-worker-thread-dependency register-worker-thread-dependency
run-job-failure run-job-failure
server-logs server-logs
server-metrics-batch server-metrics-batch
server-started server-started
server-starting server-starting
server-stopping server-stopping
server-thread-state server-thread-state
server-worker-comms-error server-worker-comms-error
shutdown-jobs shutdown-jobs
shutdown-worker shutdown-worker
system-shutdown system-shutdown
update-upstream-sync-for-job update-upstream-sync-for-job
update-upstream-sync-for-worker update-upstream-sync-for-worker
update-variable update-variable
user-alert user-alert
user-generated user-generated
trigger-response trigger-response
trigger-proxy-response trigger-proxy-response
workflow-step-terminal workflow-step-terminal
workflow-step-terminal-ack workflow-step-terminal-ack
user-notification user-notification
worker-command-for-job worker-command-for-job
worker-auth-lease-ack worker-auth-lease-ack
worker-connected worker-connected
worker-created worker-created
worker-debug-heart-beat worker-debug-heart-beat
worker-error worker-error
worker-first-seen worker-first-seen
worker-heart-beat worker-heart-beat
worker-logs worker-logs
worker-metrics-batch worker-metrics-batch
worker-offline worker-offline
worker-requests-auth-lease worker-requests-auth-lease
worker-server-comms-error worker-server-comms-error
worker-settings worker-settings
worker-shutdown worker-shutdown
worker-shutting-down worker-shutting-down
worker-started worker-started
worker-state-uuid worker-state-uuid
worker-stopping worker-stopping
worker-suspicious-silence worker-suspicious-silence
worker-system-information worker-system-information
worker-thread-state worker-thread-state
worker-updated worker-updated
worker-modified worker-modified
worker-removed worker-removed
context-changed context-changed
rerender-deployment rerender-deployment
job-killed job-killed
message-serviced message-serviced
failed-to-service-message failed-to-service-message
worker-wants-initial-settings worker-wants-initial-settings
worker-wants-initial-settings-reply worker-wants-initial-settings-reply
worker-wants-deployed-jobs worker-wants-deployed-jobs
worker-wants-deployed-jobs-reply worker-wants-deployed-jobs-reply
worker-wants-job-configuration worker-wants-job-configuration
worker-wants-job-configuration-reply worker-wants-job-configuration-reply
job-wants-dslir-key job-wants-dslir-key
job-wants-dslir-key-reply job-wants-dslir-key-reply
worker-wants-dslir-key worker-wants-dslir-key
worker-wants-dslir-key-reply worker-wants-dslir-key-reply
job-wants-variables job-wants-variables
job-wants-variables-reply job-wants-variables-reply
job-wants-credentials job-wants-credentials
job-wants-credentials-reply job-wants-credentials-reply
job-wants-credentials-error job-wants-credentials-error
job-wants-secret-variables-reply job-wants-secret-variables-reply
job-credentials-invalidated job-credentials-invalidated
aggregator-health aggregator-health
worker-aggregator-health worker-aggregator-health
worker-verification-token worker-verification-token
worker-requests-verification-token worker-requests-verification-token
runtime-artifact-update runtime-artifact-update
runtime-artifact-update-ack runtime-artifact-update-ack
runtime-artifact-clear runtime-artifact-clear
runtime-artifact-clear-ack runtime-artifact-clear-ack
runtime-artifact-fetch runtime-artifact-fetch
runtime-artifact-fetch-reply runtime-artifact-fetch-reply
runtime-artifact-fetch-error runtime-artifact-fetch-error