Skip to content
LyftData Docs (Beta)

HTTP Poll

HTTP Poll (http-poll)

Run HTTP queries.

Http binary json raw

Minimal example

input:
  http-poll:
    url: ""
JSON 
{
  "input": {
    "http-poll": {
      "url": ""
    }
  }
}

Contents

Authentication

Authentication
FieldTypeRequiredDescription
authAuthHTTP authentication / signing strategy.
Allowed values: basic, header-token, facebook-app-secret, aws-sig-v4, oauth2-via-credential-manager

Behavior

Behavior
FieldTypeRequiredDescription
json ✓boolean (bool)Use for when data is text or already JSON.
Default: false
payload-mode ✓Payload ModeSelect how the response body should be interpreted (auto, json, raw, binary).
Default: auto
Allowed values: auto, json, raw, binary
ignore-line-breaks ✓boolean (bool)Do not treat separate lines as distinct events.
Default: false

Pagination

Pagination
FieldTypeRequiredDescription
paginationPaginationOptional pagination handler configuration.
Allowed values: cursor, link-header

Processing

Processing
FieldTypeRequiredDescription
batchBatchMark each group of input events in a distinct way.
events-fieldfield (string)extract events from this field of the response, assumed to be an array. Body is assumed to be JSON and we ignore line breaks.
Examples: data_field
document-mode ✓boolean (bool)collect all the output together as a single document (will affect batching on the output).
Default: false

Reliability

Reliability
FieldTypeRequiredDescription
retryRetryHow to retry operation if it fails.
timeouttime-interval (string)Timeout for the request (e.g. 500ms, 2s, etc. - the default is 30 seconds).
Examples: 500ms, 2h

Request

Request
FieldTypeRequiredDescription
urlurl (string)✅The URL for this request.
Examples: https://example.com/path
headersmap (string)Headers to send with the query.
querymap (string)Query parameters to send with the query.
bodymultiline-text (string)The payload to send.
form-urlencoded-bodymap (string)Payload as key-value pairs posted as form-urlencoded.
methodMethodHTTP method to use for the query - default is GET.
Default: get
Allowed values: get, post, put, patch, delete, options, head

Response

Response
FieldTypeRequiredDescription
responseHTTP Response MappingFields to include in returned event.

Security

Security
FieldTypeRequiredDescription
insecure ✓boolean (bool)Ignore TLS certificate validation errors (This is unsafe to use).
Default: false

Trigger

Trigger
FieldTypeRequiredDescription
triggerTriggerWhen and how to run this input. If omitted, runs exactly once.
Allowed values: message, cron, interval

Schema

Auth Options

OptionNameTypeDescription
basicBasicobject
header-tokenHeader Tokenobject
facebook-app-secretFacebook App Secretobject
aws-sig-v4Aws Sig V4object
oauth2-via-credential-managerOauth2 Via Credential Managerobject

Pagination Options

OptionNameTypeDescription
cursorCursorobject
link-headerLink Headerobject

Trigger - Cron - Window - Window Start Options

OptionNameTypeDescription
start-timeStart Timeobject
trackedTrackedstringExamples: /path/to/file, c:\users\joe\data\file.txt

Trigger - Interval - Window - Window Start Options

OptionNameTypeDescription
start-timeStart Timeobject
trackedTrackedstringExamples: /path/to/file, c:\users\joe\data\file.txt

Trigger Options

OptionNameTypeDescription
messageMessageobject
cronCronobject
intervalIntervalobject

Auth - Basic Fields

FieldTypeRequiredDescription
usernamestring✅Name of User.
passwordstring✅Password.

Auth - Header Token Fields

FieldTypeRequiredDescription
token ✓string✅Token value (literal or context/secret/credential reference).
auth-headerstringHeader used to carry the token.
Default: Authorization
prefixstringPrefix prepended to the token (use an empty string to send the token as-is).
Default: Bearer

Auth - Facebook App Secret Fields

FieldTypeRequiredDescription
access-token ✓string✅Access token (usually a user or page token). Accepts literal values or context/secret references ({{ ctx.facebook.token }}).
app-secret ✓string✅Facebook App Secret used to compute the HMAC proof (literal or context/secret reference).
proof-paramstringQuery parameter name used for the proof.
Default: appsecret_proof
auth-headerstringHeader used to carry the access token.
Default: Authorization

Auth - Aws Sig V4 Fields

FieldTypeRequiredDescription
access-key-id ✓string✅AWS access key id (literal or context/secret/credential reference).
secret-access-key ✓string✅AWS secret access key (literal or context/secret/credential reference).
session-token ✓stringOptional AWS session token for temporary credentials (literal or context reference).
region ✓string✅AWS region used to compute the signature (e.g. us-east-1).
service ✓stringAWS service name (defaults to execute-api when omitted).

Auth - Oauth2 Via Credential Manager Fields

FieldTypeRequiredDescription
credential-id ✓credential-id (string)✅Credential manager id used to hydrate the OAuth2 token.
Examples: 2b64d2a0-0b76-4cd0-9ef0-3f0ef3d2a9a1
credential-scope-modeCredential scope mode✅How to choose the scopes bound to this job when using the credential.
Allowed values: allowed, custom
credential-scopesstring[]Custom scopes to bind to this job when credential-scope-mode is custom.
token-fieldstringCredential field name used as the token (e.g. access_token).
Default: access_token
auth-headerstringHeader used to carry the token.
Default: Authorization
prefixstringPrefix prepended to the token (use an empty string to send the token as-is).
Default: Bearer

Pagination - Cursor Fields

FieldTypeRequiredDescription
namespace ✓string✅Worker KV namespace in tenant:namespace form used to persist cursors.
cursor-key ✓string✅Key used to store the latest cursor for this poller.
cursor-query-paramstringQuery parameter appended for cursor pagination.
Default: after
cursor-pointerjson-pointer (string)JSON pointer to extract the cursor value.
Default: /paging/cursors/after
next-pointerjson-pointer (string)JSON pointer to extract the next page URL.
Default: /paging/next
max-pages-per-runnumberMaximum number of pages fetched per run.
Default: 5

Pagination - Link Header Fields

FieldTypeRequiredDescription
headerstringHeader to parse for pagination links.
Default: Link
relstringLink relation to follow (typically next).
Default: next
max-pages-per-runnumberMaximum number of pages fetched per run.
Default: 5

Batch Fields

FieldTypeRequiredDescription
uuid-fieldfield (string)Field where generated uuid, the unique marker for the group, will be stored.
Examples: data_field
invocation-time-fieldfield (string)Field where invocation time will be stored.
Examples: data_field
completion-time-fieldfield (string)Field where completion (end of execution) time will be stored.
Examples: data_field
begin-marker-fieldfield (string)Field used to mark first event in the group.
Examples: data_field
end-marker-fieldfield (string)Field used to mark last event in the group.
Examples: data_field
line-count-fieldfield (string)Field used to store the line count of the batch.
Examples: data_field
line-num-fieldfield (string)Field used to store the line number of the batch.
Examples: data_field

Retry Fields

FieldTypeRequiredDescription
timeouttime-interval (string)✅timeout (e.g. 500ms, 2s etc. - default is 30).
Examples: 500ms, 2h
retriesnumber (integer)number of retries.
Examples: 42, 1.2e-10

HTTP Response Mapping Fields

FieldTypeRequiredDescription
json ✓boolean (bool)Return response in JSON format as single line-delimited event.
Default: true
status-fieldfield (string)Field to store HTTP status code.
Examples: data_field
headers-fieldfield (string)Field to store response headers.
Examples: data_field
response-fieldfield (string)Field to store the body of the response.
Default: _raw
Examples: data_field

Trigger - Message Fields

FieldTypeRequiredDescription
limit ✓number (integer)The number of times to run the input.
Examples: 42, 1.2e-10
filter-kindFilter KindSpecifies whether the message originated from the “system” or by the “user”.
Allowed values: system, user, runtime-artifact-fetch, runtime-artifact-fetch-error, runtime-artifact-clear, runtime-artifact-clear-ack, runtime-artifact-fetch-reply, runtime-artifact-update, runtime-artifact-update-ack
filter-sourceFilter Source[]Specifies what process generated the message. Was it a “server”, “worker” or “job”?
Allowed values: job, worker, server
filter-workerstringSpecifies what worker to select.
filter-jobstringSpecifies the name of the job that the message came from.
filter-typeFilter Type[]Specifies that particular types of message ought to match.
Allowed values: worker-licensed, worker-unlicensed, variable, variable-deleted, begin-shutting-down-job, begin-shutting-down-server, begin-shutting-down-worker, broadcast-job-thread-state, broadcast-server-thread-state, broadcast-worker-thread-state, …
filter-tagstringSpecifies that messages matched ought to carry a tag with a particular value. This only matches against user-generated messages.

Trigger - Cron - Window - Window Start - Start Time Fields

FieldTypeRequiredDescription
start-timetime-format (string)✅Allows the windowing to start at a specified time.
Hint: %Y-%m-%d %H:%M:%S%.3f %z
highwatermark-filepath (string)✅Specify file where timestamp would be stored in order to resume, for when Job has been restarted.
Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger - Cron - Window Fields

FieldTypeRequiredDescription
sizeduration (string)✅Window size.
offsetduration (string)Window offset.
Default: 0s
startWindow StartSpecify file where timestamp would be stored in order to resume, for when Job has been restarted.
Allowed values: start-time, tracked

Trigger - Cron Fields

FieldTypeRequiredDescription
croncron-expression (string)✅The Cron pattern.
immediate ✓boolean (bool)Run as soon as invoked, instead of waiting for the specified cron interval.
Default: false
random-offsetduration (string)Sets a random offset to the schedule, then sticks to it.
Default: 0s
windowWindowOptional window definition when the schedule should only read a bounded range.

Trigger - Interval - Window - Window Start - Start Time Fields

FieldTypeRequiredDescription
start-timetime-format (string)✅Allows the windowing to start at a specified time.
Hint: %Y-%m-%d %H:%M:%S%.3f %z
highwatermark-filepath (string)✅Specify file where timestamp would be stored in order to resume, for when Job has been restarted.
Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger - Interval - Window Fields

FieldTypeRequiredDescription
sizeduration (string)✅Window size.
offsetduration (string)Window offset.
Default: 0s
startWindow StartSpecify file where timestamp would be stored in order to resume, for when Job has been restarted.
Allowed values: start-time, tracked

Trigger - Interval Fields

FieldTypeRequiredDescription
durationduration (string)✅Duration to wait between events.
random-offsetduration (string)Sets a random offset to the schedule, then sticks to it.
Default: 0s
windowWindowOptional window definition when the interval should only cover a bounded range.

Headers Table

HeaderValue
Header-Namevalue

Value format: templated-text.

Query Table

ParameterValue
paramvalue

Value format: templated-text.

Form Urlencoded Body Table

FieldValue
fieldvalue

Value format: templated-text.

Auth - Oauth2 Via Credential Manager - Credential scope mode Options

ValueNameDescription
allowedallowedAllowed
customcustomCustom

Payload Mode Options

ValueNameDescription
autoautoAuto
jsonjsonJson
rawrawRaw
binarybinaryBinary

Method Options

ValueNameDescription
getgetGet
postpostPost
putputPut
patchpatchPatch
deletedeleteDelete
optionsoptionsOptions
headheadHead

Trigger - Message - Filter Kind Options

ValueNameDescription
systemsystemSystem
useruserUser
runtime-artifact-fetchruntime-artifact-fetchRuntime Artifact Fetch
runtime-artifact-fetch-errorruntime-artifact-fetch-errorRuntime Artifact Fetch Error
runtime-artifact-clearruntime-artifact-clearRuntime Artifact Clear
runtime-artifact-clear-ackruntime-artifact-clear-ackRuntime Artifact Clear Ack
runtime-artifact-fetch-replyruntime-artifact-fetch-replyRuntime Artifact Fetch Reply
runtime-artifact-updateruntime-artifact-updateRuntime Artifact Update
runtime-artifact-update-ackruntime-artifact-update-ackRuntime Artifact Update Ack

Trigger - Message - Filter Source Options

ValueNameDescription
jobjobJob
workerworkerWorker
serverserverServer

Trigger - Message - Filter Type Options

ValueNameDescription
worker-licensedworker-licensedWorker Licensed
worker-unlicensedworker-unlicensedWorker Unlicensed
variablevariableVariable
variable-deletedvariable-deletedVariable Deleted
begin-shutting-down-jobbegin-shutting-down-jobBegin Shutting Down Job
begin-shutting-down-serverbegin-shutting-down-serverBegin Shutting Down Server
begin-shutting-down-workerbegin-shutting-down-workerBegin Shutting Down Worker
broadcast-job-thread-statebroadcast-job-thread-stateBroadcast Job Thread State
broadcast-server-thread-statebroadcast-server-thread-stateBroadcast Server Thread State
broadcast-worker-thread-statebroadcast-worker-thread-stateBroadcast Worker Thread State
check-job-report-timecheck-job-report-timeCheck Job Report Time
check-worker-report-timecheck-worker-report-timeCheck Worker Report Time
de-register-job-thread-dependencyde-register-job-thread-dependencyDe Register Job Thread Dependency
de-register-server-thread-dependencyde-register-server-thread-dependencyDe Register Server Thread Dependency
de-register-worker-thread-dependencyde-register-worker-thread-dependencyDe Register Worker Thread Dependency
deployed-job-activedeployed-job-activeDeployed Job Active
deployed-job-removeddeployed-job-removedDeployed Job Removed
deployed-job-should-be-runningdeployed-job-should-be-runningDeployed Job Should Be Running
deployment-phasedeployment-phaseDeployment Phase
heart-beatheart-beatHeart Beat
initialise-internal-stateinitialise-internal-stateInitialise Internal State
initialise-job-statesinitialise-job-statesInitialise Job States
job-batch-endjob-batch-endJob Batch End
job-backlog-updatejob-backlog-updateJob Backlog Update
job-checkpoint-updatejob-checkpoint-updateJob Checkpoint Update
job-deploy-readyjob-deploy-readyJob Deploy Ready
job-deployedjob-deployedJob Deployed
job-document-endjob-document-endJob Document End
job-document-startjob-document-startJob Document Start
job-errorsjob-errorsJob Errors
job-execution-anomalyjob-execution-anomalyJob Execution Anomaly
job-execution-statusjob-execution-statusJob Execution Status
job-emit-customjob-emit-customJob Emit Custom
job-finishedjob-finishedJob Finished
job-idlejob-idleJob Idle
job-initiatedjob-initiatedJob Initiated
job-is-processingjob-is-processingJob Is Processing
job-logsjob-logsJob Logs
job-metricsjob-metricsJob Metrics
job-notificationsjob-notificationsJob Notifications
job-removingjob-removingJob Removing
job-removedjob-removedJob Removed
job-remove-failedjob-remove-failedJob Remove Failed
job-remove-readyjob-remove-readyJob Remove Ready
job-replacedjob-replacedJob Replaced
job-requiredjob-requiredJob Required
job-run-endedjob-run-endedJob Run Ended
job-runtime-errorjob-runtime-errorJob Runtime Error
job-runtime-settingsjob-runtime-settingsJob Runtime Settings
job-run-startedjob-run-startedJob Run Started
job-running-dockerjob-running-dockerJob Running Docker
job-running-scriptjob-running-scriptJob Running Script
job-running-subprocessjob-running-subprocessJob Running Subprocess
job-running-system-djob-running-system-dJob Running System D
job-settingsjob-settingsJob Settings
job-step-statisticsjob-step-statisticsJob Step Statistics
job-startedjob-startedJob Started
job-stagedjob-stagedJob Staged
job-state-transitionjob-state-transitionJob State Transition
job-shutting-downjob-shutting-downJob Shutting Down
job-stoppingjob-stoppingJob Stopping
job-stoppedjob-stoppedJob Stopped
job-timed-outjob-timed-outJob Timed Out
job-suspicious-silencejob-suspicious-silenceJob Suspicious Silence
job-thread-statejob-thread-stateJob Thread State
job-tracejob-traceJob Trace
job-trace-requires-samplesjob-trace-requires-samplesJob Trace Requires Samples
job-updatedjob-updatedJob Updated
job-worker-comms-errorjob-worker-comms-errorJob Worker Comms Error
job-unstagedjob-unstagedJob Unstaged
license-state-changedlicense-state-changedLicense State Changed
license-validation-failedlicense-validation-failedLicense Validation Failed
license-validation-oklicense-validation-okLicense Validation Ok
license-volume-violationlicense-volume-violationLicense Volume Violation
new-licensenew-licenseNew License
override-job-coordinated-shutdownoverride-job-coordinated-shutdownOverride Job Coordinated Shutdown
override-server-coordinated-shutdownoverride-server-coordinated-shutdownOverride Server Coordinated Shutdown
override-worker-coordinated-shutdownoverride-worker-coordinated-shutdownOverride Worker Coordinated Shutdown
register-job-thread-dependencyregister-job-thread-dependencyRegister Job Thread Dependency
register-server-thread-dependencyregister-server-thread-dependencyRegister Server Thread Dependency
register-worker-thread-dependencyregister-worker-thread-dependencyRegister Worker Thread Dependency
run-job-failurerun-job-failureRun Job Failure
server-logsserver-logsServer Logs
server-metrics-batchserver-metrics-batchServer Metrics Batch
server-startedserver-startedServer Started
server-startingserver-startingServer Starting
server-stoppingserver-stoppingServer Stopping
server-thread-stateserver-thread-stateServer Thread State
server-worker-comms-errorserver-worker-comms-errorServer Worker Comms Error
shutdown-jobsshutdown-jobsShutdown Jobs
shutdown-workershutdown-workerShutdown Worker
system-shutdownsystem-shutdownSystem Shutdown
update-upstream-sync-for-jobupdate-upstream-sync-for-jobUpdate Upstream Sync For Job
update-upstream-sync-for-workerupdate-upstream-sync-for-workerUpdate Upstream Sync For Worker
update-variableupdate-variableUpdate Variable
user-alertuser-alertUser Alert
user-generateduser-generatedUser Generated
user-notificationuser-notificationUser Notification
worker-command-for-jobworker-command-for-jobWorker Command For Job
worker-auth-lease-ackworker-auth-lease-ackWorker Auth Lease Ack
worker-connectedworker-connectedWorker Connected
worker-createdworker-createdWorker Created
worker-debug-heart-beatworker-debug-heart-beatWorker Debug Heart Beat
worker-errorworker-errorWorker Error
worker-first-seenworker-first-seenWorker First Seen
worker-heart-beatworker-heart-beatWorker Heart Beat
worker-logsworker-logsWorker Logs
worker-metrics-batchworker-metrics-batchWorker Metrics Batch
worker-offlineworker-offlineWorker Offline
worker-requests-auth-leaseworker-requests-auth-leaseWorker Requests Auth Lease
worker-server-comms-errorworker-server-comms-errorWorker Server Comms Error
worker-settingsworker-settingsWorker Settings
worker-shutdownworker-shutdownWorker Shutdown
worker-shutting-downworker-shutting-downWorker Shutting Down
worker-startedworker-startedWorker Started
worker-state-uuidworker-state-uuidWorker State Uuid
worker-stoppingworker-stoppingWorker Stopping
worker-suspicious-silenceworker-suspicious-silenceWorker Suspicious Silence
worker-system-informationworker-system-informationWorker System Information
worker-thread-stateworker-thread-stateWorker Thread State
worker-updatedworker-updatedWorker Updated
worker-modifiedworker-modifiedWorker Modified
worker-removedworker-removedWorker Removed
context-changedcontext-changedContext Changed
rerender-deploymentrerender-deploymentRerender Deployment
job-killedjob-killedJob Killed
message-servicedmessage-servicedMessage Serviced
failed-to-service-messagefailed-to-service-messageFailed To Service Message
worker-wants-initial-settingsworker-wants-initial-settingsWorker Wants Initial Settings
worker-wants-initial-settings-replyworker-wants-initial-settings-replyWorker Wants Initial Settings Reply
worker-wants-deployed-jobsworker-wants-deployed-jobsWorker Wants Deployed Jobs
worker-wants-deployed-jobs-replyworker-wants-deployed-jobs-replyWorker Wants Deployed Jobs Reply
worker-wants-job-configurationworker-wants-job-configurationWorker Wants Job Configuration
worker-wants-job-configuration-replyworker-wants-job-configuration-replyWorker Wants Job Configuration Reply
job-wants-dslir-keyjob-wants-dslir-keyJob Wants Dslir Key
job-wants-dslir-key-replyjob-wants-dslir-key-replyJob Wants Dslir Key Reply
worker-wants-dslir-keyworker-wants-dslir-keyWorker Wants Dslir Key
worker-wants-dslir-key-replyworker-wants-dslir-key-replyWorker Wants Dslir Key Reply
job-wants-variablesjob-wants-variablesJob Wants Variables
job-wants-variables-replyjob-wants-variables-replyJob Wants Variables Reply
job-wants-credentialsjob-wants-credentialsJob Wants Credentials
job-wants-credentials-replyjob-wants-credentials-replyJob Wants Credentials Reply
job-wants-credentials-errorjob-wants-credentials-errorJob Wants Credentials Error
job-wants-secret-variables-replyjob-wants-secret-variables-replyJob Wants Secret Variables Reply
job-credentials-invalidatedjob-credentials-invalidatedJob Credentials Invalidated
aggregator-healthaggregator-healthAggregator Health
worker-aggregator-healthworker-aggregator-healthWorker Aggregator Health
worker-verification-tokenworker-verification-tokenWorker Verification Token
worker-requests-verification-tokenworker-requests-verification-tokenWorker Requests Verification Token
runtime-artifact-updateruntime-artifact-updateRuntime Artifact Update
runtime-artifact-update-ackruntime-artifact-update-ackRuntime Artifact Update Ack
runtime-artifact-clearruntime-artifact-clearRuntime Artifact Clear
runtime-artifact-clear-ackruntime-artifact-clear-ackRuntime Artifact Clear Ack
runtime-artifact-fetchruntime-artifact-fetchRuntime Artifact Fetch
runtime-artifact-fetch-replyruntime-artifact-fetch-replyRuntime Artifact Fetch Reply
runtime-artifact-fetch-errorruntime-artifact-fetch-errorRuntime Artifact Fetch Error