Skip to content
LyftData Docs

ODBC SQL

ODBC SQL (sql-odbc)

Section titled “ODBC SQL (sql-odbc)”

Query relational databases over ODBC or stream change data capture events.

Database Enterprise edition json

Minimal example

Section titled “Minimal example”
input:
  sql-odbc:
    connection:
      url:
        url: ""
JSON 
{
  "input": {
    "sql-odbc": {
      "connection": {
        "url": {
          "url": ""
        }
      }
    }
  }
}

Contents

Section titled “Contents”

A ✓ marks a field that accepts a context variable such as {{ VARIABLE }}.

Connection

Section titled “Connection”
Connection
Field Type Required Description
connection Connection ✅ Connection details for the target database.
Allowed values: url, parts

Mode

Section titled “Mode”
Mode
Field Type Required Description
mode Mode Operating mode (one-shot queries or CDC polling).
Allowed values: query, cdc
cdc Cdc CDC configuration (required when mode = cdc).

Output

Section titled “Output”
Output
Field Type Required Description
stream-results ✓ boolean (bool) Stream rows to downstream consumers as they arrive (ODBC streams by default).
Default: false
auto-row-json ✓ boolean (bool) Automatically build JSON objects per row using column metadata.
Default: false
auto-type-mapping ✓ boolean (bool) Convert native SQL types to JSON (requires auto-row-json).
Default: false
column-overrides Column Overrides[] Per-column coercions when auto-row-json is enabled.

Processing

Section titled “Processing”
Processing
Field Type Required Description
batch Batch Optional batching semantics shared with other inputs.

Query

Section titled “Query”
Query
Field Type Required Description
query multiline-text (string) Inline SQL statement executed when query-file is not provided.
query-file path (string) Path to a SQL file executed when query is not provided.
Examples: /path/to/file, c:\users\joe\data\file.txt
parameters Parameters[] Bindings that map event fields into parameter placeholders.

Reliability

Section titled “Reliability”
Reliability
Field Type Required Description
retry Retry Retry policy for transient failures.

Trigger

Section titled “Trigger”
Trigger
Field Type Required Description
trigger Trigger Optional schedule for polling queries and CDC refresh.
Allowed values: message, cron, interval

Schema

Section titled “Schema”

Connection Options

Section titled “Connection Options”
Option Name Type Description
url Url object
parts Parts object

Trigger - Cron - Window - Start Options

Section titled “Trigger - Cron - Window - Start Options”
Option Name Type Description
start-time Start Time object
tracked Tracked string Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger - Interval - Window - Start Options

Section titled “Trigger - Interval - Window - Start Options”
Option Name Type Description
start-time Start Time object
tracked Tracked string Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger Options

Section titled “Trigger Options”
Option Name Type Description
message Message object
cron Cron object
interval Interval object

Connection - Url Fields

Section titled “Connection - Url Fields”
Field Type Required Description
url url (string) ✅ Database connection URL (sqlx/ODBC compatible).
Examples: https://example.com/path
max-connections number (integer) Maximum size of the async connection pool.
Examples: 42, 1.2e-10
min-connections number (integer) Minimum number of connections retained in the pool.
Examples: 42, 1.2e-10
connect-timeout string Optional connection timeout (humantime duration string).
idle-timeout string Optional idle timeout (humantime duration string).
schema string Optional schema override applied after connect.
options string[] Statements executed after connect (e.g. SET search_path TO …).
disable-tls-verification ✓ boolean (bool) Skip TLS verification (development only).
Default: false

Connection - Parts Fields

Section titled “Connection - Parts Fields”
Field Type Required Description
max-connections number (integer) Maximum size of the async connection pool.
Examples: 42, 1.2e-10
min-connections number (integer) Minimum number of connections retained in the pool.
Examples: 42, 1.2e-10
connect-timeout string Optional connection timeout (humantime duration string).
idle-timeout string Optional idle timeout (humantime duration string).
schema string Optional schema override applied after connect.
options string[] Statements executed after connect (e.g. SET search_path TO …).
disable-tls-verification ✓ boolean (bool) Skip TLS verification (development only).
Default: false
connection-string string ✅ ODBC connection string (translated to odbc:<connection-string>).

Cdc Fields

Section titled “Cdc Fields”
Field Type Required Description
cursor-column string Column used to order and resume polling.
resume-token-field field (string) JSON field containing the resume token in emitted rows.
Examples: data_field
poll-interval string Optional polling cadence override (humantime duration).
state-id string Explicit runtime-artifact state identifier override.
state-version number (integer) Optional state version used to rotate runtime artefacts on schema changes.
Examples: 42, 1.2e-10

Column Overrides Fields

Section titled “Column Overrides Fields”
Field Type Required Description
column string ✅
as-type As Type Allowed values: string, number, boolean, json, base64

Batch Fields

Section titled “Batch Fields”
Field Type Required Description
uuid-field field (string) Field where generated uuid, the unique marker for the group, will be stored.
Examples: data_field
invocation-time-field field (string) Field where invocation time will be stored.
Examples: data_field
completion-time-field field (string) Field where completion (end of execution) time will be stored.
Examples: data_field
begin-marker-field field (string) Field used to mark first event in the group.
Examples: data_field
end-marker-field field (string) Field used to mark last event in the group.
Examples: data_field
line-count-field field (string) Field used to store the line count of the batch.
Examples: data_field
line-num-field field (string) Field used to store the line number of the batch.
Examples: data_field

Parameters Fields

Section titled “Parameters Fields”
Field Type Required Description
name string ✅
field string ✅
default multiline-text (string)

Retry - Backoff Fields

Section titled “Retry - Backoff Fields”
Field Type Required Description
strategy Strategy Backoff strategy to use (default exponential).
Allowed values: exponential, linear, fixed
base string Base delay before retrying (e.g. “200ms”).
max string Maximum delay between retries.
jitter ✓ boolean (bool) Whether to add jitter to retry delays.
Default: false

Retry Fields

Section titled “Retry Fields”
Field Type Required Description
max-attempts number (integer) Maximum attempts before giving up.
Examples: 42, 1.2e-10
forever ✓ boolean (bool) Retry indefinitely until cancelled.
Default: false
backoff Backoff Backoff strategy configuration.

Trigger - Message Fields

Section titled “Trigger - Message Fields”
Field Type Required Description
limit ✓ number (integer) The number of times to run the input.
Examples: 42, 1.2e-10
filter-kind Filter Kind Specifies whether the message originated from the “system” or by the “user”.
Allowed values: system, user, runtime-artifact-fetch, runtime-artifact-fetch-error, runtime-artifact-clear, runtime-artifact-clear-ack, runtime-artifact-fetch-reply, runtime-artifact-update, runtime-artifact-update-ack
filter-source Filter Source[] Specifies what process generated the message. Was it a “server”, “worker” or “job”?
Allowed values: job, worker, server
filter-worker string Specifies what worker to select.
filter-job string Specifies the name of the job that the message came from.
filter-type Filter Type[] Specifies that particular types of message ought to match.
Allowed values: worker-licensed, worker-unlicensed, variable, variable-deleted, begin-shutting-down-job, begin-shutting-down-server, begin-shutting-down-worker, broadcast-job-thread-state, broadcast-server-thread-state, broadcast-worker-thread-state, …
filter-tag string Specifies that messages matched ought to carry a tag with a particular value. This only matches against user-generated messages.

Trigger - Cron - Window - Start - Start Time Fields

Section titled “Trigger - Cron - Window - Start - Start Time Fields”
Field Type Required Description
start-time time-format (string) ✅ Allows the windowing to start at a specified time.
Hint: %Y-%m-%d %H:%M:%S%.3f %z
It should in the following format: 2019-07-10 18:45:00.000 +0200
highwatermark-file path (string) ✅ Specify file where timestamp would be stored in order to resume, for when Job has been restarted.
Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger - Cron - Window Fields

Section titled “Trigger - Cron - Window Fields”
Field Type Required Description
size duration (string) ✅ Window size.
offset duration (string) Window offset.
Default: 0s
start Start Specify file where timestamp would be stored in order to resume, for when Job has been restarted.
Allowed values: start-time, tracked

Trigger - Cron Fields

Section titled “Trigger - Cron Fields”
Field Type Required Description
cron cron-expression (string) ✅ The Cron pattern.
immediate ✓ boolean (bool) Run as soon as invoked, instead of waiting for the specified cron interval.
Default: false
random-offset duration (string) Sets a random offset to the schedule, then sticks to it.
Default: 0s
window Window Optional window definition when the schedule should only read a bounded range.

Trigger - Interval - Window - Start - Start Time Fields

Section titled “Trigger - Interval - Window - Start - Start Time Fields”
Field Type Required Description
start-time time-format (string) ✅ Allows the windowing to start at a specified time.
Hint: %Y-%m-%d %H:%M:%S%.3f %z
It should in the following format: 2019-07-10 18:45:00.000 +0200
highwatermark-file path (string) ✅ Specify file where timestamp would be stored in order to resume, for when Job has been restarted.
Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger - Interval - Window Fields

Section titled “Trigger - Interval - Window Fields”
Field Type Required Description
size duration (string) ✅ Window size.
offset duration (string) Window offset.
Default: 0s
start Start Specify file where timestamp would be stored in order to resume, for when Job has been restarted.
Allowed values: start-time, tracked

Trigger - Interval Fields

Section titled “Trigger - Interval Fields”
Field Type Required Description
duration duration (string) ✅ Duration to wait between events.
random-offset duration (string) Sets a random offset to the schedule, then sticks to it.
Default: 0s
window Window Optional window definition when the interval should only cover a bounded range.

Mode Options

Section titled “Mode Options”
Value Aliases Name Description
query query
cdc cdc

Column Overrides - As Type Options

Section titled “Column Overrides - As Type Options”
Value Aliases Name Description
string string
number number
boolean boolean
json json
base64 base64

Retry - Backoff - Strategy Options

Section titled “Retry - Backoff - Strategy Options”
Value Aliases Name Description
exponential exponential
linear linear
fixed fixed

Trigger - Message - Filter Kind Options

Section titled “Trigger - Message - Filter Kind Options”
Value Aliases Name Description
system system
user user
runtime-artifact-fetch runtime-artifact-fetch
runtime-artifact-fetch-error runtime-artifact-fetch-error
runtime-artifact-clear runtime-artifact-clear
runtime-artifact-clear-ack runtime-artifact-clear-ack
runtime-artifact-fetch-reply runtime-artifact-fetch-reply
runtime-artifact-update runtime-artifact-update
runtime-artifact-update-ack runtime-artifact-update-ack

Trigger - Message - Filter Source Options

Section titled “Trigger - Message - Filter Source Options”
Value Aliases Name Description
job job
worker worker
server server

Trigger - Message - Filter Type Options

Section titled “Trigger - Message - Filter Type Options”
Value Aliases Name Description
worker-licensed worker-licensed
worker-unlicensed worker-unlicensed
variable variable
variable-deleted variable-deleted
begin-shutting-down-job begin-shutting-down-job
begin-shutting-down-server begin-shutting-down-server
begin-shutting-down-worker begin-shutting-down-worker
broadcast-job-thread-state broadcast-job-thread-state
broadcast-server-thread-state broadcast-server-thread-state
broadcast-worker-thread-state broadcast-worker-thread-state
check-job-report-time check-job-report-time
check-worker-report-time check-worker-report-time
de-register-job-thread-dependency de-register-job-thread-dependency
de-register-server-thread-dependency de-register-server-thread-dependency
de-register-worker-thread-dependency de-register-worker-thread-dependency
deployed-job-active deployed-job-active
deployed-job-removed deployed-job-removed
deployed-job-should-be-running deployed-job-should-be-running
deployment-phase deployment-phase
durable-transport-effect-ack durable-transport-effect-ack
heart-beat heart-beat
initialise-internal-state initialise-internal-state
initialise-job-states initialise-job-states
job-batch-end job-batch-end
job-backlog-update job-backlog-update
job-analytics-batch job-analytics-batch
job-checkpoint-update job-checkpoint-update
job-deploy-ready job-deploy-ready
job-deployed job-deployed
job-document-end job-document-end
job-document-start job-document-start
job-errors job-errors
job-execution-anomaly job-execution-anomaly
job-execution-status job-execution-status
job-emit-custom job-emit-custom
job-finished job-finished
job-idle job-idle
job-initiated job-initiated
job-is-processing job-is-processing
job-logs job-logs
job-metrics job-metrics
job-notifications job-notifications
job-removing job-removing
job-removed job-removed
job-remove-failed job-remove-failed
job-remove-ready job-remove-ready
job-replaced job-replaced
job-required job-required
job-run-ended job-run-ended
job-runtime-error job-runtime-error
job-runtime-settings job-runtime-settings
job-run-started job-run-started
job-running-docker job-running-docker
job-running-script job-running-script
job-running-subprocess job-running-subprocess
job-running-system-d job-running-system-d
job-settings job-settings
job-step-statistics job-step-statistics
job-started job-started
job-staged job-staged
job-state-transition job-state-transition
job-shutting-down job-shutting-down
job-stopping job-stopping
job-stopped job-stopped
job-timed-out job-timed-out
job-suspicious-silence job-suspicious-silence
job-thread-state job-thread-state
job-trace job-trace
job-trace-requires-samples job-trace-requires-samples
job-updated job-updated
job-worker-comms-error job-worker-comms-error
job-unstaged job-unstaged
license-state-changed license-state-changed
license-validation-failed license-validation-failed
license-validation-ok license-validation-ok
license-volume-violation license-volume-violation
new-license new-license
override-job-coordinated-shutdown override-job-coordinated-shutdown
override-server-coordinated-shutdown override-server-coordinated-shutdown
override-worker-coordinated-shutdown override-worker-coordinated-shutdown
register-job-thread-dependency register-job-thread-dependency
register-server-thread-dependency register-server-thread-dependency
register-worker-thread-dependency register-worker-thread-dependency
run-job-failure run-job-failure
server-logs server-logs
server-metrics-batch server-metrics-batch
server-started server-started
server-starting server-starting
server-stopping server-stopping
server-thread-state server-thread-state
server-worker-comms-error server-worker-comms-error
shutdown-jobs shutdown-jobs
shutdown-worker shutdown-worker
system-shutdown system-shutdown
update-upstream-sync-for-job update-upstream-sync-for-job
update-upstream-sync-for-worker update-upstream-sync-for-worker
update-variable update-variable
user-alert user-alert
user-generated user-generated
trigger-response trigger-response
trigger-proxy-response trigger-proxy-response
workflow-step-terminal workflow-step-terminal
workflow-step-terminal-ack workflow-step-terminal-ack
user-notification user-notification
worker-command-for-job worker-command-for-job
worker-auth-lease-ack worker-auth-lease-ack
worker-connected worker-connected
worker-created worker-created
worker-debug-heart-beat worker-debug-heart-beat
worker-error worker-error
worker-first-seen worker-first-seen
worker-heart-beat worker-heart-beat
worker-logs worker-logs
worker-metrics-batch worker-metrics-batch
worker-offline worker-offline
worker-requests-auth-lease worker-requests-auth-lease
worker-server-comms-error worker-server-comms-error
worker-settings worker-settings
worker-shutdown worker-shutdown
worker-shutting-down worker-shutting-down
worker-started worker-started
worker-state-uuid worker-state-uuid
worker-stopping worker-stopping
worker-suspicious-silence worker-suspicious-silence
worker-system-information worker-system-information
worker-thread-state worker-thread-state
worker-updated worker-updated
worker-modified worker-modified
worker-removed worker-removed
context-changed context-changed
rerender-deployment rerender-deployment
job-killed job-killed
message-serviced message-serviced
failed-to-service-message failed-to-service-message
worker-wants-initial-settings worker-wants-initial-settings
worker-wants-initial-settings-reply worker-wants-initial-settings-reply
worker-wants-deployed-jobs worker-wants-deployed-jobs
worker-wants-deployed-jobs-reply worker-wants-deployed-jobs-reply
worker-wants-job-configuration worker-wants-job-configuration
worker-wants-job-configuration-reply worker-wants-job-configuration-reply
job-wants-dslir-key job-wants-dslir-key
job-wants-dslir-key-reply job-wants-dslir-key-reply
worker-wants-dslir-key worker-wants-dslir-key
worker-wants-dslir-key-reply worker-wants-dslir-key-reply
job-wants-variables job-wants-variables
job-wants-variables-reply job-wants-variables-reply
job-wants-credentials job-wants-credentials
job-wants-credentials-reply job-wants-credentials-reply
job-wants-credentials-error job-wants-credentials-error
job-wants-secret-variables-reply job-wants-secret-variables-reply
job-credentials-invalidated job-credentials-invalidated
aggregator-health aggregator-health
worker-aggregator-health worker-aggregator-health
worker-verification-token worker-verification-token
worker-requests-verification-token worker-requests-verification-token
runtime-artifact-update runtime-artifact-update
runtime-artifact-update-ack runtime-artifact-update-ack
runtime-artifact-clear runtime-artifact-clear
runtime-artifact-clear-ack runtime-artifact-clear-ack
runtime-artifact-fetch runtime-artifact-fetch
runtime-artifact-fetch-reply runtime-artifact-fetch-reply
runtime-artifact-fetch-error runtime-artifact-fetch-error