BigQuery

BigQuery (big-query)

Ingest data from Google BigQuery.

Data Warehouse Enterprise edition json

Minimal example

input:
  big-query:
    connection:
      project: ~

JSON

{
  "input": {
    "big-query": {
      "connection": {
        "project": null
      }
    }
  }
}

Contents

• Minimal example

• Authentication

• Connection

• Operation

• Parameters

• Query Job

• Read Session

• Reliability

• Schema

• Trigger

• Schema

A ✓ marks a field that accepts a context variable such as {{ VARIABLE }}.

Authentication

Authentication

Field	Type	Required	Description
credentials	Credentials		Authentication credentials. Allowed values: service-account, service-account-file, workload-identity, application-default

Connection

Connection

Field	Type	Required	Description
connection	Connection	✅	Connection details (project, dataset, table).

Operation

Operation

Field	Type	Required	Description
mode	Mode		Operation mode (query-job or legacy read-session). Default: query-job Allowed values: read-session, query-job

Parameters

Parameters

Field	Type	Required	Description
parameters	Parameters[]		Query parameters or filter bindings.

Query Job

Query Job

Field	Type	Required	Description
query-job	Query Job		Configuration for SQL query jobs.

Read Session

Read Session

Field	Type	Required	Description
read-session	Read Session		Legacy Storage Read API session configuration (deprecated). Note: this mode is not supported in 2.0.1; use mode: query-job + query_job.

Reliability

Reliability

Field	Type	Required	Description
retry	Retry		Retry configuration for transient failures.

Schema

Schema

Field	Type	Required	Description
schema	Schema		Schema handling and coercion settings.

Trigger

Trigger

Field	Type	Required	Description
trigger	Trigger		Run on the provided cadence; if omitted, this input runs exactly once. Allowed values: message, cron, interval

Schema

• Credentials Options
• Trigger - Cron - Window - Start Options
• Trigger - Interval - Window - Start Options
• Trigger Options
• Credentials - Service Account Fields
• Credentials - Service Account File Fields
• Credentials - Workload Identity Fields
• Connection - Impersonation Fields
• Connection Fields
• Parameters Fields
• Query Job Fields
• Read Session Fields
• Retry - Backoff Fields
• Retry Fields
• Schema - Overrides Fields
• Schema Fields
• Trigger - Message Fields
• Trigger - Cron - Window - Start - Start Time Fields
• Trigger - Cron - Window Fields
• Trigger - Cron Fields
• Trigger - Interval - Window - Start - Start Time Fields
• Trigger - Interval - Window Fields
• Trigger - Interval Fields
• Mode Options
• Query Job - Priority Options
• Retry - Backoff - Strategy Options
• Trigger - Message - Filter Kind Options
• Trigger - Message - Filter Source Options
• Trigger - Message - Filter Type Options

Credentials Options

Option	Name	Type	Description
service-account	Service Account	object
service-account-file	Service Account File	object
workload-identity	Workload Identity	object
application-default	Application Default	map	Supported in 2.0.1 via Application Default Credentials discovery.

Trigger - Cron - Window - Start Options

Option	Name	Type	Description
start-time	Start Time	object
tracked	Tracked	string	Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger - Interval - Window - Start Options

Option	Name	Type	Description
start-time	Start Time	object
tracked	Tracked	string	Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger Options

Option	Name	Type	Description
message	Message	object
cron	Cron	object
interval	Interval	object

Credentials - Service Account Fields

Field	Type	Required	Description
secret-id ✓	string	✅

Credentials - Service Account File Fields

Field	Type	Required	Description
path	string	✅	Not supported in 2.0.1. Use service-account or application-default.

Credentials - Workload Identity Fields

Field	Type	Required	Description
audience ✓	string	✅	Not supported in 2.0.1. Follow-on work needs a full external-account source definition.
service-account-email ✓	string		Not supported in 2.0.1. Follow-on work needs a full external-account source definition.

Connection - Impersonation Fields

Field	Type	Required	Description
service-account ✓	string	✅
delegates ✓	string[]

Connection Fields

Field	Type	Required	Description
project ✓	string	✅
dataset ✓	string
table ✓	string
billing-project ✓	string
location ✓	string
impersonation	Impersonation		Impersonate a service account (requires IAM Credentials API).

Parameters Fields

Field	Type	Required	Description
name	string	✅
value ✓	string
r-type	string

Query Job Fields

Field	Type	Required	Description
query	string	✅
use-legacy-sql ✓	boolean (bool)		Default: false
dry-run ✓	boolean (bool)		Default: false
destination-table	string
priority	Priority		Default: INTERACTIVE Allowed values: INTERACTIVE, BATCH

Read Session Fields

Field	Type	Required	Description
max-stream-count	number (integer)		Examples: 42, 1.2e-10
selected-fields	string[]
row-restriction	string
snapshot-time	string

Retry - Backoff Fields

Field	Type	Required	Description
strategy	Strategy		Backoff strategy to use (default exponential). Allowed values: exponential, linear, fixed
base	string		Base delay before retrying (e.g. “200ms”).
max	string		Maximum delay between retries.
jitter ✓	boolean (bool)		Whether to add jitter to retry delays. Default: false

Retry Fields

Field	Type	Required	Description
max-attempts	number (integer)		Maximum attempts before giving up. Examples: 42, 1.2e-10
forever ✓	boolean (bool)		Retry indefinitely until cancelled. Default: false
backoff	Backoff		Backoff strategy configuration.

Schema - Overrides Fields

Field	Type	Required	Description
field-path	string	✅
target-type	string

Schema Fields

Field	Type	Required	Description
auto-fetch ✓	boolean (bool)		Reserved for GA follow-on work; BigQuery schema settings are not supported in 2.0.1. Default: true
overrides	Overrides[]		Reserved for GA follow-on work; BigQuery schema settings are not supported in 2.0.1.

Trigger - Message Fields

Field	Type	Required	Description
limit ✓	number (integer)		The number of times to run the input. Examples: 42, 1.2e-10
filter-kind	Filter Kind		Specifies whether the message originated from the “system” or by the “user”. Allowed values: system, user, runtime-artifact-fetch, runtime-artifact-fetch-error, runtime-artifact-clear, runtime-artifact-clear-ack, runtime-artifact-fetch-reply, runtime-artifact-update, runtime-artifact-update-ack
filter-source	Filter Source[]		Specifies what process generated the message. Was it a “server”, “worker” or “job”? Allowed values: job, worker, server
filter-worker	string		Specifies what worker to select.
filter-job	string		Specifies the name of the job that the message came from.
filter-type	Filter Type[]		Specifies that particular types of message ought to match. Allowed values: worker-licensed, worker-unlicensed, variable, variable-deleted, begin-shutting-down-job, begin-shutting-down-server, begin-shutting-down-worker, broadcast-job-thread-state, broadcast-server-thread-state, broadcast-worker-thread-state, …
filter-tag	string		Specifies that messages matched ought to carry a tag with a particular value. This only matches against user-generated messages.

Trigger - Cron - Window - Start - Start Time Fields

Field	Type	Required	Description
start-time	time-format (string)	✅	Allows the windowing to start at a specified time. Hint: %Y-%m-%d %H:%M:%S%.3f %z It should in the following format: 2019-07-10 18:45:00.000 +0200
highwatermark-file	path (string)	✅	Specify file where timestamp would be stored in order to resume, for when Job has been restarted. Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger - Cron - Window Fields

Field	Type	Required	Description
size	duration (string)	✅	Window size.
offset	duration (string)		Window offset. Default: 0s
start	Start		Specify file where timestamp would be stored in order to resume, for when Job has been restarted. Allowed values: start-time, tracked

Trigger - Cron Fields

Field	Type	Required	Description
cron	cron-expression (string)	✅	The Cron pattern.
immediate ✓	boolean (bool)		Run as soon as invoked, instead of waiting for the specified cron interval. Default: false
random-offset	duration (string)		Sets a random offset to the schedule, then sticks to it. Default: 0s
window	Window		Optional window definition when the schedule should only read a bounded range.

Trigger - Interval - Window - Start - Start Time Fields

Field	Type	Required	Description
start-time	time-format (string)	✅	Allows the windowing to start at a specified time. Hint: %Y-%m-%d %H:%M:%S%.3f %z It should in the following format: 2019-07-10 18:45:00.000 +0200
highwatermark-file	path (string)	✅	Specify file where timestamp would be stored in order to resume, for when Job has been restarted. Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger - Interval - Window Fields

Field	Type	Required	Description
size	duration (string)	✅	Window size.
offset	duration (string)		Window offset. Default: 0s
start	Start		Specify file where timestamp would be stored in order to resume, for when Job has been restarted. Allowed values: start-time, tracked

Trigger - Interval Fields

Field	Type	Required	Description
duration	duration (string)	✅	Duration to wait between events.
random-offset	duration (string)		Sets a random offset to the schedule, then sticks to it. Default: 0s
window	Window		Optional window definition when the interval should only cover a bounded range.

Mode Options

Value	Aliases	Name	Description
read-session		read-session
query-job		query-job

Query Job - Priority Options

Value	Aliases	Name	Description
INTERACTIVE		INTERACTIVE
BATCH		BATCH

Retry - Backoff - Strategy Options

Value	Aliases	Name	Description
exponential		exponential
linear		linear
fixed		fixed

Trigger - Message - Filter Kind Options

Value	Aliases	Name	Description
system		system
user		user
runtime-artifact-fetch		runtime-artifact-fetch
runtime-artifact-fetch-error		runtime-artifact-fetch-error
runtime-artifact-clear		runtime-artifact-clear
runtime-artifact-clear-ack		runtime-artifact-clear-ack
runtime-artifact-fetch-reply		runtime-artifact-fetch-reply
runtime-artifact-update		runtime-artifact-update
runtime-artifact-update-ack		runtime-artifact-update-ack

Trigger - Message - Filter Source Options

Value	Aliases	Name	Description
job		job
worker		worker
server		server

Trigger - Message - Filter Type Options

Value	Aliases	Name	Description
worker-licensed		worker-licensed
worker-unlicensed		worker-unlicensed
variable		variable
variable-deleted		variable-deleted
begin-shutting-down-job		begin-shutting-down-job
begin-shutting-down-server		begin-shutting-down-server
begin-shutting-down-worker		begin-shutting-down-worker
broadcast-job-thread-state		broadcast-job-thread-state
broadcast-server-thread-state		broadcast-server-thread-state
broadcast-worker-thread-state		broadcast-worker-thread-state
check-job-report-time		check-job-report-time
check-worker-report-time		check-worker-report-time
de-register-job-thread-dependency		de-register-job-thread-dependency
de-register-server-thread-dependency		de-register-server-thread-dependency
de-register-worker-thread-dependency		de-register-worker-thread-dependency
deployed-job-active		deployed-job-active
deployed-job-removed		deployed-job-removed
deployed-job-should-be-running		deployed-job-should-be-running
deployment-phase		deployment-phase
durable-transport-effect-ack		durable-transport-effect-ack
heart-beat		heart-beat
initialise-internal-state		initialise-internal-state
initialise-job-states		initialise-job-states
job-batch-end		job-batch-end
job-backlog-update		job-backlog-update
job-analytics-batch		job-analytics-batch
job-checkpoint-update		job-checkpoint-update
job-deploy-ready		job-deploy-ready
job-deployed		job-deployed
job-document-end		job-document-end
job-document-start		job-document-start
job-errors		job-errors
job-execution-anomaly		job-execution-anomaly
job-execution-status		job-execution-status
job-emit-custom		job-emit-custom
job-finished		job-finished
job-idle		job-idle
job-initiated		job-initiated
job-is-processing		job-is-processing
job-logs		job-logs
job-metrics		job-metrics
job-notifications		job-notifications
job-removing		job-removing
job-removed		job-removed
job-remove-failed		job-remove-failed
job-remove-ready		job-remove-ready
job-replaced		job-replaced
job-required		job-required
job-run-ended		job-run-ended
job-runtime-error		job-runtime-error
job-runtime-settings		job-runtime-settings
job-run-started		job-run-started
job-running-docker		job-running-docker
job-running-script		job-running-script
job-running-subprocess		job-running-subprocess
job-running-system-d		job-running-system-d
job-settings		job-settings
job-step-statistics		job-step-statistics
job-started		job-started
job-staged		job-staged
job-state-transition		job-state-transition
job-shutting-down		job-shutting-down
job-stopping		job-stopping
job-stopped		job-stopped
job-timed-out		job-timed-out
job-suspicious-silence		job-suspicious-silence
job-thread-state		job-thread-state
job-trace		job-trace
job-trace-requires-samples		job-trace-requires-samples
job-updated		job-updated
job-worker-comms-error		job-worker-comms-error
job-unstaged		job-unstaged
license-state-changed		license-state-changed
license-validation-failed		license-validation-failed
license-validation-ok		license-validation-ok
license-volume-violation		license-volume-violation
new-license		new-license
override-job-coordinated-shutdown		override-job-coordinated-shutdown
override-server-coordinated-shutdown		override-server-coordinated-shutdown
override-worker-coordinated-shutdown		override-worker-coordinated-shutdown
register-job-thread-dependency		register-job-thread-dependency
register-server-thread-dependency		register-server-thread-dependency
register-worker-thread-dependency		register-worker-thread-dependency
run-job-failure		run-job-failure
server-logs		server-logs
server-metrics-batch		server-metrics-batch
server-started		server-started
server-starting		server-starting
server-stopping		server-stopping
server-thread-state		server-thread-state
server-worker-comms-error		server-worker-comms-error
shutdown-jobs		shutdown-jobs
shutdown-worker		shutdown-worker
system-shutdown		system-shutdown
update-upstream-sync-for-job		update-upstream-sync-for-job
update-upstream-sync-for-worker		update-upstream-sync-for-worker
update-variable		update-variable
user-alert		user-alert
user-generated		user-generated
trigger-response		trigger-response
trigger-proxy-response		trigger-proxy-response
workflow-step-terminal		workflow-step-terminal
workflow-step-terminal-ack		workflow-step-terminal-ack
user-notification		user-notification
worker-command-for-job		worker-command-for-job
worker-auth-lease-ack		worker-auth-lease-ack
worker-connected		worker-connected
worker-created		worker-created
worker-debug-heart-beat		worker-debug-heart-beat
worker-error		worker-error
worker-first-seen		worker-first-seen
worker-heart-beat		worker-heart-beat
worker-logs		worker-logs
worker-metrics-batch		worker-metrics-batch
worker-offline		worker-offline
worker-requests-auth-lease		worker-requests-auth-lease
worker-server-comms-error		worker-server-comms-error
worker-settings		worker-settings
worker-shutdown		worker-shutdown
worker-shutting-down		worker-shutting-down
worker-started		worker-started
worker-state-uuid		worker-state-uuid
worker-stopping		worker-stopping
worker-suspicious-silence		worker-suspicious-silence
worker-system-information		worker-system-information
worker-thread-state		worker-thread-state
worker-updated		worker-updated
worker-modified		worker-modified
worker-removed		worker-removed
context-changed		context-changed
rerender-deployment		rerender-deployment
job-killed		job-killed
message-serviced		message-serviced
failed-to-service-message		failed-to-service-message
worker-wants-initial-settings		worker-wants-initial-settings
worker-wants-initial-settings-reply		worker-wants-initial-settings-reply
worker-wants-deployed-jobs		worker-wants-deployed-jobs
worker-wants-deployed-jobs-reply		worker-wants-deployed-jobs-reply
worker-wants-job-configuration		worker-wants-job-configuration
worker-wants-job-configuration-reply		worker-wants-job-configuration-reply
job-wants-dslir-key		job-wants-dslir-key
job-wants-dslir-key-reply		job-wants-dslir-key-reply
worker-wants-dslir-key		worker-wants-dslir-key
worker-wants-dslir-key-reply		worker-wants-dslir-key-reply
job-wants-variables		job-wants-variables
job-wants-variables-reply		job-wants-variables-reply
job-wants-credentials		job-wants-credentials
job-wants-credentials-reply		job-wants-credentials-reply
job-wants-credentials-error		job-wants-credentials-error
job-wants-secret-variables-reply		job-wants-secret-variables-reply
job-credentials-invalidated		job-credentials-invalidated
aggregator-health		aggregator-health
worker-aggregator-health		worker-aggregator-health
worker-verification-token		worker-verification-token
worker-requests-verification-token		worker-requests-verification-token
runtime-artifact-update		runtime-artifact-update
runtime-artifact-update-ack		runtime-artifact-update-ack
runtime-artifact-clear		runtime-artifact-clear
runtime-artifact-clear-ack		runtime-artifact-clear-ack
runtime-artifact-fetch		runtime-artifact-fetch
runtime-artifact-fetch-reply		runtime-artifact-fetch-reply
runtime-artifact-fetch-error		runtime-artifact-fetch-error