BigQuery

BigQuery (big-query)

Stream data into Google BigQuery.

Data Warehouse Enterprise edition json

Minimal example

output:
  big-query:
    connection:
      project: ~

JSON

{
  "output": {
    "big-query": {
      "connection": {
        "project": null
      }
    }
  }
}

Contents

• Minimal example

• Authentication

• Connection

• Load Job

• Operation

• Parameters

• Processing

• Reliability

• Schema

• Streaming

• Schema

A ✓ marks a field that accepts a context variable such as {{ VARIABLE }}.

Authentication

Authentication

Field	Type	Required	Description
credentials	Credentials		Authentication credentials. Allowed values: service-account, service-account-file, workload-identity, application-default

Connection

Connection

Field	Type	Required	Description
connection	Connection	✅	Connection details (project, dataset, table).

Load Job

Load Job

Field	Type	Required	Description
load-job	Load Job		Legacy Load Job configuration (deprecated). Note: this mode is not supported in 2.0.1; use mode: streaming.

Operation

Operation

Field	Type	Required	Description
mode	Mode		Output mode (streaming or legacy load-job). Default: streaming Allowed values: streaming, load-job

Parameters

Parameters

Field	Type	Required	Description
parameters	Parameters[]		Reserved for GA follow-on work; not supported in 2.0.1.

Processing

Processing

Field	Type	Required	Description
batch	Batch		Batching configuration (size/timeout).

Reliability

Reliability

Field	Type	Required	Description
retry	Retry		Retry configuration for transient failures.

Schema

Schema

Field	Type	Required	Description
schema	Schema		Reserved for GA follow-on work; not supported in 2.0.1.

Streaming

Streaming

Field	Type	Required	Description
streaming	Streaming		Configuration for the current streaming insert path (tabledata.insertAll).

Schema

• Credentials Options
• Credentials - Service Account Fields
• Credentials - Service Account File Fields
• Credentials - Workload Identity Fields
• Connection - Impersonation Fields
• Connection Fields
• Load Job Fields
• Parameters Fields
• Batch Fields
• Retry - Backoff Fields
• Retry Fields
• Schema - Overrides Fields
• Schema Fields
• Streaming Fields
• Load Job - Write Disposition Options
• Load Job - Create Disposition Options
• Mode Options
• Batch - Mode Options
• Retry - Backoff - Strategy Options
• Streaming - Stream Type Options

Credentials Options

Option	Name	Type	Description
service-account	Service Account	object
service-account-file	Service Account File	object
workload-identity	Workload Identity	object
application-default	Application Default	map	Supported in 2.0.1 via Application Default Credentials discovery.

Credentials - Service Account Fields

Field	Type	Required	Description
secret-id ✓	string	✅

Credentials - Service Account File Fields

Field	Type	Required	Description
path	string	✅	Not supported in 2.0.1. Use service-account or application-default.

Credentials - Workload Identity Fields

Field	Type	Required	Description
audience ✓	string	✅	Not supported in 2.0.1. Follow-on work needs a full external-account source definition.
service-account-email ✓	string		Not supported in 2.0.1. Follow-on work needs a full external-account source definition.

Connection - Impersonation Fields

Field	Type	Required	Description
service-account ✓	string	✅
delegates ✓	string[]

Connection Fields

Field	Type	Required	Description
project ✓	string	✅
dataset ✓	string
table ✓	string
billing-project ✓	string
location ✓	string
impersonation	Impersonation		Impersonate a service account (requires IAM Credentials API).

Load Job Fields

Field	Type	Required	Description
staging-uri ✓	string	✅	GCS URI for staging files (gs://bucket/path).
write-disposition	Write Disposition		Default: WRITE_APPEND Allowed values: WRITE_APPEND, WRITE_TRUNCATE, WRITE_EMPTY
create-disposition	Create Disposition		Default: CREATE_IF_NEEDED Allowed values: CREATE_IF_NEEDED, CREATE_NEVER

Parameters Fields

Field	Type	Required	Description
name	string	✅
value ✓	string
r-type	string

Batch Fields

Field	Type	Required	Description
fixed-size ✓	number (integer)		maximum number of events in an output batch. Examples: 42, 1.2e-10
max-bytes ✓	number (integer)		Close the batch before adding an event that would make the serialized request payload exceed this many bytes. Examples: 42, 1.2e-10
mode	Mode	✅	If ‘document’ send on end of document generated by input. If ‘fixed’, use fixed_size. Allowed values: fixed, document
timeout	time-interval (string)	✅	interval after which the batch is sent, to keep throughput going (default 100ms). Default: 100ms Examples: 500ms, 2h
header	multiline-text (string)		put a header line before the batch.
footer	multiline-text (string)		put a header line after the last line of the batch.
use-document-marker ✓	boolean (bool)		Enrich the job metadata with a document marker (for document handling in batch mode). Default: false
wrap-as-json ✓	boolean (bool)		Format the output batch as a JSON array. Default: false

Retry - Backoff Fields

Field	Type	Required	Description
strategy	Strategy		Backoff strategy to use (default exponential). Allowed values: exponential, linear, fixed
base	string		Base delay before retrying (e.g. “200ms”).
max	string		Maximum delay between retries.
jitter ✓	boolean (bool)		Whether to add jitter to retry delays. Default: false

Retry Fields

Field	Type	Required	Description
max-attempts	number (integer)		Maximum attempts before giving up. Examples: 42, 1.2e-10
forever ✓	boolean (bool)		Retry indefinitely until cancelled. Default: false
backoff	Backoff		Backoff strategy configuration.

Schema - Overrides Fields

Field	Type	Required	Description
field-path	string	✅
target-type	string

Schema Fields

Field	Type	Required	Description
auto-fetch ✓	boolean (bool)		Reserved for GA follow-on work; BigQuery schema settings are not supported in 2.0.1. Default: true
overrides	Overrides[]		Reserved for GA follow-on work; BigQuery schema settings are not supported in 2.0.1.

Streaming Fields

Field	Type	Required	Description
stream-type	Stream Type		Only default is supported in 2.0.1. Storage Write stream types are GA follow-on work. Default: default Allowed values: default, committed, pending, buffered
insert-id-field	string		Field to use for deduplication (insertId).

Load Job - Write Disposition Options

Value	Aliases	Name	Description
WRITE_APPEND		WRITE_APPEND
WRITE_TRUNCATE		WRITE_TRUNCATE
WRITE_EMPTY		WRITE_EMPTY

Load Job - Create Disposition Options

Value	Aliases	Name	Description
CREATE_IF_NEEDED		CREATE_IF_NEEDED
CREATE_NEVER		CREATE_NEVER

Mode Options

Value	Aliases	Name	Description
streaming		streaming
load-job		load-job

Batch - Mode Options

Value	Aliases	Name	Description
fixed		fixed
document		document

Retry - Backoff - Strategy Options

Value	Aliases	Name	Description
exponential		exponential
linear		linear
fixed		fixed

Streaming - Stream Type Options

Value	Aliases	Name	Description
default		default
committed		committed
pending		pending
buffered		buffered