Skip to content
LyftData Docs

Google Cloud Storage

Google Cloud Storage (gcs)

Section titled “Google Cloud Storage (gcs)”

Read objects from Google Cloud Storage.

Block Store Cloud Enterprise edition binary json raw

Minimal example

Section titled “Minimal example”
input:
  gcs:
    bucket-name: ~
    credentials:
      storage-url: ~
    object-names: []
JSON 
{
  "input": {
    "gcs": {
      "bucket-name": null,
      "credentials": {
        "storage-url": null
      },
      "object-names": []
    }
  }
}

Contents

Section titled “Contents”

A ✓ marks a field that accepts a context variable such as {{ VARIABLE }}.

Fields

Section titled “Fields”
Field Type Required Description
payload-mode ✓ Payload Mode Select how payloads should be interpreted (auto, json, raw, binary).
Allowed values: auto, json, raw, binary

Authentication

Section titled “Authentication”
Authentication
Field Type Required Description
credentials Credentials ✅ Google Cloud credentials (service account JSON, workload identity, etc.). Supports literal values or context interpolation via PossibleContextString.
Allowed values: storage-url, service-account, application-credentials

Behavior

Section titled “Behavior”
Behavior
Field Type Required Description
mode Mode Input behavior: use list-and-download, list, or download depending on whether you want to enumerate objects, fetch contents, or download specific names directly.
Allowed values: list-and-download-objects, list-objects, download-objects
ignore-linebreaks ✓ boolean (bool) Treat the entire object as a single event. When false, objects are split on newlines unless json=true instructs the runtime to parse JSON arrays.
Default: false
timestamp-mode Timestamp Mode Derive a timestamp for this blob for filtering purposes based on the selected strategy (e.g., object creation time vs. last modification).
Allowed values: none, last-modified, object-name-pattern

Filtering

Section titled “Filtering”
Filtering
Field Type Required Description
maximum-age Maximum Age Ignore objects older than the provided duration (e.g., 5m, 1h30m). Leave empty to process all visible objects.
include-regex regex[] (string) Include objects matching the specified regular expressions.
Examples: \d+[A-Z]*
exclude-regex regex[] (string) Exclude objects matching the specified regular expressions.
Examples: \d+[A-Z]*

Location

Section titled “Location”
Location
Field Type Required Description
bucket-name string ✅ The GCS bucket to read from.
object-names string[] ✅ Object names or prefixes to target. Leave empty to consider every object exposed by the selected mode—take care with large buckets.

Object Properties

Section titled “Object Properties”
Object Properties
Field Type Required Description
object-name-field field (string) The field that the object name from an operation should be stored in.
Examples: data_field
creation-time-field field (string) The field that the object creation time should be stored in.
Examples: data_field
last-modified-field field (string) The field that the object last modified time should be stored in.
Examples: data_field
content-length-field field (string) The field that the object content length information should be stored in.
Examples: data_field
content-type-field field (string) The field that the object content type information should be stored in.
Examples: data_field
etag-field field (string) The field that the object ETag should be stored in.
Examples: data_field
data-field field (string) A field to take the object data (default is to merge fields if possible).
Examples: data_field

Processing

Section titled “Processing”
Processing
Field Type Required Description
preprocessors Preprocessors[] Preprocessors (process downloaded data before making it available to the job) these processors will be run in the order they are specified.
Allowed values: extension, gzip, parquet, base64

Reliability

Section titled “Reliability”
Reliability
Field Type Required Description
fingerprinting ✓ boolean (bool) Enable object fingerprinting to download each object only once, even across restarts.
Default: false
maximum-fingerprint-age duration (string) How long to retain stored fingerprints before they are eligible for cleanup.
Default: 30 days
retry Retry How to retry failed operations (backoff, max attempts, etc.).

Trigger

Section titled “Trigger”
Trigger
Field Type Required Description
trigger Trigger Run on the provided cadence; if omitted, this input runs exactly once.
Allowed values: message, cron, interval

Schema

Section titled “Schema”

Credentials - Service Account Options

Section titled “Credentials - Service Account Options”
Option Name Type Description
path Path string Service Account File location.
Examples: /path/to/file, c:\users\joe\data\file.txt
key Key string

Credentials - Application Credentials Options

Section titled “Credentials - Application Credentials Options”
Option Name Type Description
path Path string Application Credential File Location.
Examples: /path/to/file, c:\users\joe\data\file.txt
key Key string Application Credential in JSON format.

Credentials Options

Section titled “Credentials Options”
Option Name Type Description
storage-url Storage Url string URL in the form gs:///path.
service-account Service Account one_of Service Account credentials.
Allowed values: path, key
application-credentials Application Credentials one_of Application credentials.
Allowed values: path, key

Timestamp Mode Options

Section titled “Timestamp Mode Options”
Option Name Type Description
none None map The default mode, do not filter based on timestamps.
last-modified Last Modified map Filter object on the last-modified timestamp reported by the service.
object-name-pattern Object Name Pattern string Filter blobs on the timestamp derived from the object name for example: relevant-name-pattern: =(?P<Y>[\\d]{4,4})-(?P<m>[\\d]{2,2})-(?P<d>[\\d]{2,2})/.

Trigger - Cron - Window - Start Options

Section titled “Trigger - Cron - Window - Start Options”
Option Name Type Description
start-time Start Time object
tracked Tracked string Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger - Interval - Window - Start Options

Section titled “Trigger - Interval - Window - Start Options”
Option Name Type Description
start-time Start Time object
tracked Tracked string Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger Options

Section titled “Trigger Options”
Option Name Type Description
message Message object
cron Cron object
interval Interval object

Maximum Age Fields

Section titled “Maximum Age Fields”
Field Type Required Description
value duration (string) ✅

Retry - Backoff Fields

Section titled “Retry - Backoff Fields”
Field Type Required Description
strategy Strategy Backoff strategy to use (default exponential).
Allowed values: exponential, linear, fixed
base string Base delay before retrying (e.g. “200ms”).
max string Maximum delay between retries.
jitter ✓ boolean (bool) Whether to add jitter to retry delays.
Default: false

Retry Fields

Section titled “Retry Fields”
Field Type Required Description
max-attempts number (integer) Maximum attempts before giving up.
Examples: 42, 1.2e-10
forever ✓ boolean (bool) Retry indefinitely until cancelled.
Default: false
backoff Backoff Backoff strategy configuration.

Trigger - Message Fields

Section titled “Trigger - Message Fields”
Field Type Required Description
limit ✓ number (integer) The number of times to run the input.
Examples: 42, 1.2e-10
filter-kind Filter Kind Specifies whether the message originated from the “system” or by the “user”.
Allowed values: system, user, runtime-artifact-fetch, runtime-artifact-fetch-error, runtime-artifact-clear, runtime-artifact-clear-ack, runtime-artifact-fetch-reply, runtime-artifact-update, runtime-artifact-update-ack
filter-source Filter Source[] Specifies what process generated the message. Was it a “server”, “worker” or “job”?
Allowed values: job, worker, server
filter-worker string Specifies what worker to select.
filter-job string Specifies the name of the job that the message came from.
filter-type Filter Type[] Specifies that particular types of message ought to match.
Allowed values: worker-licensed, worker-unlicensed, variable, variable-deleted, begin-shutting-down-job, begin-shutting-down-server, begin-shutting-down-worker, broadcast-job-thread-state, broadcast-server-thread-state, broadcast-worker-thread-state, …
filter-tag string Specifies that messages matched ought to carry a tag with a particular value. This only matches against user-generated messages.

Trigger - Cron - Window - Start - Start Time Fields

Section titled “Trigger - Cron - Window - Start - Start Time Fields”
Field Type Required Description
start-time time-format (string) ✅ Allows the windowing to start at a specified time.
Hint: %Y-%m-%d %H:%M:%S%.3f %z
It should in the following format: 2019-07-10 18:45:00.000 +0200
highwatermark-file path (string) ✅ Specify file where timestamp would be stored in order to resume, for when Job has been restarted.
Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger - Cron - Window Fields

Section titled “Trigger - Cron - Window Fields”
Field Type Required Description
size duration (string) ✅ Window size.
offset duration (string) Window offset.
Default: 0s
start Start Specify file where timestamp would be stored in order to resume, for when Job has been restarted.
Allowed values: start-time, tracked

Trigger - Cron Fields

Section titled “Trigger - Cron Fields”
Field Type Required Description
cron cron-expression (string) ✅ The Cron pattern.
immediate ✓ boolean (bool) Run as soon as invoked, instead of waiting for the specified cron interval.
Default: false
random-offset duration (string) Sets a random offset to the schedule, then sticks to it.
Default: 0s
window Window Optional window definition when the schedule should only read a bounded range.

Trigger - Interval - Window - Start - Start Time Fields

Section titled “Trigger - Interval - Window - Start - Start Time Fields”
Field Type Required Description
start-time time-format (string) ✅ Allows the windowing to start at a specified time.
Hint: %Y-%m-%d %H:%M:%S%.3f %z
It should in the following format: 2019-07-10 18:45:00.000 +0200
highwatermark-file path (string) ✅ Specify file where timestamp would be stored in order to resume, for when Job has been restarted.
Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger - Interval - Window Fields

Section titled “Trigger - Interval - Window Fields”
Field Type Required Description
size duration (string) ✅ Window size.
offset duration (string) Window offset.
Default: 0s
start Start Specify file where timestamp would be stored in order to resume, for when Job has been restarted.
Allowed values: start-time, tracked

Trigger - Interval Fields

Section titled “Trigger - Interval Fields”
Field Type Required Description
duration duration (string) ✅ Duration to wait between events.
random-offset duration (string) Sets a random offset to the schedule, then sticks to it.
Default: 0s
window Window Optional window definition when the interval should only cover a bounded range.

Payload Mode Options

Section titled “Payload Mode Options”
Value Aliases Name Description
auto auto
json json
raw raw
binary binary

Mode Options

Section titled “Mode Options”
Value Aliases Name Description
list-and-download-objects list-and-download-objects List Objects and Download
list-objects list-objects List Objects
download-objects download-objects Download Given Objects

Preprocessors Options

Section titled “Preprocessors Options”
Value Aliases Name Description
extension extension Preprocess the object or blob based on the extension of the object or blob name (.gz, .parquet)
gzip gzip UnGzip the received data
parquet parquet Extract the received data as JSON rows from a parquet file
base64 base64 Encode the binary data as base64

Retry - Backoff - Strategy Options

Section titled “Retry - Backoff - Strategy Options”
Value Aliases Name Description
exponential exponential
linear linear
fixed fixed

Trigger - Message - Filter Kind Options

Section titled “Trigger - Message - Filter Kind Options”
Value Aliases Name Description
system system
user user
runtime-artifact-fetch runtime-artifact-fetch
runtime-artifact-fetch-error runtime-artifact-fetch-error
runtime-artifact-clear runtime-artifact-clear
runtime-artifact-clear-ack runtime-artifact-clear-ack
runtime-artifact-fetch-reply runtime-artifact-fetch-reply
runtime-artifact-update runtime-artifact-update
runtime-artifact-update-ack runtime-artifact-update-ack

Trigger - Message - Filter Source Options

Section titled “Trigger - Message - Filter Source Options”
Value Aliases Name Description
job job
worker worker
server server

Trigger - Message - Filter Type Options

Section titled “Trigger - Message - Filter Type Options”
Value Aliases Name Description
worker-licensed worker-licensed
worker-unlicensed worker-unlicensed
variable variable
variable-deleted variable-deleted
begin-shutting-down-job begin-shutting-down-job
begin-shutting-down-server begin-shutting-down-server
begin-shutting-down-worker begin-shutting-down-worker
broadcast-job-thread-state broadcast-job-thread-state
broadcast-server-thread-state broadcast-server-thread-state
broadcast-worker-thread-state broadcast-worker-thread-state
check-job-report-time check-job-report-time
check-worker-report-time check-worker-report-time
de-register-job-thread-dependency de-register-job-thread-dependency
de-register-server-thread-dependency de-register-server-thread-dependency
de-register-worker-thread-dependency de-register-worker-thread-dependency
deployed-job-active deployed-job-active
deployed-job-removed deployed-job-removed
deployed-job-should-be-running deployed-job-should-be-running
deployment-phase deployment-phase
durable-transport-effect-ack durable-transport-effect-ack
heart-beat heart-beat
initialise-internal-state initialise-internal-state
initialise-job-states initialise-job-states
job-batch-end job-batch-end
job-backlog-update job-backlog-update
job-analytics-batch job-analytics-batch
job-checkpoint-update job-checkpoint-update
job-deploy-ready job-deploy-ready
job-deployed job-deployed
job-document-end job-document-end
job-document-start job-document-start
job-errors job-errors
job-execution-anomaly job-execution-anomaly
job-execution-status job-execution-status
job-emit-custom job-emit-custom
job-finished job-finished
job-idle job-idle
job-initiated job-initiated
job-is-processing job-is-processing
job-logs job-logs
job-metrics job-metrics
job-notifications job-notifications
job-removing job-removing
job-removed job-removed
job-remove-failed job-remove-failed
job-remove-ready job-remove-ready
job-replaced job-replaced
job-required job-required
job-run-ended job-run-ended
job-runtime-error job-runtime-error
job-runtime-settings job-runtime-settings
job-run-started job-run-started
job-running-docker job-running-docker
job-running-script job-running-script
job-running-subprocess job-running-subprocess
job-running-system-d job-running-system-d
job-settings job-settings
job-step-statistics job-step-statistics
job-started job-started
job-staged job-staged
job-state-transition job-state-transition
job-shutting-down job-shutting-down
job-stopping job-stopping
job-stopped job-stopped
job-timed-out job-timed-out
job-suspicious-silence job-suspicious-silence
job-thread-state job-thread-state
job-trace job-trace
job-trace-requires-samples job-trace-requires-samples
job-updated job-updated
job-worker-comms-error job-worker-comms-error
job-unstaged job-unstaged
license-state-changed license-state-changed
license-validation-failed license-validation-failed
license-validation-ok license-validation-ok
license-volume-violation license-volume-violation
new-license new-license
override-job-coordinated-shutdown override-job-coordinated-shutdown
override-server-coordinated-shutdown override-server-coordinated-shutdown
override-worker-coordinated-shutdown override-worker-coordinated-shutdown
register-job-thread-dependency register-job-thread-dependency
register-server-thread-dependency register-server-thread-dependency
register-worker-thread-dependency register-worker-thread-dependency
run-job-failure run-job-failure
server-logs server-logs
server-metrics-batch server-metrics-batch
server-started server-started
server-starting server-starting
server-stopping server-stopping
server-thread-state server-thread-state
server-worker-comms-error server-worker-comms-error
shutdown-jobs shutdown-jobs
shutdown-worker shutdown-worker
system-shutdown system-shutdown
update-upstream-sync-for-job update-upstream-sync-for-job
update-upstream-sync-for-worker update-upstream-sync-for-worker
update-variable update-variable
user-alert user-alert
user-generated user-generated
trigger-response trigger-response
trigger-proxy-response trigger-proxy-response
workflow-step-terminal workflow-step-terminal
workflow-step-terminal-ack workflow-step-terminal-ack
user-notification user-notification
worker-command-for-job worker-command-for-job
worker-auth-lease-ack worker-auth-lease-ack
worker-connected worker-connected
worker-created worker-created
worker-debug-heart-beat worker-debug-heart-beat
worker-error worker-error
worker-first-seen worker-first-seen
worker-heart-beat worker-heart-beat
worker-logs worker-logs
worker-metrics-batch worker-metrics-batch
worker-offline worker-offline
worker-requests-auth-lease worker-requests-auth-lease
worker-server-comms-error worker-server-comms-error
worker-settings worker-settings
worker-shutdown worker-shutdown
worker-shutting-down worker-shutting-down
worker-started worker-started
worker-state-uuid worker-state-uuid
worker-stopping worker-stopping
worker-suspicious-silence worker-suspicious-silence
worker-system-information worker-system-information
worker-thread-state worker-thread-state
worker-updated worker-updated
worker-modified worker-modified
worker-removed worker-removed
context-changed context-changed
rerender-deployment rerender-deployment
job-killed job-killed
message-serviced message-serviced
failed-to-service-message failed-to-service-message
worker-wants-initial-settings worker-wants-initial-settings
worker-wants-initial-settings-reply worker-wants-initial-settings-reply
worker-wants-deployed-jobs worker-wants-deployed-jobs
worker-wants-deployed-jobs-reply worker-wants-deployed-jobs-reply
worker-wants-job-configuration worker-wants-job-configuration
worker-wants-job-configuration-reply worker-wants-job-configuration-reply
job-wants-dslir-key job-wants-dslir-key
job-wants-dslir-key-reply job-wants-dslir-key-reply
worker-wants-dslir-key worker-wants-dslir-key
worker-wants-dslir-key-reply worker-wants-dslir-key-reply
job-wants-variables job-wants-variables
job-wants-variables-reply job-wants-variables-reply
job-wants-credentials job-wants-credentials
job-wants-credentials-reply job-wants-credentials-reply
job-wants-credentials-error job-wants-credentials-error
job-wants-secret-variables-reply job-wants-secret-variables-reply
job-credentials-invalidated job-credentials-invalidated
aggregator-health aggregator-health
worker-aggregator-health worker-aggregator-health
worker-verification-token worker-verification-token
worker-requests-verification-token worker-requests-verification-token
runtime-artifact-update runtime-artifact-update
runtime-artifact-update-ack runtime-artifact-update-ack
runtime-artifact-clear runtime-artifact-clear
runtime-artifact-clear-ack runtime-artifact-clear-ack
runtime-artifact-fetch runtime-artifact-fetch
runtime-artifact-fetch-reply runtime-artifact-fetch-reply
runtime-artifact-fetch-error runtime-artifact-fetch-error