FTP

FTP (ftp)

Poll remote FTP/SFTP/TFTP servers for new files.

File Utility Enterprise edition binary json raw

Minimal example

input:
  ftp:
    endpoint:
      host: ~
    path: ~

JSON

{
  "input": {
    "ftp": {
      "endpoint": {
        "host": null
      },
      "path": null
    }
  }
}

Contents

• Minimal example

• Fields

• Schema

A ✓ marks a field that accepts a context variable such as {{ VARIABLE }}.

Fields

Field	Type	Required	Description
trigger	Trigger		Allowed values: message, cron, interval
retry	Retry
protocol	Protocol		Allowed values: ftp, ftps-explicit, ftps-implicit, sftp, tftp
endpoint	Endpoint	✅
credentials	Credentials		Allowed values: username-password, tls-client-cert, ssh-key, anonymous, none
working-directory	string
path	string	✅
include-regex	regex[] (string)		Examples: \d+[A-Z]*
exclude-regex	regex[] (string)		Examples: \d+[A-Z]*
recursive ✓	boolean (bool)		Default: false
metadata	Metadata
payload-mode ✓	Payload Mode		Allowed values: auto, json, raw, binary
concurrency	number (integer)		Examples: 42, 1.2e-10
delete-after-download ✓	boolean (bool)		Default: false
preprocessors	Preprocessors[]		Allowed values: extension, gzip, parquet, base64
tmp-suffix	string
passive ✓	boolean (bool)
ftps	Ftps
sftp	Sftp
tftp	Tftp
stream-buffer-size	number (integer)		Examples: 42, 1.2e-10
resume-checkpoint-variable	string

Schema

• Trigger - Cron - Window - Start Options
• Trigger - Interval - Window - Start Options
• Trigger Options
• Credentials Options
• Sftp - Host Key Verification Options
• Trigger - Message Fields
• Trigger - Cron - Window - Start - Start Time Fields
• Trigger - Cron - Window Fields
• Trigger - Cron Fields
• Trigger - Interval - Window - Start - Start Time Fields
• Trigger - Interval - Window Fields
• Trigger - Interval Fields
• Retry - Backoff Fields
• Retry Fields
• Endpoint Fields
• Credentials - Username Password Fields
• Credentials - Tls Client Cert Fields
• Credentials - Ssh Key Fields
• Metadata Fields
• Ftps Fields
• Sftp - Host Key Verification - Known Hosts File Fields
• Sftp - Host Key Verification - Fingerprint Fields
• Sftp Fields
• Tftp Fields
• Trigger - Message - Filter Kind Options
• Trigger - Message - Filter Source Options
• Trigger - Message - Filter Type Options
• Retry - Backoff - Strategy Options
• Protocol Options
• Payload Mode Options
• Preprocessors Options

Trigger - Cron - Window - Start Options

Option	Name	Type	Description
start-time	Start Time	object
tracked	Tracked	string	Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger - Interval - Window - Start Options

Option	Name	Type	Description
start-time	Start Time	object
tracked	Tracked	string	Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger Options

Option	Name	Type	Description
message	Message	object
cron	Cron	object
interval	Interval	object

Credentials Options

Option	Name	Type	Description
username-password	Username Password	object
tls-client-cert	Tls Client Cert	object
ssh-key	Ssh Key	object
anonymous	Anonymous	map
none	None	map

Sftp - Host Key Verification Options

Option	Name	Type	Description
known-hosts-file	Known Hosts File	object
fingerprint	Fingerprint	object
accept-new	Accept New	map

Trigger - Message Fields

Field	Type	Required	Description
limit ✓	number (integer)		The number of times to run the input. Examples: 42, 1.2e-10
filter-kind	Filter Kind		Specifies whether the message originated from the “system” or by the “user”. Allowed values: system, user, runtime-artifact-fetch, runtime-artifact-fetch-error, runtime-artifact-clear, runtime-artifact-clear-ack, runtime-artifact-fetch-reply, runtime-artifact-update, runtime-artifact-update-ack
filter-source	Filter Source[]		Specifies what process generated the message. Was it a “server”, “worker” or “job”? Allowed values: job, worker, server
filter-worker	string		Specifies what worker to select.
filter-job	string		Specifies the name of the job that the message came from.
filter-type	Filter Type[]		Specifies that particular types of message ought to match. Allowed values: worker-licensed, worker-unlicensed, variable, variable-deleted, begin-shutting-down-job, begin-shutting-down-server, begin-shutting-down-worker, broadcast-job-thread-state, broadcast-server-thread-state, broadcast-worker-thread-state, …
filter-tag	string		Specifies that messages matched ought to carry a tag with a particular value. This only matches against user-generated messages.

Trigger - Cron - Window - Start - Start Time Fields

Field	Type	Required	Description
start-time	time-format (string)	✅	Allows the windowing to start at a specified time. Hint: %Y-%m-%d %H:%M:%S%.3f %z It should in the following format: 2019-07-10 18:45:00.000 +0200
highwatermark-file	path (string)	✅	Specify file where timestamp would be stored in order to resume, for when Job has been restarted. Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger - Cron - Window Fields

Field	Type	Required	Description
size	duration (string)	✅	Window size.
offset	duration (string)		Window offset. Default: 0s
start	Start		Specify file where timestamp would be stored in order to resume, for when Job has been restarted. Allowed values: start-time, tracked

Trigger - Cron Fields

Field	Type	Required	Description
cron	cron-expression (string)	✅	The Cron pattern.
immediate ✓	boolean (bool)		Run as soon as invoked, instead of waiting for the specified cron interval. Default: false
random-offset	duration (string)		Sets a random offset to the schedule, then sticks to it. Default: 0s
window	Window		Optional window definition when the schedule should only read a bounded range.

Trigger - Interval - Window - Start - Start Time Fields

Field	Type	Required	Description
start-time	time-format (string)	✅	Allows the windowing to start at a specified time. Hint: %Y-%m-%d %H:%M:%S%.3f %z It should in the following format: 2019-07-10 18:45:00.000 +0200
highwatermark-file	path (string)	✅	Specify file where timestamp would be stored in order to resume, for when Job has been restarted. Examples: /path/to/file, c:\users\joe\data\file.txt

Trigger - Interval - Window Fields

Field	Type	Required	Description
size	duration (string)	✅	Window size.
offset	duration (string)		Window offset. Default: 0s
start	Start		Specify file where timestamp would be stored in order to resume, for when Job has been restarted. Allowed values: start-time, tracked

Trigger - Interval Fields

Field	Type	Required	Description
duration	duration (string)	✅	Duration to wait between events.
random-offset	duration (string)		Sets a random offset to the schedule, then sticks to it. Default: 0s
window	Window		Optional window definition when the interval should only cover a bounded range.

Retry - Backoff Fields

Field	Type	Required	Description
strategy	Strategy		Backoff strategy to use (default exponential). Allowed values: exponential, linear, fixed
base	string		Base delay before retrying (e.g. “200ms”).
max	string		Maximum delay between retries.
jitter ✓	boolean (bool)		Whether to add jitter to retry delays. Default: false

Retry Fields

Field	Type	Required	Description
max-attempts	number (integer)		Maximum attempts before giving up. Examples: 42, 1.2e-10
forever ✓	boolean (bool)		Retry indefinitely until cancelled. Default: false
backoff	Backoff		Backoff strategy configuration.

Endpoint Fields

Field	Type	Required	Description
host	string	✅
port	number (integer)		Examples: 42, 1.2e-10
proxy	string
bastion-host	string

Credentials - Username Password Fields

Field	Type	Required	Description
username-variable	string	✅
password-variable	string	✅

Credentials - Tls Client Cert Fields

Field	Type	Required	Description
username-variable	string	✅
password-variable	string	✅
client-certificate-variable	string	✅
client-key-variable	string	✅

Credentials - Ssh Key Fields

Field	Type	Required	Description
username-variable	string	✅
private-key-variable	string	✅
passphrase-variable	string

Metadata Fields

Field	Type	Required	Description
object-name-field	field (string)		Examples: data_field
content-length-field	field (string)		Examples: data_field
last-modified-field	field (string)		Examples: data_field
checksum-field	field (string)		Examples: data_field
content-type-field	field (string)		Examples: data_field
etag-field	field (string)		Examples: data_field

Ftps Fields

Field	Type	Required	Description
ca-certificate-variable	string
client-certificate-variable	string
client-key-variable	string
cipher-list	string
verify-peer ✓	boolean (bool)		Default: false
min-tls-version	string

Sftp - Host Key Verification - Known Hosts File Fields

Field	Type	Required	Description
path	path (string)	✅	Examples: /path/to/file, c:\users\joe\data\file.txt

Sftp - Host Key Verification - Fingerprint Fields

Field	Type	Required	Description
fingerprint	string	✅

Sftp Fields

Field	Type	Required	Description
agent-forwarding ✓	boolean (bool)		Default: false
known-hosts-path	string
host-key-verification	Host Key Verification		Allowed values: known-hosts-file, fingerprint, accept-new
jump-host	string

Tftp Fields

Field	Type	Required	Description
block-size	number (integer)		Examples: 42, 1.2e-10
timeout-ms	number (integer)		Examples: 42, 1.2e-10
max-retries	number
file-name	string

Trigger - Message - Filter Kind Options

Value	Aliases	Name	Description
system		system
user		user
runtime-artifact-fetch		runtime-artifact-fetch
runtime-artifact-fetch-error		runtime-artifact-fetch-error
runtime-artifact-clear		runtime-artifact-clear
runtime-artifact-clear-ack		runtime-artifact-clear-ack
runtime-artifact-fetch-reply		runtime-artifact-fetch-reply
runtime-artifact-update		runtime-artifact-update
runtime-artifact-update-ack		runtime-artifact-update-ack

Trigger - Message - Filter Source Options

Value	Aliases	Name	Description
job		job
worker		worker
server		server

Trigger - Message - Filter Type Options

Value	Aliases	Name	Description
worker-licensed		worker-licensed
worker-unlicensed		worker-unlicensed
variable		variable
variable-deleted		variable-deleted
begin-shutting-down-job		begin-shutting-down-job
begin-shutting-down-server		begin-shutting-down-server
begin-shutting-down-worker		begin-shutting-down-worker
broadcast-job-thread-state		broadcast-job-thread-state
broadcast-server-thread-state		broadcast-server-thread-state
broadcast-worker-thread-state		broadcast-worker-thread-state
check-job-report-time		check-job-report-time
check-worker-report-time		check-worker-report-time
de-register-job-thread-dependency		de-register-job-thread-dependency
de-register-server-thread-dependency		de-register-server-thread-dependency
de-register-worker-thread-dependency		de-register-worker-thread-dependency
deployed-job-active		deployed-job-active
deployed-job-removed		deployed-job-removed
deployed-job-should-be-running		deployed-job-should-be-running
deployment-phase		deployment-phase
durable-transport-effect-ack		durable-transport-effect-ack
heart-beat		heart-beat
initialise-internal-state		initialise-internal-state
initialise-job-states		initialise-job-states
job-batch-end		job-batch-end
job-backlog-update		job-backlog-update
job-analytics-batch		job-analytics-batch
job-checkpoint-update		job-checkpoint-update
job-deploy-ready		job-deploy-ready
job-deployed		job-deployed
job-document-end		job-document-end
job-document-start		job-document-start
job-errors		job-errors
job-execution-anomaly		job-execution-anomaly
job-execution-status		job-execution-status
job-emit-custom		job-emit-custom
job-finished		job-finished
job-idle		job-idle
job-initiated		job-initiated
job-is-processing		job-is-processing
job-logs		job-logs
job-metrics		job-metrics
job-notifications		job-notifications
job-removing		job-removing
job-removed		job-removed
job-remove-failed		job-remove-failed
job-remove-ready		job-remove-ready
job-replaced		job-replaced
job-required		job-required
job-run-ended		job-run-ended
job-runtime-error		job-runtime-error
job-runtime-settings		job-runtime-settings
job-run-started		job-run-started
job-running-docker		job-running-docker
job-running-script		job-running-script
job-running-subprocess		job-running-subprocess
job-running-system-d		job-running-system-d
job-settings		job-settings
job-step-statistics		job-step-statistics
job-started		job-started
job-staged		job-staged
job-state-transition		job-state-transition
job-shutting-down		job-shutting-down
job-stopping		job-stopping
job-stopped		job-stopped
job-timed-out		job-timed-out
job-suspicious-silence		job-suspicious-silence
job-thread-state		job-thread-state
job-trace		job-trace
job-trace-requires-samples		job-trace-requires-samples
job-updated		job-updated
job-worker-comms-error		job-worker-comms-error
job-unstaged		job-unstaged
license-state-changed		license-state-changed
license-validation-failed		license-validation-failed
license-validation-ok		license-validation-ok
license-volume-violation		license-volume-violation
new-license		new-license
override-job-coordinated-shutdown		override-job-coordinated-shutdown
override-server-coordinated-shutdown		override-server-coordinated-shutdown
override-worker-coordinated-shutdown		override-worker-coordinated-shutdown
register-job-thread-dependency		register-job-thread-dependency
register-server-thread-dependency		register-server-thread-dependency
register-worker-thread-dependency		register-worker-thread-dependency
run-job-failure		run-job-failure
server-logs		server-logs
server-metrics-batch		server-metrics-batch
server-started		server-started
server-starting		server-starting
server-stopping		server-stopping
server-thread-state		server-thread-state
server-worker-comms-error		server-worker-comms-error
shutdown-jobs		shutdown-jobs
shutdown-worker		shutdown-worker
system-shutdown		system-shutdown
update-upstream-sync-for-job		update-upstream-sync-for-job
update-upstream-sync-for-worker		update-upstream-sync-for-worker
update-variable		update-variable
user-alert		user-alert
user-generated		user-generated
trigger-response		trigger-response
trigger-proxy-response		trigger-proxy-response
workflow-step-terminal		workflow-step-terminal
workflow-step-terminal-ack		workflow-step-terminal-ack
user-notification		user-notification
worker-command-for-job		worker-command-for-job
worker-auth-lease-ack		worker-auth-lease-ack
worker-connected		worker-connected
worker-created		worker-created
worker-debug-heart-beat		worker-debug-heart-beat
worker-error		worker-error
worker-first-seen		worker-first-seen
worker-heart-beat		worker-heart-beat
worker-logs		worker-logs
worker-metrics-batch		worker-metrics-batch
worker-offline		worker-offline
worker-requests-auth-lease		worker-requests-auth-lease
worker-server-comms-error		worker-server-comms-error
worker-settings		worker-settings
worker-shutdown		worker-shutdown
worker-shutting-down		worker-shutting-down
worker-started		worker-started
worker-state-uuid		worker-state-uuid
worker-stopping		worker-stopping
worker-suspicious-silence		worker-suspicious-silence
worker-system-information		worker-system-information
worker-thread-state		worker-thread-state
worker-updated		worker-updated
worker-modified		worker-modified
worker-removed		worker-removed
context-changed		context-changed
rerender-deployment		rerender-deployment
job-killed		job-killed
message-serviced		message-serviced
failed-to-service-message		failed-to-service-message
worker-wants-initial-settings		worker-wants-initial-settings
worker-wants-initial-settings-reply		worker-wants-initial-settings-reply
worker-wants-deployed-jobs		worker-wants-deployed-jobs
worker-wants-deployed-jobs-reply		worker-wants-deployed-jobs-reply
worker-wants-job-configuration		worker-wants-job-configuration
worker-wants-job-configuration-reply		worker-wants-job-configuration-reply
job-wants-dslir-key		job-wants-dslir-key
job-wants-dslir-key-reply		job-wants-dslir-key-reply
worker-wants-dslir-key		worker-wants-dslir-key
worker-wants-dslir-key-reply		worker-wants-dslir-key-reply
job-wants-variables		job-wants-variables
job-wants-variables-reply		job-wants-variables-reply
job-wants-credentials		job-wants-credentials
job-wants-credentials-reply		job-wants-credentials-reply
job-wants-credentials-error		job-wants-credentials-error
job-wants-secret-variables-reply		job-wants-secret-variables-reply
job-credentials-invalidated		job-credentials-invalidated
aggregator-health		aggregator-health
worker-aggregator-health		worker-aggregator-health
worker-verification-token		worker-verification-token
worker-requests-verification-token		worker-requests-verification-token
runtime-artifact-update		runtime-artifact-update
runtime-artifact-update-ack		runtime-artifact-update-ack
runtime-artifact-clear		runtime-artifact-clear
runtime-artifact-clear-ack		runtime-artifact-clear-ack
runtime-artifact-fetch		runtime-artifact-fetch
runtime-artifact-fetch-reply		runtime-artifact-fetch-reply
runtime-artifact-fetch-error		runtime-artifact-fetch-error

Retry - Backoff - Strategy Options

Value	Aliases	Name	Description
exponential		exponential
linear		linear
fixed		fixed

Protocol Options

Value	Aliases	Name	Description
ftp		ftp
ftps-explicit		ftps-explicit
ftps-implicit		ftps-implicit
sftp		sftp
tftp		tftp

Payload Mode Options

Value	Aliases	Name	Description
auto		auto
json		json
raw		raw
binary		binary

Preprocessors Options

Value	Aliases	Name	Description
extension		extension	Preprocess the object or blob based on the extension of the object or blob name (.gz, .parquet)
gzip		gzip	UnGzip the received data
parquet		parquet	Extract the received data as JSON rows from a parquet file
base64		base64	Encode the binary data as base64