Web Dav Store
Web Dav Store (web-dav-store)
Section titled “Web Dav Store (web-dav-store)”Read from a WebDAV-compatible object store.
Block Store WebDAV Enterprise edition binary json raw
Minimal example
Section titled “Minimal example”input: web-dav-store: base-url: ~ object-names: []JSON
{ "input": { "web-dav-store": { "base-url": null, "object-names": [] } }}Contents
Section titled “Contents”A ✓ marks a field that accepts a context variable such as
{{ VARIABLE }}.
Fields
Section titled “Fields”| Field | Type | Required | Description |
|---|---|---|---|
fingerprinting-db-path |
path (string) |
Specify a custom path for the fingerprinting database. Examples: /path/to/file, c:\users\joe\data\file.txt |
Advanced
Section titled “Advanced”Advanced
| Field | Type | Required | Description |
|---|---|---|---|
config |
map (string) |
Additional configuration passed directly to the HTTP client (see object_store http configs). |
Behavior
Section titled “Behavior”Behavior
| Field | Type | Required | Description |
|---|---|---|---|
mode |
Mode |
Input behavior: list-and-download, list, or download.Allowed values: list-and-download-objects, list-objects, download-objects |
|
ignore-linebreaks ✓ |
boolean (bool) |
Treat the entire object as a single event. Default: false |
|
payload-mode ✓ |
Payload Mode |
Select how payloads should be interpreted (auto, json, raw, binary).Allowed values: auto, json, raw, binary |
|
timestamp-mode |
Timestamp Mode |
Derive a timestamp for this object for filtering purposes based on the selected strategy. Allowed values: none, last-modified, object-name-pattern |
Filtering
Section titled “Filtering”Filtering
| Field | Type | Required | Description |
|---|---|---|---|
maximum-age |
Maximum Age |
Ignore objects older than the provided duration (e.g., 5m, 1h30m). Leave empty to process all visible objects. |
|
include-regex |
regex[] (string) |
Use regex filters to restrict which objects are listed or downloaded. Examples: \d+[A-Z]* |
|
exclude-regex |
regex[] (string) |
Use regex filters to exclude objects from being processed. Examples: \d+[A-Z]* |
Location
Section titled “Location”Location
| Field | Type | Required | Description |
|---|---|---|---|
base-url |
string |
✅ | Base URL for the WebDAV endpoint (e.g., https://host/webdav). |
bucket-name |
string |
Optional bucket or root path that will be prepended to object names. | |
object-names |
string[] |
✅ | Names for objects. If using a list mode these are treated as prefixes. |
Object Properties
Section titled “Object Properties”Object Properties
| Field | Type | Required | Description |
|---|---|---|---|
object-name-field |
field (string) |
The field that an object name from an operation should be stored in. Examples: data_field |
|
creation-time-field |
field (string) |
The field that the object creation time should be stored in. Examples: data_field |
|
last-modified-field |
field (string) |
The field that the object last modified time should be stored in. Examples: data_field |
|
content-length-field |
field (string) |
The field that the object content length information should be stored in. Examples: data_field |
|
content-type-field |
field (string) |
The field that the object content type information should be stored in. Examples: data_field |
|
etag-field |
field (string) |
The field that the object ETag should be stored in. Examples: data_field |
|
data-field |
field (string) |
A field that the object data should be nested in. Examples: data_field |
Processing
Section titled “Processing”Processing
| Field | Type | Required | Description |
|---|---|---|---|
preprocessors |
Preprocessors[] |
Preprocessors (process downloaded data before making it available to the job) these processors will be run in the order they are specified. Allowed values: extension, gzip, parquet, base64 |
Reliability
Section titled “Reliability”Reliability
| Field | Type | Required | Description |
|---|---|---|---|
fingerprinting ✓ |
boolean (bool) |
Enable object fingerprinting to download each object only once, even across restarts. Default: false |
|
maximum-fingerprint-age |
duration (string) |
How long to retain stored fingerprints before they are eligible for cleanup. Default: 30 days |
|
retry |
Retry |
Retry configuration for WebDAV operations. |
Trigger
Section titled “Trigger”Trigger
| Field | Type | Required | Description |
|---|---|---|---|
trigger |
Trigger |
Run on the provided cadence; if omitted, this input runs exactly once. Allowed values: message, cron, interval |
Schema
Section titled “Schema”- Timestamp Mode Options
- Trigger - Cron - Window - Start Options
- Trigger - Interval - Window - Start Options
- Trigger Options
- Maximum Age Fields
- Retry - Backoff Fields
- Retry Fields
- Trigger - Message Fields
- Trigger - Cron - Window - Start - Start Time Fields
- Trigger - Cron - Window Fields
- Trigger - Cron Fields
- Trigger - Interval - Window - Start - Start Time Fields
- Trigger - Interval - Window Fields
- Trigger - Interval Fields
- Config Table
- Mode Options
- Payload Mode Options
- Preprocessors Options
- Retry - Backoff - Strategy Options
- Trigger - Message - Filter Kind Options
- Trigger - Message - Filter Source Options
- Trigger - Message - Filter Type Options
Timestamp Mode Options
Section titled “Timestamp Mode Options”| Option | Name | Type | Description |
|---|---|---|---|
none |
None | map |
The default mode, do not filter based on timestamps. |
last-modified |
Last Modified | map |
Filter object on the last-modified timestamp reported by the service. |
object-name-pattern |
Object Name Pattern | string |
Filter blobs on the timestamp derived from the object name for example: relevant-name-pattern: =(?P<Y>[\\d]{4,4})-(?P<m>[\\d]{2,2})-(?P<d>[\\d]{2,2})/. |
Trigger - Cron - Window - Start Options
Section titled “Trigger - Cron - Window - Start Options”| Option | Name | Type | Description |
|---|---|---|---|
start-time |
Start Time | object |
|
tracked |
Tracked | string |
Examples: /path/to/file, c:\users\joe\data\file.txt |
Trigger - Interval - Window - Start Options
Section titled “Trigger - Interval - Window - Start Options”| Option | Name | Type | Description |
|---|---|---|---|
start-time |
Start Time | object |
|
tracked |
Tracked | string |
Examples: /path/to/file, c:\users\joe\data\file.txt |
Trigger Options
Section titled “Trigger Options”| Option | Name | Type | Description |
|---|---|---|---|
message |
Message | object |
|
cron |
Cron | object |
|
interval |
Interval | object |
Maximum Age Fields
Section titled “Maximum Age Fields”| Field | Type | Required | Description |
|---|---|---|---|
value |
duration (string) |
✅ |
Retry - Backoff Fields
Section titled “Retry - Backoff Fields”| Field | Type | Required | Description |
|---|---|---|---|
strategy |
Strategy |
Backoff strategy to use (default exponential). Allowed values: exponential, linear, fixed |
|
base |
string |
Base delay before retrying (e.g. “200ms”). | |
max |
string |
Maximum delay between retries. | |
jitter ✓ |
boolean (bool) |
Whether to add jitter to retry delays. Default: false |
Retry Fields
Section titled “Retry Fields”| Field | Type | Required | Description |
|---|---|---|---|
max-attempts |
number (integer) |
Maximum attempts before giving up. Examples: 42, 1.2e-10 |
|
forever ✓ |
boolean (bool) |
Retry indefinitely until cancelled. Default: false |
|
backoff |
Backoff |
Backoff strategy configuration. |
Trigger - Message Fields
Section titled “Trigger - Message Fields”| Field | Type | Required | Description |
|---|---|---|---|
limit ✓ |
number (integer) |
The number of times to run the input. Examples: 42, 1.2e-10 |
|
filter-kind |
Filter Kind |
Specifies whether the message originated from the “system” or by the “user”. Allowed values: system, user, runtime-artifact-fetch, runtime-artifact-fetch-error, runtime-artifact-clear, runtime-artifact-clear-ack, runtime-artifact-fetch-reply, runtime-artifact-update, runtime-artifact-update-ack |
|
filter-source |
Filter Source[] |
Specifies what process generated the message. Was it a “server”, “worker” or “job”? Allowed values: job, worker, server |
|
filter-worker |
string |
Specifies what worker to select. | |
filter-job |
string |
Specifies the name of the job that the message came from. | |
filter-type |
Filter Type[] |
Specifies that particular types of message ought to match. Allowed values: worker-licensed, worker-unlicensed, variable, variable-deleted, begin-shutting-down-job, begin-shutting-down-server, begin-shutting-down-worker, broadcast-job-thread-state, broadcast-server-thread-state, broadcast-worker-thread-state, … |
|
filter-tag |
string |
Specifies that messages matched ought to carry a tag with a particular value. This only matches against user-generated messages. |
Trigger - Cron - Window - Start - Start Time Fields
Section titled “Trigger - Cron - Window - Start - Start Time Fields”| Field | Type | Required | Description |
|---|---|---|---|
start-time |
time-format (string) |
✅ | Allows the windowing to start at a specified time. Hint: %Y-%m-%d %H:%M:%S%.3f %zIt should in the following format: 2019-07-10 18:45:00.000 +0200 |
highwatermark-file |
path (string) |
✅ | Specify file where timestamp would be stored in order to resume, for when Job has been restarted. Examples: /path/to/file, c:\users\joe\data\file.txt |
Trigger - Cron - Window Fields
Section titled “Trigger - Cron - Window Fields”| Field | Type | Required | Description |
|---|---|---|---|
size |
duration (string) |
✅ | Window size. |
offset |
duration (string) |
Window offset. Default: 0s |
|
start |
Start |
Specify file where timestamp would be stored in order to resume, for when Job has been restarted. Allowed values: start-time, tracked |
Trigger - Cron Fields
Section titled “Trigger - Cron Fields”| Field | Type | Required | Description |
|---|---|---|---|
cron |
cron-expression (string) |
✅ | The Cron pattern. |
immediate ✓ |
boolean (bool) |
Run as soon as invoked, instead of waiting for the specified cron interval. Default: false |
|
random-offset |
duration (string) |
Sets a random offset to the schedule, then sticks to it. Default: 0s |
|
window |
Window |
Optional window definition when the schedule should only read a bounded range. |
Trigger - Interval - Window - Start - Start Time Fields
Section titled “Trigger - Interval - Window - Start - Start Time Fields”| Field | Type | Required | Description |
|---|---|---|---|
start-time |
time-format (string) |
✅ | Allows the windowing to start at a specified time. Hint: %Y-%m-%d %H:%M:%S%.3f %zIt should in the following format: 2019-07-10 18:45:00.000 +0200 |
highwatermark-file |
path (string) |
✅ | Specify file where timestamp would be stored in order to resume, for when Job has been restarted. Examples: /path/to/file, c:\users\joe\data\file.txt |
Trigger - Interval - Window Fields
Section titled “Trigger - Interval - Window Fields”| Field | Type | Required | Description |
|---|---|---|---|
size |
duration (string) |
✅ | Window size. |
offset |
duration (string) |
Window offset. Default: 0s |
|
start |
Start |
Specify file where timestamp would be stored in order to resume, for when Job has been restarted. Allowed values: start-time, tracked |
Trigger - Interval Fields
Section titled “Trigger - Interval Fields”| Field | Type | Required | Description |
|---|---|---|---|
duration |
duration (string) |
✅ | Duration to wait between events. |
random-offset |
duration (string) |
Sets a random offset to the schedule, then sticks to it. Default: 0s |
|
window |
Window |
Optional window definition when the interval should only cover a bounded range. |
Config Table
Section titled “Config Table”| Setting | Value |
|---|---|
setting.name |
value |
Value format: templated-text.
Mode Options
Section titled “Mode Options”| Value | Aliases | Name | Description |
|---|---|---|---|
list-and-download-objects |
list-and-download-objects | List Objects and Download | |
list-objects |
list-objects | List Objects | |
download-objects |
download-objects | Download Given Objects |
Payload Mode Options
Section titled “Payload Mode Options”| Value | Aliases | Name | Description |
|---|---|---|---|
auto |
auto | ||
json |
json | ||
raw |
raw | ||
binary |
binary |
Preprocessors Options
Section titled “Preprocessors Options”| Value | Aliases | Name | Description |
|---|---|---|---|
extension |
extension | Preprocess the object or blob based on the extension of the object or blob name (.gz, .parquet) | |
gzip |
gzip | UnGzip the received data | |
parquet |
parquet | Extract the received data as JSON rows from a parquet file | |
base64 |
base64 | Encode the binary data as base64 |
Retry - Backoff - Strategy Options
Section titled “Retry - Backoff - Strategy Options”| Value | Aliases | Name | Description |
|---|---|---|---|
exponential |
exponential | ||
linear |
linear | ||
fixed |
fixed |
Trigger - Message - Filter Kind Options
Section titled “Trigger - Message - Filter Kind Options”| Value | Aliases | Name | Description |
|---|---|---|---|
system |
system | ||
user |
user | ||
runtime-artifact-fetch |
runtime-artifact-fetch | ||
runtime-artifact-fetch-error |
runtime-artifact-fetch-error | ||
runtime-artifact-clear |
runtime-artifact-clear | ||
runtime-artifact-clear-ack |
runtime-artifact-clear-ack | ||
runtime-artifact-fetch-reply |
runtime-artifact-fetch-reply | ||
runtime-artifact-update |
runtime-artifact-update | ||
runtime-artifact-update-ack |
runtime-artifact-update-ack |
Trigger - Message - Filter Source Options
Section titled “Trigger - Message - Filter Source Options”| Value | Aliases | Name | Description |
|---|---|---|---|
job |
job | ||
worker |
worker | ||
server |
server |
Trigger - Message - Filter Type Options
Section titled “Trigger - Message - Filter Type Options”| Value | Aliases | Name | Description |
|---|---|---|---|
worker-licensed |
worker-licensed | ||
worker-unlicensed |
worker-unlicensed | ||
variable |
variable | ||
variable-deleted |
variable-deleted | ||
begin-shutting-down-job |
begin-shutting-down-job | ||
begin-shutting-down-server |
begin-shutting-down-server | ||
begin-shutting-down-worker |
begin-shutting-down-worker | ||
broadcast-job-thread-state |
broadcast-job-thread-state | ||
broadcast-server-thread-state |
broadcast-server-thread-state | ||
broadcast-worker-thread-state |
broadcast-worker-thread-state | ||
check-job-report-time |
check-job-report-time | ||
check-worker-report-time |
check-worker-report-time | ||
de-register-job-thread-dependency |
de-register-job-thread-dependency | ||
de-register-server-thread-dependency |
de-register-server-thread-dependency | ||
de-register-worker-thread-dependency |
de-register-worker-thread-dependency | ||
deployed-job-active |
deployed-job-active | ||
deployed-job-removed |
deployed-job-removed | ||
deployed-job-should-be-running |
deployed-job-should-be-running | ||
deployment-phase |
deployment-phase | ||
durable-transport-effect-ack |
durable-transport-effect-ack | ||
heart-beat |
heart-beat | ||
initialise-internal-state |
initialise-internal-state | ||
initialise-job-states |
initialise-job-states | ||
job-batch-end |
job-batch-end | ||
job-backlog-update |
job-backlog-update | ||
job-analytics-batch |
job-analytics-batch | ||
job-checkpoint-update |
job-checkpoint-update | ||
job-deploy-ready |
job-deploy-ready | ||
job-deployed |
job-deployed | ||
job-document-end |
job-document-end | ||
job-document-start |
job-document-start | ||
job-errors |
job-errors | ||
job-execution-anomaly |
job-execution-anomaly | ||
job-execution-status |
job-execution-status | ||
job-emit-custom |
job-emit-custom | ||
job-finished |
job-finished | ||
job-idle |
job-idle | ||
job-initiated |
job-initiated | ||
job-is-processing |
job-is-processing | ||
job-logs |
job-logs | ||
job-metrics |
job-metrics | ||
job-notifications |
job-notifications | ||
job-removing |
job-removing | ||
job-removed |
job-removed | ||
job-remove-failed |
job-remove-failed | ||
job-remove-ready |
job-remove-ready | ||
job-replaced |
job-replaced | ||
job-required |
job-required | ||
job-run-ended |
job-run-ended | ||
job-runtime-error |
job-runtime-error | ||
job-runtime-settings |
job-runtime-settings | ||
job-run-started |
job-run-started | ||
job-running-docker |
job-running-docker | ||
job-running-script |
job-running-script | ||
job-running-subprocess |
job-running-subprocess | ||
job-running-system-d |
job-running-system-d | ||
job-settings |
job-settings | ||
job-step-statistics |
job-step-statistics | ||
job-started |
job-started | ||
job-staged |
job-staged | ||
job-state-transition |
job-state-transition | ||
job-shutting-down |
job-shutting-down | ||
job-stopping |
job-stopping | ||
job-stopped |
job-stopped | ||
job-timed-out |
job-timed-out | ||
job-suspicious-silence |
job-suspicious-silence | ||
job-thread-state |
job-thread-state | ||
job-trace |
job-trace | ||
job-trace-requires-samples |
job-trace-requires-samples | ||
job-updated |
job-updated | ||
job-worker-comms-error |
job-worker-comms-error | ||
job-unstaged |
job-unstaged | ||
license-state-changed |
license-state-changed | ||
license-validation-failed |
license-validation-failed | ||
license-validation-ok |
license-validation-ok | ||
license-volume-violation |
license-volume-violation | ||
new-license |
new-license | ||
override-job-coordinated-shutdown |
override-job-coordinated-shutdown | ||
override-server-coordinated-shutdown |
override-server-coordinated-shutdown | ||
override-worker-coordinated-shutdown |
override-worker-coordinated-shutdown | ||
register-job-thread-dependency |
register-job-thread-dependency | ||
register-server-thread-dependency |
register-server-thread-dependency | ||
register-worker-thread-dependency |
register-worker-thread-dependency | ||
run-job-failure |
run-job-failure | ||
server-logs |
server-logs | ||
server-metrics-batch |
server-metrics-batch | ||
server-started |
server-started | ||
server-starting |
server-starting | ||
server-stopping |
server-stopping | ||
server-thread-state |
server-thread-state | ||
server-worker-comms-error |
server-worker-comms-error | ||
shutdown-jobs |
shutdown-jobs | ||
shutdown-worker |
shutdown-worker | ||
system-shutdown |
system-shutdown | ||
update-upstream-sync-for-job |
update-upstream-sync-for-job | ||
update-upstream-sync-for-worker |
update-upstream-sync-for-worker | ||
update-variable |
update-variable | ||
user-alert |
user-alert | ||
user-generated |
user-generated | ||
trigger-response |
trigger-response | ||
trigger-proxy-response |
trigger-proxy-response | ||
workflow-step-terminal |
workflow-step-terminal | ||
workflow-step-terminal-ack |
workflow-step-terminal-ack | ||
user-notification |
user-notification | ||
worker-command-for-job |
worker-command-for-job | ||
worker-auth-lease-ack |
worker-auth-lease-ack | ||
worker-connected |
worker-connected | ||
worker-created |
worker-created | ||
worker-debug-heart-beat |
worker-debug-heart-beat | ||
worker-error |
worker-error | ||
worker-first-seen |
worker-first-seen | ||
worker-heart-beat |
worker-heart-beat | ||
worker-logs |
worker-logs | ||
worker-metrics-batch |
worker-metrics-batch | ||
worker-offline |
worker-offline | ||
worker-requests-auth-lease |
worker-requests-auth-lease | ||
worker-server-comms-error |
worker-server-comms-error | ||
worker-settings |
worker-settings | ||
worker-shutdown |
worker-shutdown | ||
worker-shutting-down |
worker-shutting-down | ||
worker-started |
worker-started | ||
worker-state-uuid |
worker-state-uuid | ||
worker-stopping |
worker-stopping | ||
worker-suspicious-silence |
worker-suspicious-silence | ||
worker-system-information |
worker-system-information | ||
worker-thread-state |
worker-thread-state | ||
worker-updated |
worker-updated | ||
worker-modified |
worker-modified | ||
worker-removed |
worker-removed | ||
context-changed |
context-changed | ||
rerender-deployment |
rerender-deployment | ||
job-killed |
job-killed | ||
message-serviced |
message-serviced | ||
failed-to-service-message |
failed-to-service-message | ||
worker-wants-initial-settings |
worker-wants-initial-settings | ||
worker-wants-initial-settings-reply |
worker-wants-initial-settings-reply | ||
worker-wants-deployed-jobs |
worker-wants-deployed-jobs | ||
worker-wants-deployed-jobs-reply |
worker-wants-deployed-jobs-reply | ||
worker-wants-job-configuration |
worker-wants-job-configuration | ||
worker-wants-job-configuration-reply |
worker-wants-job-configuration-reply | ||
job-wants-dslir-key |
job-wants-dslir-key | ||
job-wants-dslir-key-reply |
job-wants-dslir-key-reply | ||
worker-wants-dslir-key |
worker-wants-dslir-key | ||
worker-wants-dslir-key-reply |
worker-wants-dslir-key-reply | ||
job-wants-variables |
job-wants-variables | ||
job-wants-variables-reply |
job-wants-variables-reply | ||
job-wants-credentials |
job-wants-credentials | ||
job-wants-credentials-reply |
job-wants-credentials-reply | ||
job-wants-credentials-error |
job-wants-credentials-error | ||
job-wants-secret-variables-reply |
job-wants-secret-variables-reply | ||
job-credentials-invalidated |
job-credentials-invalidated | ||
aggregator-health |
aggregator-health | ||
worker-aggregator-health |
worker-aggregator-health | ||
worker-verification-token |
worker-verification-token | ||
worker-requests-verification-token |
worker-requests-verification-token | ||
runtime-artifact-update |
runtime-artifact-update | ||
runtime-artifact-update-ack |
runtime-artifact-update-ack | ||
runtime-artifact-clear |
runtime-artifact-clear | ||
runtime-artifact-clear-ack |
runtime-artifact-clear-ack | ||
runtime-artifact-fetch |
runtime-artifact-fetch | ||
runtime-artifact-fetch-reply |
runtime-artifact-fetch-reply | ||
runtime-artifact-fetch-error |
runtime-artifact-fetch-error |