Azure Blob

Azure Blob (azure-blob)

Send data to a Microsoft Azure Storage blob (Block Storage).

Block Store Cloud Enterprise edition binary json raw

Minimal example

output:
  azure-blob:
    blob-destination:
      name: ~
    container-name: ~
    storage-account: ~
    storage-master-key: ~

JSON

{
  "output": {
    "azure-blob": {
      "blob-destination": {
        "name": null
      },
      "container-name": null,
      "storage-account": null,
      "storage-master-key": null
    }
  }
}

Contents

• Minimal example

• Authentication

• Behavior

• Location

• Object Properties

• Processing

• Reliability

• Schema

A ✓ marks a field that accepts a context variable such as {{ VARIABLE }}.

Authentication

Authentication

Field	Type	Required	Description
storage-account	string	✅	The Storage Account Name to be used (credential).
storage-master-key	string	✅	The Storage Master Key to be used (credential).

Behavior

Behavior

Field	Type	Required	Description
mode	Mode		Either writing or deleting a blob. Allowed values: put, delete

Location

Location

Field	Type	Required	Description
container-name	string	✅	The storage service container for created blobs.
blob-destination	Blob Destination	✅	Either a literal blob name or the name of the field to read it from. Allowed values: name, field

Object Properties

Object Properties

Field	Type	Required	Description
disable-blob-name-guid ✓	boolean (bool)		Do not append a GUID to the output blob name. Default: false
guid-prefix	string		GUID Prefix, will be prepended to the GUID, the default value is “/”.
guid-suffix	string		GUID Suffix, will be appended to the GUID if specified.
content-type	string		Set the created blob to the following content-type. Default: text/plain

Processing

Processing

Field	Type	Required	Description
batch	Batch		Batching output events.
input-field	field (string)		Send only the content of the specified field to the blob. Examples: data_field
preprocessors	Preprocessors[]		Preprocessors (process data before making it available for upload) these processors will be run in the order they are specified. Allowed values: gzip, parquet, base64

Reliability

Reliability

Field	Type	Required	Description
retry	Retry		How to retry operation if it fails.
track-schema ✓	boolean (bool)		Check the schema of the written data and update __SCHEMA_NUMBER (written data must be JSON). Default: false

Schema

• Blob Destination Options
• Batch Fields
• Retry - Backoff Fields
• Retry Fields
• Mode Options
• Batch - Mode Options
• Preprocessors Options
• Retry - Backoff - Strategy Options

Blob Destination Options

Option	Name	Type	Description
name	Name	string	The name for the output blob, this will be suffixed by a GUID by default.
field	Field	string	A field containing the value for output_blob.

Batch Fields

Field	Type	Required	Description
fixed-size ✓	number (integer)		maximum number of events in an output batch. Examples: 42, 1.2e-10
max-bytes ✓	number (integer)		Close the batch before adding an event that would make the serialized request payload exceed this many bytes. Examples: 42, 1.2e-10
mode	Mode	✅	If ‘document’ send on end of document generated by input. If ‘fixed’, use fixed_size. Allowed values: fixed, document
timeout	time-interval (string)	✅	interval after which the batch is sent, to keep throughput going (default 100ms). Default: 100ms Examples: 500ms, 2h
header	multiline-text (string)		put a header line before the batch.
footer	multiline-text (string)		put a header line after the last line of the batch.
use-document-marker ✓	boolean (bool)		Enrich the job metadata with a document marker (for document handling in batch mode). Default: false
wrap-as-json ✓	boolean (bool)		Format the output batch as a JSON array. Default: false

Retry - Backoff Fields

Field	Type	Required	Description
strategy	Strategy		Backoff strategy to use (default exponential). Allowed values: exponential, linear, fixed
base	string		Base delay before retrying (e.g. “200ms”).
max	string		Maximum delay between retries.
jitter ✓	boolean (bool)		Whether to add jitter to retry delays. Default: false

Retry Fields

Field	Type	Required	Description
max-attempts	number (integer)		Maximum attempts before giving up. Examples: 42, 1.2e-10
forever ✓	boolean (bool)		Retry indefinitely until cancelled. Default: false
backoff	Backoff		Backoff strategy configuration.

Mode Options

Value	Aliases	Name	Description
put		put	Put Objects
delete		delete	Delete Objects

Batch - Mode Options

Value	Aliases	Name	Description
fixed		fixed
document		document

Preprocessors Options

Value	Aliases	Name	Description
gzip		gzip	Gzip the output data
parquet		parquet	Extract the received data as JSON rows from a parquet file
base64		base64	Decode base64 as binary

Retry - Backoff - Strategy Options

Value	Aliases	Name	Description
exponential		exponential
linear		linear
fixed		fixed