S3

S3 (s3)

Write events to an S3 bucket.

Block Store Cloud binary json raw

Minimal example

output:
  s3:
    bucket-name: ~
    object-name:
      name: ~

JSON

{
  "output": {
    "s3": {
      "bucket-name": null,
      "object-name": {
        "name": null
      }
    }
  }
}

Contents

• Minimal example

• Authentication

• Behavior

• Location

• Object Properties

• Processing

• Reliability

• Schema

Authentication

Authentication

Field	Type	Required	Description
access-key	string		Access Key ID.
secret-key	string		Secret Key ID.
security-token	string		Security Token.
session-token	string		Session Token.
role-arn	string		A Role ARN for assuming role using above credentials.

Behavior

Behavior

Field	Type	Required	Description
mode	Mode		Put or delete the object? Default is Put. Allowed values: put, delete

Location

Location

Field	Type	Required	Description
bucket-name	string	✅	Bucket Name.
object-name	Object Name	✅	Name of object, either as text or from a field. Allowed values: name, field
region	string		S3 Region.
endpoint	string		S3 Endpoint, if needed.

Object Properties

Object Properties

Field	Type	Required	Description
disable-object-name-guid ✓	boolean (bool)		Disable the GUID prefix if you want the object name to be treated literally (this is disabled for deletes). Default: false
guid-prefix	string		GUID Prefix, will be prepended to the GUID, the default value is ”/”.
guid-suffix	string		GUID Suffix, will be appended to the GUID if specified.

Processing

Processing

Field	Type	Required	Description
batch	Batch		Batching input events together.
input-field	field (string)		Use the specified field as the content for the object. Examples: data_field
preprocessors	Output Preprocessor[]		Preprocessors (process data before making it available for upload) these processors will be run in the order they are specified. Allowed values: gzip, parquet, base64

Reliability

Reliability

Field	Type	Required	Description
retry	Retry		How to retry failed requests.
track-schema ✓	boolean (bool)		Check the schema of the written data and update __SCHEMA_NUMBER (written data must be JSON). Default: false

Schema

• Object Name Options
• Batch Fields
• Retry Fields
• Mode Options
• Batch - Mode Options
• Output Preprocessor Options

Object Name Options

Option	Name	Type	Description
name	Name	string	Object Name.
field	Field	string	Field containing the Object Name.

Batch Fields

Field	Type	Required	Description
fixed-size ✓	number (integer)		maximum number of events in an output batch. Examples: 42, 1.2e-10
mode	Mode	✅	If ‘document’ send on end of document generated by input. If ‘fixed’, use fixed_size. Allowed values: fixed, document
timeout	time-interval (string)	✅	interval after which the batch is sent, to keep throughput going (default 100ms). Default: 100ms Examples: 500ms, 2h
header	multiline-text (string)		put a header line before the batch.
footer	multiline-text (string)		put a header line after the last line of the batch.
use-document-marker ✓	boolean (bool)		Enrich the job metadata with a document marker (for document handling in batch mode). Default: false
wrap-as-json ✓	boolean (bool)		Format the output batch as a JSON array. Default: false

Retry Fields

Field	Type	Required	Description
timeout	time-interval (string)	✅	timeout (e.g. 500ms, 2s etc. - default is 30). Examples: 500ms, 2h
retries	number (integer)		number of retries. Examples: 42, 1.2e-10

Mode Options

Value	Name	Description
put	put	Put Objects
delete	delete	Delete Objects

Batch - Mode Options

Value	Name	Description
fixed	fixed	Fixed
document	document	Document

Output Preprocessor Options

Value	Name	Description
gzip	gzip	Gzip the output data
parquet	parquet	Extract the received data as JSON rows from a parquet file
base64	base64	Decode base64 as binary