Kafka

Kafka (`kafka`)

Publish events to Kafka/Redpanda clusters.

Messaging binary json raw

output:
  kafka:
    bootstrap-servers: ""

JSON

{
  "output": {
    "kafka": {
      "bootstrap-servers": ""
    }
  }
}

Field	Type	Required	Description
`global-options` ✓	`map` (`string`)		Additional librdkafka producer configuration.
`topic-options` ✓	`map` (`string`)		Topic level overrides forwarded to librdkafka.

Field	Type	Required	Description
`batch`	`Batch`		Logical batching hints shared with other outputs.

Field	Type	Required	Description
`bootstrap-servers`	`hostname` (`string`)	✅	Comma separated bootstrap servers. Examples: `example.com`, `localhost`
`client-id`	`string`		Optional producer client identifier.

Field	Type	Description
`partitioner`	`Kafka Partitioner`	Partitioning strategy for the producer. Allowed values: `round-robin`, `hash-key`, `explicit`
`key-field`	`field` (`string`)	Event field containing message key (used for partitioning). Examples: `data_field`
`key-pointer`	`json-pointer` (`string`)	JSON pointer to message key.
`key-expression`	`string`	Expression used to compute message key.
`partition-field`	`field` (`string`)	Event field providing explicit partition. Examples: `data_field`
`partition-pointer`	`json-pointer` (`string`)	JSON pointer providing explicit partition.
`partition-value` ✓	`string`	Static partition value when using explicit partitioner.

Field	Type	Description
`topic`	`string`	Static topic name.
`topic-field`	`field` (`string`)	Event field containing the target topic. Examples: `data_field`
`topic-pointer`	`json-pointer` (`string`)	JSON pointer resolving to target topic.
`input-field`	`field` (`string`)	Optional event field to treat as payload instead of entire event. Examples: `data_field`
`headers` ✓	`map` (`string`)	Message headers to attach to each record.
`payload`	`Kafka Payload Format`	Encoding strategy for message payloads. Allowed values: `json`, `raw`, `avro-bridge`
`avro`	`Schema Registry`	Schema registry configuration required when payload=avro-bridge.

Field	Type	Description
`linger-ms`	`number` (`integer`)	Linger time before flushing a batch (ms). Examples: `42`, `1.2e-10`
`message-timeout-ms`	`number` (`integer`)	Timeout waiting for delivery reports (ms). Examples: `42`, `1.2e-10`
`compression`	`Kafka Compression`	Compression codec used by the producer. Allowed values: `none`, `gzip`, `snappy`, `lz4`, `zstd`

Field	Type	Description
`retry`	`Retry`	Retry policy for failed deliveries.
`acks`	`Kafka Acknowledgements`	Acknowledgement requirement. Allowed values: `none`, `leader`, `all`
`idempotent` ✓	`boolean` (`bool`)	Enable Kafka idempotent producer mode. Default: `false`
`max-in-flight`	`number` (`integer`)	Maximum in-flight requests per connection (relevant for idempotency). Examples: `42`, `1.2e-10`

Field	Type	Required	Description
`auth`	`Kafka Authentication`		Authentication and TLS configuration.

Field	Type	Required	Description
`fixed-size` ✓	`number` (`integer`)		maximum number of events in an output batch. Examples: `42`, `1.2e-10`
`mode`	`Mode`	✅	If ‘document’ send on end of document generated by input. If ‘fixed’, use `fixed_size`. Allowed values: `fixed`, `document`
`timeout`	`time-interval` (`string`)	✅	interval after which the batch is sent, to keep throughput going (default 100ms). Default: `100ms` Examples: `500ms`, `2h`
`header`	`multiline-text` (`string`)		put a header line before the batch.
`footer`	`multiline-text` (`string`)		put a header line after the last line of the batch.
`use-document-marker` ✓	`boolean` (`bool`)		Enrich the job metadata with a document marker (for document handling in batch mode). Default: `false`
`wrap-as-json` ✓	`boolean` (`bool`)		Format the output batch as a JSON array. Default: `false`

Field	Type	Required	Description
`url`	`url` (`string`)	✅	Examples: `https://example.com/path`
`username`	`string`
`password`	`string`
`access-token`	`string`
`subject-suffix`	`string`		Optional suffix appended to subject names (e.g. `-value`).
`use-latest` ✓	`boolean` (`bool`)		When true, fetch the latest compatible schema instead of an explicit version. Default: `false`

Field	Type	Required	Description
`timeout`	`time-interval` (`string`)	✅	timeout (e.g. 500ms, 2s etc. - default is 30). Examples: `500ms`, `2h`
`retries`	`number` (`integer`)		number of retries. Examples: `42`, `1.2e-10`

Field	Type	Required	Description
`mechanism`	`Kafka SASL Mechanism`	✅	Allowed values: `plain`, `scram-sha256`, `scram-sha512`, `o-auth-bearer`
`username` ✓	`string`
`password` ✓	`string`
`token` ✓	`string`
`token-command` ✓	`string`
`extensions` ✓	`map` (`string`)

Field	Type	Description
`ca-certificate`	`string`
`client-certificate`	`string`
`client-key`	`string`
`insecure-skip-verify` ✓	`boolean` (`bool`)	Default: `false`