Log Files

Log Files (files)

Monitor one or more log files for new lines.

File binary json raw

Minimal example

input:
  files:
    path: ""

JSON

{
  "input": {
    "files": {
      "path": ""
    }
  }
}

Contents

• Minimal example

• Behavior

• Filtering

• Location

• Object Properties

• Performance

• Reliability

• Schema

A ✓ marks a field that accepts a context variable such as {{ VARIABLE }}.

Behavior

Behavior

Field	Type	Required	Description
encoding	encoding (string)		the text encoding for the monitored files. Default: UTF-8 Examples: UTF-8
stop-reading-after ✓	boolean (bool)		do not wait for new files. Default: false
ignore-line-breaks ✓	boolean (bool)		consume each file as one event. Default: false
payload-mode ✓	Payload Mode		Select how file content should be interpreted (auto, json, raw, binary). Raw text is emitted in the _raw event field. Prefer this setting over the legacy json flag; do not configure both. Default: auto Allowed values: auto, json, raw, binary
remove-after	number (integer)		remove files that are done reading after this period (seconds). Examples: 42, 1.2e-10
oldest-first ✓	boolean (bool)		default is youngest first. Default: false
file-basename ✓	boolean (bool)		the file-path-field path will be the basename (e.g. ‘/path/frodo.txt’ becomes just ‘frodo’). Default: false
start-at-beginning ✓	boolean (bool)		ignore stored checkpoints and re-read all specified files. Default: false
json ✓	boolean (bool)		assume lines are already JSON. Default: false

Filtering

Filtering

Field	Type	Required	Description
ignore-older-than	number (integer)		ignore files older than this (epoch time in seconds). Examples: 42, 1.2e-10

Location

Location

Field	Type	Required	Description
path	path (string)	✅	a path with wildcards identifying files to be read. Examples: /path/to/file, c:\users\joe\data\file.txt
include	path[] (string)		any other paths to be included. Examples: /path/to/file, c:\users\joe\data\file.txt
exclude	path[] (string)		paths to be excluded. Examples: /path/to/file, c:\users\joe\data\file.txt

Object Properties

Object Properties

Field	Type	Required	Description
file-path-field	field (string)		fill this field with the file we are currently reading from. Examples: data_field

Performance

Performance

Field	Type	Required	Description
glob-minimum-cooldown ✓	number (integer)		pause after scanning for new files (in milliseconds). Default: 300 Examples: 42, 1.2e-10
max-read-size ✓	number (integer)		will not attempt to read files larger than this. Default: 104857600 Examples: 42, 1.2e-10
max-line-size ✓	number (integer)		will not attempt to read lines longer than this. Default: 1024 Examples: 42, 1.2e-10

Reliability

Reliability

Field	Type	Required	Description
fingerprinting	Fingerprinting		File fingerprinting strategy. Default: dev-inode Allowed values: checksum, first-line-checksum, dev-inode

Schema

• Fingerprinting Options
• Fingerprinting - Checksum Fields
• Fingerprinting - First Line Checksum Fields
• Payload Mode Options
• Fingerprinting - Dev Inode Options

Fingerprinting Options

Option	Name	Type	Description
checksum	Checksum	object	Fingerprint strategy is set to a checksum of the start of the file determined by the bytes that are skipped (ignored-header-bytes) and then the fingerprint size in bytes.
first-line-checksum	First Line Checksum	object	Fingerprint strategy that uses the first line of the file to detect if a file has changed.
dev-inode	Dev Inode	symbol	Fingerprint strategy is set to the inode of the file causing the file monitor to detect new files when a file is recreated. Allowed values: fingerprinter-dev-inode

Fingerprinting - Checksum Fields

Field	Type	Required	Description
fingerprint-bytes ✓	number (integer)		The number of bytes to use for the fingerprint, starts after ignore-header-bytes if specified. Examples: 42, 1.2e-10
ignored-header-bytes ✓	number (integer)		The number of bytes from the start of the file to ignore before starting the fingerprint. Examples: 42, 1.2e-10

Fingerprinting - First Line Checksum Fields

Field	Type	Required	Description
max-line-length ✓	number (integer)		The maximum number of bytes to read from the first line of the file to calculate the fingerprint. Examples: 42, 1.2e-10

Payload Mode Options

Value	Aliases	Name	Description
auto		auto
json		json
raw		raw
binary		binary

Fingerprinting - Dev Inode Options

Value	Aliases	Name	Description
fingerprinter-dev-inode		fingerprinter-dev-inode	Fingerprint strategy is set to the inode of the file causing the file monitor to detect new files when a file is recreated