Log Files

Log Files (`files`)

Monitor one or more log files for new lines.

File binary json raw

input:
  files:
    path: ""

JSON

{
  "input": {
    "files": {
      "path": ""
    }
  }
}

Field	Type	Description
`encoding`	`encoding` (`string`)	the text encoding for the monitored files. Default: `UTF-8` Examples: `UTF-8`
`stop-reading-after` ✓	`boolean` (`bool`)	do not wait for new files. Default: `false`
`ignore-line-breaks` ✓	`boolean` (`bool`)	consume each file as one event. Default: `false`
`payload-mode` ✓	`Payload Mode`	Select how file content should be interpreted (`auto`, `json`, `raw`, `binary`). Default: `auto` Allowed values: `auto`, `json`, `raw`, `binary`
`remove-after`	`number` (`integer`)	remove files that are done reading after this period (seconds). Examples: `42`, `1.2e-10`
`oldest-first` ✓	`boolean` (`bool`)	default is youngest first. Default: `false`
`file-basename` ✓	`boolean` (`bool`)	the `file-path-field` path will be the basename (e.g. ‘/path/frodo.txt’ becomes just ‘frodo’). Default: `false`
`start-at-beginning` ✓	`boolean` (`bool`)	ignore stored checkpoints and re-read all specified files. Default: `false`
`json` ✓	`boolean` (`bool`)	assume lines are already JSON. Default: `false`

Field	Type	Required	Description
`ignore-older-than`	`number` (`integer`)		ignore files older than this (epoch time in seconds). Examples: `42`, `1.2e-10`

Field	Type	Required	Description
`path`	`path` (`string`)	✅	a path with wildcards identifying files to be read. Examples: `/path/to/file`, `c:\users\joe\data\file.txt`
`include`	`path[]` (`string`)		any other paths to be included. Examples: `/path/to/file`, `c:\users\joe\data\file.txt`
`exclude`	`path[]` (`string`)		paths to be excluded. Examples: `/path/to/file`, `c:\users\joe\data\file.txt`

Field	Type	Required	Description
`file-path-field`	`field` (`string`)		fill this field with the file we are currently reading from. Examples: `data_field`

Field	Type	Description
`glob-minimum-cooldown` ✓	`number` (`integer`)	pause after scanning for new files (in milliseconds). Default: `300` Examples: `42`, `1.2e-10`
`max-read-size` ✓	`number` (`integer`)	will not attempt to read files larger than this. Default: `100MB` Examples: `42`, `1.2e-10`
`max-line-size` ✓	`number` (`integer`)	will not attempt to read lines longer than this. Default: `1MB` Examples: `42`, `1.2e-10`

Field	Type	Required	Description
`fingerprinting`	`Fingerprint Strategy`		File fingerprinting strategy. Default: `dev-inode` Allowed values: `checksum`, `first-line-checksum`, `dev-inode`

Option	Name	Type	Description
`checksum`	Checksum	`object`	Fingerprint strategy is set to a checksum of the start of the file determined by the bytes that are skipped (`ignored-header-bytes`) and then the fingerprint size in bytes.
`first-line-checksum`	First Line Checksum	`object`	Fingerprint strategy that uses the first line of the file to detect if a file has changed.
`dev-inode`	Dev Inode	`symbol`	Fingerprint strategy is set to the inode of the file causing the file monitor to detect new files when a file is recreated. Allowed values: `fingerprinter-dev-inode`

Field	Type	Required	Description
`fingerprint-bytes` ✓	`number` (`integer`)		The number of bytes to use for the fingerprint, starts after `ignore-header-bytes` if specified. Examples: `42`, `1.2e-10`
`ignored-header-bytes` ✓	`number` (`integer`)		The number of bytes from the start of the file to ignore before starting the fingerprint. Examples: `42`, `1.2e-10`

Field	Type	Required	Description
`max-line-length` ✓	`number` (`integer`)		The maximum number of bytes to read from the first line of the file to calculate the fingerprint. Examples: `42`, `1.2e-10`

Value	Name	Description
`fingerprinter-dev-inode`	fingerprinter-dev-inode	Fingerprint strategy is set to the inode of the file causing the file monitor to detect new files when a file is recreated