Journald Source

The Vector journald source ingests data through log records from journald and outputs log events.

Configuration

  • Common
  • Advanced
vector.toml
[sources.my_source_id]
# REQUIRED
type = "journald" # must be: "journald"
# OPTIONAL
current_boot_only = true # default
units = [] # default

Requirements

Options

5 items
intoptional

batch_size

The systemd journal is read in batches, and a checkpoint is set at the end of each batch. This option limits the size of the batch. See Checkpointing for more info.

Default: 16
View examples
boolcommonoptional

current_boot_only

Include only entries from the current boot.

Default: true
View examples
stringoptional

data_dir

The directory used to persist the journal checkpoint position. By default, the global data_dir is used. Please make sure the Vector project has write permissions to this dir. See Checkpointing for more info.

No default
View examples
stringoptional

journalctl_path

The full path of the journalctl executable. If not set, Vector will search the path for journalctl. See Communication strategy for more info.

Default: "journalctl"
View examples
[string]commonoptional

units

The list of units names to monitor. If empty or not present, all units are accepted. Unit names lacking a "." will have ".service" appended to make them a valid service unit name.

Default: []
View examples

Output

The journald source ingests data through log records from journald and outputs log events. For example:

Given the following input:

__REALTIME_TIMESTAMP=1564173027000443
__MONOTONIC_TIMESTAMP=98694000446
_BOOT_ID=124c781146e841ae8d9b4590df8b9231
SYSLOG_FACILITY=3
_UID=0
_GID=0
_CAP_EFFECTIVE=3fffffffff
_MACHINE_ID=c36e9ea52800a19d214cb71b53263a28
_HOSTNAME=lorien.example.com
PRIORITY=6
_TRANSPORT=stdout
_STREAM_ID=92c79f4b45c4457490ebdefece29995e
SYSLOG_IDENTIFIER=ntpd
_PID=2156
_COMM=ntpd
_EXE=/usr/sbin/ntpd
_CMDLINE=ntpd: [priv]
_SYSTEMD_CGROUP=/system.slice/ntpd.service
_SYSTEMD_UNIT=ntpd.service
_SYSTEMD_SLICE=system.slice
_SYSTEMD_INVOCATION_ID=496ad5cd046d48e29f37f559a6d176f8
MESSAGE=reply from 192.168.1.2: offset -0.001791 delay 0.000176, next query 1500s

A log event will be output with the following structure:

{
"timestamp": <2019-07-26T20:30:27.000443Z>,
"message": "reply from 192.168.1.2: offset -0.001791 delay 0.000176, next query 1500s",
"host": "lorien.example.com",
"__REALTIME_TIMESTAMP": "1564173027000443",
"__MONOTONIC_TIMESTAMP": "98694000446",
"_BOOT_ID": "124c781146e841ae8d9b4590df8b9231",
"SYSLOG_FACILITY": "3",
"_UID": "0",
"_GID": "0",
"_CAP_EFFECTIVE": "3fffffffff",
"_MACHINE_ID": "c36e9ea52800a19d214cb71b53263a28",
"PRIORITY": "6",
"_TRANSPORT": "stdout",
"_STREAM_ID": "92c79f4b45c4457490ebdefece29995e",
"SYSLOG_IDENTIFIER": "ntpd",
"_PID": "2156",
"_COMM": "ntpd",
"_EXE": "/usr/sbin/ntpd",
"_CMDLINE": "ntpd: [priv]",
"_SYSTEMD_CGROUP": "/system.slice/ntpd.service",
"_SYSTEMD_UNIT": "ntpd.service",
"_SYSTEMD_SLICE": "system.slice",
"_SYSTEMD_INVOCATION_ID": "496ad5cd046d48e29f37f559a6d176f8"
}

More detail on the output schema is below.

4 items
*optional

[record-key]

Additional Journald fields are passed through as log fields.

No default
View examples
stringcommonrequired

host

The value of the journald _HOSTNAME field.

No default
View examples
stringcommonrequired

message

The value of the journald MESSAGE field.

No default
View examples
timestampcommonrequired

timestamp

The value of the journald _SOURCE_REALTIME_TIMESTAMP field.

No default
View examples

How It Works

Checkpointing

Vector checkpoints the journal position after every batch read. The size of the batch is controlled via the batch_size option. Checkpointing ensures that Vector resumes where it left off if restarted, preventing data from being read twice. The checkpoint positions are stored in the data directory which is specified via the global [data_dir](#data_dir) option but can be overridden via the data_dir option in the journald source directly.

Communication strategy

To ensure the journald source works across all platforms, Vector interacts with the Systemd journal via the journalctl command. This is accomplished by spawning a subprocess that Vector diligently interacts with. If the journalctl command is not in the environment path you can specify the exact location via the journalctl_path option. For more information on this communication strategy please see issue #1473.

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.

User permissions

If you run Vector from a non-root user, you need to add that user to the systemd-journal group.

For example, if the user is named vector, it can be done by running

usermod -aG systemd-journal vector