Journald Source

The Vector journald source ingests data through Systemd's Journald utility and outputs log events.

Requirements

Configuration

vector.toml
[sources.my_source_id]
type = "journald" # required
current_boot_only = true # optional, default
units = [] # optional, default
  • intoptional

    batch_size

    The systemd journal is read in batches, and a checkpoint is set at the end of each batch. This option limits the size of the batch.

    See Checkpointing for more info.

    • Default: 16
    • View examples
  • boolcommonoptional

    current_boot_only

    Include only entries from the current boot.

    • Default: true
    • View examples
  • stringoptional

    data_dir

    The directory used to persist the journal checkpoint position. By default, the global data_dir is used. Please make sure the Vector project has write permissions to this dir.

    See Checkpointing for more info.

    • No default
    • View examples
  • stringoptional

    journalctl_path

    The full path of the journalctl executable. If not set, Vector will search the path for journalctl.

    See Communication strategy for more info.

    • Default: "journalctl"
    • View examples
  • [string]commonoptional

    units

    The list of units names to monitor. If empty or not present, all units are accepted. Unit names lacking a "." will have ".service" appended to make them a valid service unit name.

    • Default: []
    • View examples

Fields

example log event
{
// ...
"_SYSTEMD_UNIT": "ntpd.service",
"host": "my.host.com",
"message": "Started GET / for 127.0.0.1 at 2012-03-10 14:28:14 +0100",
"timestamp": "2019-11-01T21:15:47+00:00"
// ...
}
  • *optional

    [record-key]

    Additional Journald fields are passed through as log fields.

    • No default
    • View examples
  • stringcommonrequired

    host

    The value of the journald _HOSTNAME field.

    • No default
    • View examples
  • stringcommonrequired

    message

    The value of the journald MESSAGE field.

    • No default
    • View examples
  • timestampcommonrequired

    timestamp

    The value of the journald _SOURCE_REALTIME_TIMESTAMP field.

    • No default
    • View examples

Examples

Given the following input:

Example input
reply from 192.168.1.2: offset -0.001791 delay 0.000176, next query 1500s

A log event will be output with the following structure:

Example log event
{
"timestamp": <2019-07-26T20:30:27.000443Z>,
"message": "reply from 192.168.1.2: offset -0.001791 delay 0.000176, next query 1500s",
"host": "lorien.example.com",
"__REALTIME_TIMESTAMP": "1564173027000443",
"__MONOTONIC_TIMESTAMP": "98694000446",
"_BOOT_ID": "124c781146e841ae8d9b4590df8b9231",
"SYSLOG_FACILITY": "3",
"_UID": "0",
"_GID": "0",
"_CAP_EFFECTIVE": "3fffffffff",
"_MACHINE_ID": "c36e9ea52800a19d214cb71b53263a28",
"PRIORITY": "6",
"_TRANSPORT": "stdout",
"_STREAM_ID": "92c79f4b45c4457490ebdefece29995e",
"SYSLOG_IDENTIFIER": "ntpd",
"_PID": "2156",
"_COMM": "ntpd",
"_EXE": "/usr/sbin/ntpd",
"_CMDLINE": "ntpd: [priv]",
"_SYSTEMD_CGROUP": "/system.slice/ntpd.service",
"_SYSTEMD_UNIT": "ntpd.service",
"_SYSTEMD_SLICE": "system.slice",
"_SYSTEMD_INVOCATION_ID": "496ad5cd046d48e29f37f559a6d176f8"
}

How It Works

Checkpointing

Vector checkpoints the journal position after every batch read. The size of the batch is controlled via the batch_size option. Checkpointing ensures that Vector resumes where it left off if restarted, preventing data from being read twice. The checkpoint positions are stored in the data directory which is specified via the global [data_dir](#data_dir) option but can be overridden via the data_dir option in the journald source directly.

Communication strategy

To ensure the journald source works across all platforms, Vector interacts with the Systemd journal via the journalctl command. This is accomplished by spawning a subprocess that Vector diligently interacts with. If the journalctl command is not in the environment path you can specify the exact location via the journalctl_path option. For more information on this communication strategy please see issue #1473.

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.

Non-ASCII messages

When journald has stored a message that is not strict ASCII, journalctl will output it in an alternate format to prevent data loss. Vector handles this alternate format by translating such messages into UTF-8 in "lossy" mode, where characters that are not valid UTF-8 are replaced with the Unicode replacement character, .

User permissions

If you run Vector from a non-root user, you need to add that user to the systemd-journal group.

For example, if the user is named vector, it can be done by running

usermod -aG systemd-journal vector