Vector Source

The Vector vector source receives data from another upstream Vector instance using the Vector sink.

Requirements

Configuration

[sources.my_source_id]
type = "vector" # required
acknowledgements = false # optional, default
address = "0.0.0.0:9000" # required
  • commonoptionalbool

    acknowledgements

    Controls if the source will wait for destination sinks to deliver the events before acknowledging receipt.

    • WARNING: Disabling this option may lead to loss of data, as destination sinks may reject events after the source acknowledges their successful receipt.
    • Default: false
  • commonrequiredstring

    address

    The TCP address to listen for connections on, or `systemd#N to use the Nth socket passed by systemd socket activation. If an address is used it must include a port.

    • Syntax: literal
  • optionaltable

    keepalive

    Configures the TCP keepalive behavior for the connection to the source.

    • optionaluint

      time_secs

      The time a connection needs to be idle before sending TCP keepalive probes.

    • optionaluint

      shutdown_timeout_secs

      The timeout before a connection is forcefully closed during shutdown.

      • Default: 30 (seconds)
    • optionaltable

      tls

      Configures the TLS options for incoming connections.

      • optionalstring

        ca_file

        Absolute path to an additional CA certificate file, in DER or PEM format (X.509), or an in-line CA certificate in PEM format.

        • Syntax: literal
      • optionalstring

        crt_file

        Absolute path to a certificate file used to identify this server, in DER or PEM format (X.509) or PKCS#12, or an in-line certificate in PEM format. If this is set, and is not a PKCS#12 archive, key_file must also be set. This is required if enabled is set to true.

        • Syntax: literal
      • optionalbool

        enabled

        Require TLS for incoming connections. If this is set, an identity certificate is also required.

        • Default: false
      • optionalstring

        key_file

        Absolute path to a private key file used to identify this server, in DER or PEM format (PKCS#8), or an in-line private key in PEM format.

        • Syntax: literal
      • optionalstring

        key_pass

        Pass phrase used to unlock the encrypted key file. This has no effect unless key_file is set.

        • Syntax: literal
      • optionalbool

        verify_certificate

        If true, Vector will require a TLS certificate from the connecting host and terminate the connection if the certificate is not valid. If false (the default), Vector will not request a certificate from the client.

        • Default: false

    Output

    Log Fields

    This component outputs log events with the following fields:

    {
    "*" : null
    }
    • commonrequired*

      *

      Vector transparently forwards data from another upstream Vector instance. The vector source will not modify or add fields.

      Metric Events

      This component outputs the following metric events:

      • counter

        counter

        A single value that can only be incremented or reset to zero value, it cannot be decremented. This metric includes the following tags:

        • * - Any tags present on the metric.

      • distribution

        distribution

        A distribution represents a distribution of sampled values. It is used with services that support global histograms and summaries. This metric includes the following tags:

        • * - Any tags present on the metric.

      • gauge

        gauge

        A gauge represents a point-in-time value that can increase and decrease. Vector's internal gauge type represents changes to that value. Gauges should be used to track fluctuations in values, like current memory or CPU usage. This metric includes the following tags:

        • * - Any tags present on the metric.

      • gauge

        histogram

        Also called a "timer". A histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. It also provides a sum of all observed values. This metric includes the following tags:

        • * - Any tags present on the metric.

      • gauge

        set

        A set represents an array of unique values. This metric includes the following tags:

        • * - Any tags present on the metric.

      Telemetry

      This component provides the following metrics that can be retrieved through the internal_metrics source. See the metrics section in the monitoring page for more info.

      • counter

        events_in_total

        The number of events accepted by this component either from tagged origin like file and uri, or cumulatively from other origins. This metric includes the following tags:

        • component_kind - The Vector component kind.

        • component_name - The Vector component ID.

        • component_type - The Vector component type.

        • container_name - The name of the container from which the event originates.

        • file - The file from which the event originates.

        • instance - The Vector instance identified by host and port.

        • job - The name of the job producing Vector metrics.

        • mode - The connection mode used by the component.

        • peer_addr - The IP from which the event originates.

        • peer_path - The pathname from which the event originates.

        • pod_name - The name of the pod from which the event originates.

        • uri - The sanitized uri from which the event originates.

      • counter

        events_out_total

        The total number of events emitted by this component. This metric includes the following tags:

        • component_kind - The Vector component kind.

        • component_name - The Vector component ID.

        • component_type - The Vector component type.

        • instance - The Vector instance identified by host and port.

        • job - The name of the job producing Vector metrics.

      • counter

        protobuf_decode_errors_total

        The total number of Protocol Buffers errors thrown during communication between Vector instances. This metric includes the following tags:

        • component_kind - The Vector component kind.

        • component_name - The Vector component ID.

        • component_type - The Vector component type.

        • instance - The Vector instance identified by host and port.

        • job - The name of the job producing Vector metrics.

      How It Works

      Communication Protocol

      Upstream Vector instances forward data to downstream Vector instances via the TCP protocol.

      Context

      By default, the vector source will augment events with helpful context keys as shown in the "Output" section.

      Encoding

      Data is encoded via Vector's event protobuf before it is sent over the wire.

      Message Acknowledgement

      Currently, Vector does not perform any application level message acknowledgement. While rare, this means the individual message could be lost.

      State

      This component is stateless, meaning its behavior is consistent across each input.

      Transport Layer Security (TLS)

      Vector uses Openssl for TLS protocols. You can adjust TLS behavior via the tls.* options.