Internal metrics

Access to the metrics produced by Vector itself and process them in your Vector pipeline

status: beta role: aggregator role: daemon role: sidecar delivery: at-least-once egress: batch state: stateless output: metrics

Configuration

Example configurations

{
  "sources": {
    "my_source_id": {
      "type": "internal_metrics",
      "scrape_interval_secs": 2
    }
  }
}
[sources.my_source_id]
type = "internal_metrics"
scrape_interval_secs = 2
---
sources:
  my_source_id:
    type: internal_metrics
    scrape_interval_secs: 2
{
  "sources": {
    "my_source_id": {
      "type": "internal_metrics",
      "scrape_interval_secs": 2
    }
  }
}
[sources.my_source_id]
type = "internal_metrics"
scrape_interval_secs = 2
---
sources:
  my_source_id:
    type: internal_metrics
    scrape_interval_secs: 2

scrape_interval_secs

common optional uint
The interval between metric gathering, in seconds.
default: 2 (seconds)

Output

Metrics

adaptive_concurrency_averaged_rtt

histogram
The average round-trip time (RTT) for the current window.

adaptive_concurrency_in_flight

histogram
The number of outbound requests currently awaiting a response.

adaptive_concurrency_limit

histogram
The concurrency limit that the adaptive concurrency feature has decided on for this current window.

adaptive_concurrency_observed_rtt

histogram
The observed round-trip time (RTT) for requests.

aggregate_events_recorded_total

counter
The number of events recorded by the aggregate transform.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

aggregate_failed_updates

counter
The number of failed metric updates, incremental adds, encountered by the aggregate transform.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

aggregate_flushes_total

counter
The number of flushes done by the aggregate transform.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

api_started_total

counter
The number of times the Vector GraphQL API has been started.

checkpoint_write_errors_total

counter
The total number of errors writing checkpoints.

checkpoints_total

counter
The total number of files checkpointed.

checksum_errors_total

counter
The total number of errors identifying files via checksum.
file optional
The file that produced the error

collect_completed_total

counter
The total number of metrics collections completed for this component.

collect_duration_seconds

histogram
The duration spent collecting of metrics for this component.

command_executed_total

counter
The total number of times a command has been executed.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

command_execution_duration_seconds

histogram
The command execution duration in seconds.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

communication_errors_total

counter
The total number of errors stemming from communication with the Docker daemon.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

config_load_errors_total

counter
The total number of errors loading the Vector configuration.

connection_errors_total

counter
The total number of connection errors for this Vector instance.

connection_established_total

counter
The total number of times a connection has been established.

connection_failed_total

counter
The total number of times a connection has failed.

connection_read_errors_total

counter
The total number of errors reading datagram.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.
mode required

connection_send_ack_errors_total

counter
The total number of protocol acknowledgement errors for this Vector instance for source protocols that support acknowledgements.

connection_send_errors_total

counter
The total number of errors sending data via the connection.

connection_shutdown_total

counter
The total number of times the connection has been shut down.

consumer_offset_updates_failed_total

counter
The total number of failures to update a Kafka consumer offset.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

container_metadata_fetch_errors_total

counter
The total number of errors encountered when fetching container metadata.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

container_processed_events_total

counter
The total number of container events processed.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

containers_unwatched_total

counter
The total number of times Vector stopped watching for container logs.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

containers_watched_total

counter
The total number of times Vector started watching for container logs.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

decode_errors_total

counter
The total number of decode errors seen when decoding data in a source component.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

encode_errors_total

counter
The total number of errors encountered when encoding an event.

events_discarded_total

counter
The total number of events discarded by this component.

events_failed_total

counter
The total number of failures to read a Kafka message.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

events_in_total

counter
The number of events accepted by this component either from tagged origin like file and uri, or cumulatively from other origins.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.
container_name optional
The name of the container from which the event originates.
file optional
The file from which the event originates.
mode optional
The connection mode used by the component.
peer_addr optional
The IP from which the event originates.
peer_path optional
The pathname from which the event originates.
pod_name optional
The name of the pod from which the event originates.
uri optional
The sanitized URI from which the event originates.

events_out_total

counter
The total number of events emitted by this component.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

file_delete_errors_total

counter
The total number of failures to delete a file.
file optional
The file that produced the error

file_watch_errors_total

counter
The total number of errors encountered when watching files.
file optional
The file that produced the error

files_added_total

counter
The total number of files Vector has found to watch.
file optional
The file that produced the error

files_deleted_total

counter
The total number of files deleted.
file optional
The file that produced the error

files_resumed_total

counter
The total number of times Vector has resumed watching a file.
file optional
The file that produced the error

files_unwatched_total

counter
The total number of times Vector has stopped watching a file.
file optional
The file that produced the error

fingerprint_read_errors_total

counter
The total number of times Vector failed to read a file for fingerprinting.
file optional
The file that produced the error

glob_errors_total

counter
The total number of errors encountered when globbing paths.
path required
The path that produced the error.

http_bad_requests_total

counter
The total number of HTTP 400 Bad Request errors encountered.

http_client_response_rtt_seconds

histogram
The round-trip time (RTT) of HTTP requests, tagged with the response code.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.
status optional
The HTTP status code of the request.

http_client_responses_total

counter
The total number of HTTP requests, tagged with the response code.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.
status optional
The HTTP status code of the request.

http_client_rtt_seconds

histogram
The round-trip time (RTT) of HTTP requests.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

http_error_response_total

counter
The total number of HTTP error responses for this component.

http_request_errors_total

counter
The total number of HTTP request errors for this component.

http_requests_total

counter
The total number of HTTP requests issued by this component.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

invalid_record_bytes_total

counter
The total number of bytes from invalid records that have been discarded.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

invalid_record_total

counter
The total number of invalid records that have been discarded.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

k8s_docker_format_parse_failures_total

counter
The total number of failures to parse a message as a JSON object.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

k8s_event_annotation_failures_total

counter
The total number of failures to annotate Vector events with Kubernetes Pod metadata.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

k8s_format_picker_edge_cases_total

counter
The total number of edge cases encountered while picking format of the Kubernetes log message.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

k8s_reflector_desyncs_total

counter
The total number of desyncs for the reflector.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

k8s_state_ops_total

counter
The total number of state operations.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.
op_kind optional
The kind of operation performed.

k8s_stream_chunks_processed_total

counter
The total number of chunks processed from the stream of Kubernetes resources.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

k8s_stream_processed_bytes_total

counter
The number of bytes processed from the stream of Kubernetes resources.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

k8s_watch_requests_failed_total

counter
The total number of watch requests failed.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

k8s_watch_requests_invoked_total

counter
The total number of watch requests invoked.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

k8s_watch_stream_failed_total

counter
The total number of watch streams failed.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

k8s_watch_stream_items_obtained_total

counter
The total number of items obtained from a watch stream.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

k8s_watcher_http_error_total

counter
The total number of HTTP error responses for the Kubernetes watcher.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

kafka_consumed_messages_bytes_total

counter
Total number of message bytes (including framing) received from Kafka brokers.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

kafka_consumed_messages_total

counter
Total number of messages consumed, not including ignored messages (due to offset, etc), from Kafka brokers.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

kafka_produced_messages_bytes_total

counter
Total number of message bytes (including framing, such as per-Message framing and MessageSet/batch framing) transmitted to Kafka brokers.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

kafka_produced_messages_total

counter
Total number of messages transmitted (produced) to Kafka brokers.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

kafka_queue_messages

gauge
Current number of messages in producer queues.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

kafka_queue_messages_bytes

gauge
Current total size of messages in producer queues.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

kafka_requests_bytes_total

counter
Total number of bytes transmitted to Kafka brokers.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

kafka_requests_total

counter
Total number of requests sent to Kafka brokers.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

kafka_responses_bytes_total

counter
Total number of bytes received from Kafka brokers.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

kafka_responses_total

counter
Total number of responses received from Kafka brokers.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

logging_driver_errors_total

counter
The total number of logging driver errors encountered caused by not using either the jsonfile or journald driver.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

memory_used_bytes

gauge
The total memory currently being used by Vector (in bytes).

metadata_refresh_failed_total

counter
The total number of failed efforts to refresh AWS EC2 metadata.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

metadata_refresh_successful_total

counter
The total number of AWS EC2 metadata refreshes.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

open_connections

gauge
The number of current open connections to Vector.

parse_errors_total

counter
The total number of errors parsing metrics for this component.

processed_bytes_total

counter
The number of bytes processed by the component.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.
container_name optional
The name of the container from which the bytes originate.
file optional
The file from which the bytes originate.
mode optional
The connection mode used by the component.
peer_addr optional
The IP from which the bytes originate.
peer_path optional
The pathname from which the bytes originate.
pod_name optional
The name of the pod from which the bytes originate.
uri optional
The sanitized URI from which the bytes originate.

processed_events_total

counter
The total number of events processed by this component. This metric is deprecated in place of using events_in_total and events_out_total metrics.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

processing_errors_total

counter
The total number of processing errors encountered by this component.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.
error_type required
The type of the error

protobuf_decode_errors_total

counter
The total number of Protocol Buffers errors thrown during communication between Vector instances.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

quit_total

counter
The total number of times the Vector instance has quit.

recover_errors_total

counter
The total number of errors caused by Vector failing to recover from a failed reload.

reload_errors_total

counter
The total number of errors encountered when reloading Vector.

reloaded_total

counter
The total number of times the Vector instance has been reloaded.

request_automatic_decode_errors_total

counter
The total number of request errors for this component when it attempted to automatically discover and handle the content-encoding of incoming request data.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

request_duration_seconds

histogram
The total request duration in seconds.

request_errors_total

counter
The total number of requests errors for this component.

request_read_errors_total

counter
The total number of request read errors for this component.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

requests_completed_total

counter
The total number of requests completed by this component.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

requests_received_total

counter
The total number of requests received by this component.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

send_errors_total

counter
The total number of errors sending messages.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

sqs_message_delete_failed_total

counter
The total number of failures to delete SQS messages.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

sqs_message_delete_succeeded_total

counter
The total number of successful deletions of SQS messages.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

sqs_message_processing_failed_total

counter
The total number of failures to process SQS messages.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

sqs_message_processing_succeeded_total

counter
The total number of SQS messages successfully processed.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

sqs_message_receive_failed_total

counter
The total number of failures to receive SQS messages.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

sqs_message_receive_succeeded_total

counter
The total number of times successfully receiving SQS messages.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

sqs_message_received_messages_total

counter
The total number of received SQS messages.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

sqs_s3_event_record_ignored_total

counter
The total number of times an S3 record in an SQS message was ignored (for an event that was not ObjectCreated).
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.
ignore_type required
The reason for ignoring the S3 record

stale_events_flushed_total

counter
The number of stale events that Vector has flushed.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

started_total

counter
The total number of times the Vector instance has been started.

stdin_reads_failed_total

counter
The total number of errors reading from stdin.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

stopped_total

counter
The total number of times the Vector instance has been stopped.

tag_value_limit_exceeded_total

counter
The total number of events discarded because the tag has been rejected after hitting the configured value_limit.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

timestamp_parse_errors_total

counter
The total number of errors encountered parsing RFC 3339 timestamps.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

uptime_seconds

gauge
The total number of seconds the Vector instance has been up.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

utf8_convert_errors_total

counter
The total number of errors converting bytes to a UTF-8 string in UDP mode.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.
mode required
The connection mode used by the component.

value_limit_reached_total

counter
The total number of times new values for a key have been rejected because the value limit has been reached.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

windows_service_does_not_exist_total

counter
The total number of errors raised due to the Windows service not existing.

windows_service_install_total

counter
The total number of times the Windows service has been installed.

windows_service_restart_total

counter
The total number of times the Windows service has been restarted.

windows_service_start_total

counter
The total number of times the Windows service has been started.

windows_service_stop_total

counter
The total number of times the Windows service has been stopped.

windows_service_uninstall_total

counter
The total number of times the Windows service has been uninstalled.

Telemetry

Metrics

link

events_out_total

counter
The total number of events emitted by this component.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

How it works

Context

By default, the internal_metrics source augments events with helpful context keys.

State

This component is stateless, meaning its behavior is consistent across each input.