Logs are an essential part of observing any service; without them you'll have significant blind spots. But collecting and analyzing them can be a real challenge -- especially at scale. Not only do you need to solve the basic task of collecting your logs, but you must do it in a reliable, performant, and robust manner. Nothing is more frustrating than having your logs pipeline fall on it's face during an outage, or even worse, cause the outage!
Fear not! In this guide we'll build an observability pipeline that will send logs from Kubernetes to Apache Pulsar.
What is Kubernetes?
Kubernetes, also known as k8s, is an open-source container-orchestration system for automating application deployment, scaling, and management.
What is Apache Pulsar?
Pulsar is a multi-tenant, high-performance solution for server-to-server messaging. Pulsar was originally developed by Yahoo, it is under the stewardship of the Apache Software Foundation. It is an excellent tool for streaming logs and metrics data.
How This Guide Works
We'll be using Vector to accomplish this task. Vector is a popular open-source observability data platform. It's written in Rust, making it lightweight, ultra-fast and highly reliable. And we'll be deploying Vector as a agent.
What We'll Accomplish
We'll build an observability data platform that:
All in just a few minutes!
Add the Vector repohelm repo add timberio https://packages.timber.io/helm/latest
Check available Helm chart configuration optionshelm show values timberio/vector-agent
Configure Vectorcat <<-'VALUES' > values.yaml# The Vector Kubernetes integration automatically defines a# kubernetes_logs source that is made available to you.# You do not need to define a log source.sinks:# Adjust as necessary. By default we use the console sink# to print all data. This allows you to see Vector working.# https://vector.dev/docs/reference/sinks/stdout:type: consoleinputs: ["kubernetes_logs"]rawConfig: |target = "stdout"encoding = "json"VALUES
Install Vectorhelm install --namespace vector --create-namespace vector timberio/vector-agent --values values.yaml
Observe Vectorkubectl logs --namespace vector daemonset/vector-agentexplain this command
Vector is powerful tool and we're just scratching the surface in this guide. Here are a few pages we recommend that demonstrate the power and flexibility of Vector: