In the previous section we covered the various deployment strategies used to collect and forward data. You combine these strategies to form topologies. This section showcases common topologies and the pros and cons of each. Use these as guidelines to build your own.
The simplest topology. This topology offers direct communication with your downstream services from your client nodes.
- Simple. Less moving parts.
- Elastic. Easily scales with your app. Resources grow as you scale.
- Less efficient. Depending on the complexity of your pipelines, this will use more local resources, which could disrupt the performance of other applications on the same host.
- Less durable. Because data is buffered on the host it is more likely you'll lose buffered data in the event of an unrecoverable crash. Often times this is the most important and useful data.
- More downstream stress. Downstream services will receive more requests with smaller payloads that could potentially disrupt stability of these services.
- Reduced downstream stability. You risk overloading downstream services if you need to scale up quickly or exceed the capacity a downstream service can handle.
- Lacks multi-host context. Lacks awareness of other hosts and eliminates the ability to perform operations across hosts, such as reducing logs to global metrics. This is typically a concern for very large deployments where individual host metrics are less useful.
A good balance of simplicity, stability, and control. For most, this topology is a good compromise between the distributed and stream-based topologies. It offers many of the advantages of the stream-based topology without the management overhead and cost of running a Kafka cluster.
- More efficient. More efficient for client nodes and downstream services. Vector agents are doing less which means they're using less resources. In addition, the centralized Vector service will buffer more data, achieve better compression, and send optimized requests downstream.
- More reliable. Vector will protect downstream services from volume spikes, buffering and flushing data at smoothed out intervals.
- Has multi-host context. Because your data is centralized, you'll be able to perform operations across hosts, such as reducing logs to global metrics. This is advantageous for large deployments where individual host metrics are not interesting.
- More complex. There are more moving parts.
- Less durable. Agent nodes are designed to get data off of the machine as quickly as possible, and this offers higher potential for data loss since the central Vector service could die and lose buffered data. If this type of outage is unacceptable, then we recommend the stream-based topology below.
The most durable and elastic topology. This topology is typically adopted for very large streams with teams that are familiar with running a stream-based service such as Kafka.
- Most durable & reliable. Stream services, like Kafka, are designed for high durability and reliability, replicating data across multiple nodes.
- Most efficient. Vector agents are doing less, making them more efficient, and Vector services do not have to worry about durability, which can be tuned towards performance.
- Ability to re-stream. Re-stream your data depending on your stream's retention period.
- Cleaner separation of responsibilities. Vector is used solely as a routing layer and is not responsible for durability. Durability is delegated to a purpose-built service that you can switch and evolve over time.
- Increased management overhead. Managing a stream service, such as Kafka, is a complex endeavor and generally requires an experienced team to setup and manage properly.
- More complex. This topology is complex and requires a deeper understand of managing production-grade streams.
- More expensive. In addition the management cost, the added stream cluster will require more resources which will increase operational cost.