Vector v0.10.0 release notes

Code name Lamport Clock

Upgrading Vector
When upgrading, we recommend stepping through minor versions as these can each contain breaking changes while Vector is pre-1.0. These breaking changes are noted in their respective upgrade guides.

Highlights

Leveraging glibc when possible

Author avatar for hoverbear
hoverbear
type: performance domain: operations
New Reduce transform

Author avatar for hoverbear
hoverbear
type: new feature domain: transforms
Kafka components support SASL

Author avatar for hoverbear
hoverbear
type: new feature domain: sinks sink: kafka
New compression options for some sinks

Author avatar for hoverbear
hoverbear
type: new feature sink: aws_kinesis_firehose sink: aws_kinesis_streams sink: aws_s3 sink: humio_logs
Now supporting the bearer auth strategy

Author avatar for hoverbear
hoverbear
type: enhancement domain: sinks sink: http
RegexSet support to regex transform

Author avatar for hoverbear
hoverbear
type: enhancement domain: transforms transform: regex_parser

Changelog

12 new features

new source New generator source 2286 operations Optimize CI test workflow with slim builds 2440 kafka sink Add support for templates in the topic config 2598 internal_metrics source Instrument few more components with metrics 2620 new sink Initial influxdb_logs sink implementation 2474 platforms Add dev workflow simplification rfc 2685 new transform Introduce WASM Plugins 2006 operations Introduce Dev Workflow improvements 2723 filter transform Adding list of string as filter option 2745 operations More usable event RFC 2692 filter transform Adding a condition to filter using IP CIDR 2831 new transform add reduce transform 2870

26 enhancements

splunk_hec sink Add sourcetype 2318 topology Shutdown Vector if all sources finish 2533 filter transform swimlanes transform Add support for negating conditions in check_fields 2514 elasticsearch sink Add explicit AWS region 2532 journald source Add support for include/exclude units 2540 regex_parser transform Add RegexSet support to regex 2493 json_parser transform log parse errors as warnings 2593 http sink add bearer auth strategy 2607 security Add support for loading multiple CAs 2616 networking Add support for inline TLS keys 2724 humio sink new relic sink Add compression to humio and new_relic 2682 elasticsearch sink add pipeline config 2734 cli Consolidate and beautify validate 2622 gcp_stackdriver_logs sink Add severity key 2732 sinks Use to_raw_value instead of to_string in JsonArrayBuffer 2860 aws_s3 sink Add content options 2848 elasticsearch sink Show client errors as warning logs 2825 elasticsearch sink add assume_role to aws auth 2895 kafka source kafka sink Add SASL to kafka 2897 sinks Batch buffer rework 2866 journald source Allow remapping priority numbers to words 2751 file source Option to remove file after some time of reaching eof 2908 sinks add compression option for aws sinks 2953 kafka source Extract timestamp from event 2956 operations Add musl and glibc support to install.sh 2969 syslog source Accept octet-counting encoded messages 2955

44 bug fixes

kafka sink Run entire healthcheck lazily 2501 socket sink Properly detect shutdown 2429 loki sink replace OrigID with OrgID 2536 operations Iterate over target dirs to avoid missing mapall command on macOS 2539 datadog_metrics sink Use post requests 2547 splunk_hec sink Send time in . format 2570 humio_logs sink remove superfluous warning 2583 elasticsearch sink remove id key from body 2581 journald source Fix units/include_units switch 2567 socket sink Fix possible infinite loop decoding too-long lines 2587 aws_s3 sink Allow removed keys to be used for partitioning 2591 datadog_logs sink Enable tls by default 2527 tests Fixup loki tests 2613 tests Remove race between sink and source 2619 file source Fix the bug with channel closing at file server 2652 kafka sink kafka source Fix the Kafka TLS key parameter handling 2658 topology Check if transform/sink is present 2617 sources Shutdown starting from tcp and unix sockets 2618 influxdb_logs sink Fix buffer compression 2679 kafka sink Three KafkaCompression typos in tests 2694 splunk_hec sink Remove extra slash on Uri join 2711 pulsar sink Add new EncodingConfig 2671 sinks increment request id 2765 config Handle templates with invalid strftime elements 2737 aws_cloudwatch_sink Make throttling errors retryable 2752 aws_cloudwatch_logs sink Drop excessive large events 2770 sinks upgrade tower-limit to fix rate limiting 2779 kafka sink Fix typo integration tests 2832 sinks address potential deadlock 2838 influxdb_metrics sink Use i for unsigned ints on Influx v1 2868 splunk_hec source fix wrap path 2873 elasticsearch sink refresh aws creds when required 2859 aws_cloudwatch_logs sink Handle and enforce batch.max_bytes 2916 operations Fix the cmark-gfm installation at builder image 2996 operations Use perl rename at builder 2999 gcp provider sync token request 2993 loki sink Set a proper default for the batch buffer initialization 3016 deps update crates in lock file after merges 3030 cli Cleanup generate command 3046 docs adjust log event in coercer transform 3072 reduce transform improve array handling 3076 loki sink Drop the batch byte size limit 3025 file sink append instead of overwriting files 3084 elasticsearch sink stop logging every received event 3085

213 chore

deps Remove proptest dep 2379 Bump cargo.toml version to 0.10.0 2386 buffers carve a seam around leveldb 2363 networking Upgrade service and retries in sink util 2362 deps Upgrade rustc 1.43.0 2422 operations Cancel previous builds automatically 2442 operations Stylize tests/Makefile 2450 operations Use specific target names in Makefile 2451 operations Allow the invocation of local script from the tests/Makefile 2452 operations Point CI GHA workflow to tests/Makefile 2453 Add missing loki integration test 2457 operations Add aws-integration-tests & gcp-integration-tests feature 2458 operations Point nightly GHA workflow to tests/Makefile 2456 tests Remove old kubernetes tests 2462 operations Move /tests/Makefile to /Makefile 2460 security Ignore RUSTSEC-2020-0014 2465 operations Add rpm builds to nightly workflow 2463 deps Upgrade snafu to 0.6 2466 operations Fix unit test failure in CI 2470 operations Fix typo at nightly CI workflow 2472 buffers add new isolated buffer benchmarks 2447 operations Fix nightly builds 2471 Encoding config accessor visibility reduction 2153 operations Fix local unit test execution 2490 operations Prevent docker from creating directories and files as root 2486 add_fields transform add_tags transform Add overwrite option 2485 Finish git merge 2502 Temporarily remove Kubernetes components to clear the way for the new integration 2488 RFC #1858 - 2020-04-06 - Automatic rate limit adjustment 2329 Kubernetes Integration RFC 2222 operations Add retries to the verification of the release artifacts 2509 operations RPM build fix 2522 rpm platform do not create /LICENSE and /README.md at RPM package 2524 operations Dynamically derive the list of paths to create at target 2525 operations Align GitHub Actions workflows with the files changed 2517 operations Upgrade deny action 2544 operations Fix rustfmt 2545 security Upgrade openssl to 1.1.1g 2546 operations Disable check-version check 2548 operations Add shellcheck to CI and correct scripts 2482 operations Remove the .github/workflows/install-script.yml 2549 stdin source Wire ShutdownSignal 2366 sinks Upgrade internal rusoto utilities 2555 operations Kubernetes test procedure 2487 operations Upgrade to rustc 1.43.1 2563 Set minimal supported Kubernetes version 2510 operations Improve check-advisories CI job 2571 operations Force disable k8s tests in CI 2574 sinks Add BatchedHttpSink back to http2 2573 clickhouse sink Upgrade clickhouse 2578 operations Copying docker images directly to minikube VMs 2575 tests remove unused import 2582 topology Test vector shutdown 2558 Mark GeneratorConfig::repeat with #[allow(dead_code)] 2586 topology Update shutdown deadlines 2534 tests Use localhost for prometheus shutdown test 2589 Generalize file server around paths provider 2541 config Fix default values for buffer 2590 operations Mute k8s tests 2594 Fix newline for check-fmt 2602 http sink Error on auth section and Auth header 2611 operations Build archives, DEB, and RPM packages with glibc 2518 Replace http:// to https:// 2646 dpkg platform Debug package-deb 2665 Improve compression option 2637 vector source Rate limit TCP connection errors, downgrade their severity 2669 lua transform Avoid hanging on timers 2672 dpkg platform Display free space at package deb task 2686 deps bump activesupport from 6.0.2.1 to 6.0.3.1 in /scripts 2689 Upgrade HttpSink based sinks 2688 Cleanup the leftovers of the old k8s implementation 2676 RFC #2341 - 2020-04-15 - WASM Plugins Support 2341 Reorder Tower service layers 2703 operations A script to install git hooks 2650 syslog source Add hostname and source_ip 2663 syslog source Update syslog_loose to 0.3.0 2709 tests Use test_util::runtime() in tests 2715 tests Test with default feature in CI 2721 operations Fix check-markdown job 2727 operations Split website workflow into generate and blog 2728 sinks Only log 200-299 responses as successful 2714 kafka sink Fix integration test 2698 prometheus sink Upgrade hyper 2754 pulsar sink Add into 2759 splunk_hec source Upgrade hyper 2761 gcp_cloud_storage sink Upgrade hyper 2758 aws_kinesis_firehose sink Updade rusoto 2735 Add clippy to CI 2576 influxdb_metrics sink Upgrade hyper 2760 deps bump websocket-extensions from 0.1.3 to 0.1.4 in /website 2756 tests Upgrade hyper 2762 aws_ec2_metadata transform Upgrade hyper 2763 dpkg platform Convert pipe to variables to debug 141 (broken pipe) error 2772 Log when components finish normally 2801 tests fix aws integration tests ci 2792 statsd sink tower update 2805 deps update rusoto to 0.44.0 2806 lua transform Don’t finish when timers finish 2809 prometheus source Update hyper 2808 prometheus source Update hyper in tests 2817 deps bump regex from 1.3.5 to 1.3.9 2818 deps update goauth to new futures 2819 humio_logs sink improve time validation in tests 2803 deps Bump rust-toolchain to 1.44.0 2813 deps bump maxminddb from 0.13.0 to 0.14.0 2791 deps bump tokio-codec from 0.1.1 to 0.1.2 2783 socket sink Fix tcp+tls hang on tcp detect read 2566 RFC #2768 - 2020-06-12 - Batch and Buffer Rework 2802 topology Allow disconnected components, and split start step 2793 Fix names of RFCs. 2839 bump typetag from 0.1.4 to 0.1.5 2835 bump assert_cmd from 0.11.1 to 1.0.1 2829 json_parser transform Use flat JSON parser inserts, add regression test. 2823 deps bump criterion from 0.3.1 to 0.3.2 2827 aws_cloudwatch_logs sink update rusoto 2830 loki sink Make encoding optional 2796 networking Remove custom DNS resolution 2812 deps Finish hyper upgrade 2852 tests Properly synchronize file updates in watcher test 2853 deps warp update 2837 operations Fix release verification. 2847 deps file-source bytes update 2865 tests Shutdown write side only in test tcp_stream_detects_disconnect 2857 Bump rust-toolchain to 1.44.1 2874 kafka sink Fix command for integration tests 2834 remove outdated uri code 2889 remove contextual QA checklist 2898 tests Adjust buffering tests 2862 deps Remove trust-dns dependencies 2884 make HttpSink::build_request async 2888 reduce http:0.1 usage 2905 deps remove native-tls 2912 kafka kink Fixup sasl related knobs 2915 Remove two unneeded files 2918 file source Add Fingerprinter::FirstLineChecksum 2904 Architecture Revisit RFC 2855 deps move reqwest to dev-dependencies 2952 kafka sink disable sasl feature 2930 sinks Construct tcp healthcheck with TLS 2958 platforms Move the builder to centos 7 2824 move topology building into tokio context, take 2 2964 docker source upgrade to bollard 2951 remove some allowed clippy rules 2959 operations Move over homebrew/s3 release jobs 2869 operations Add smoke tests to docker images 2974 deps bump nom from 5.1.0 to 5.1.2 2967 Update README.md 2976 deps bump scan_fmt from 0.2.4 to 0.2.5 2849 deps Configure dependabot to use correct issue names 2821 deps Fixup dependabot config 2980 deps bump built from 0.3.2 to 0.4.2 2850 operations Make environment use ubuntu 20.04 2971 deps bump @types/fs-extra from 8.1.0 to 8.1.1 in /website 2986 deps bump yard from 0.9.24 to 0.9.25 in /scripts 2981 deps bump @types/lodash from 4.14.150 to 4.14.157 in /website 2988 deps Fix dependency label to be longer again. 2990 deps bump qs from 6.9.3 to 6.9.4 in /website 2982 operations Add smoke test for cmark-gfm at builder 2995 security remove outdated security advisories 3009 deps Make dependabot only cover Rust code 3003 journald source Fix filter_unit_works_correctly test 3005 operations Fixup release script formats 3002 operations Fixup release-s3 3015 pulsar sink update to 1.0.0-alpha-2 2977 operations Remove dead envrc file 3023 deps update prost 3017 Drop 3 more clippy exclusions 3026 tests Delete topology::config::watcher::tests::multi_file_update 3034 tests Delete topology::reload_tests::topology_reuse_old_port 3036 sinks Build HttpClient once 3010 deps remove tokio-codec crate 3011 tests Delete tests\tcp::merge test 3042 tests Delete test_udp_syslog test 3044 tests Delete test_max_size_resume 3040 deps bump num_cpus from 1.12.0 to 1.13.0 2910 deps bump serde_json from 1.0.55 to 1.0.56 3051 deps bump bytesize from 1.0.0 to 1.0.1 3050 pulsar sink bump to pulsar-rs 1.0.0 3053 reduce std::io::Cursor usage 3038 tests Fixup integration test invocations 3004 operations Fix broken make generate 3058 deps bump bollard from 0.7.0 to 0.7.1 3048 sinks Build aws sinks clients only once 3045 kafka source remove owning_ref 3059 transforms update runtime_transform 3065 Add .rustfmt.toml to assist IDE formatters. 3063 sinks add SinkConfig::build_async 3066 operations Remove CircleCI 3001 deps bump flate2 from 1.0.13 to 1.0.16 3049 sinks fix endpoint strip for aws region 3070 operations Add sync-install to GA 3056 deps bump tokio-uds from 0.2.5 to 0.2.7 2946 Fix warning message due .rustfmt.toml 3075 tests use localstack instead minio 3073 deps bump inventory from 0.1.5 to 0.1.7 2851 deps Reenable sasl 3081 tests Remove tcp_graceful_shutdown 3103 tests Remove s3_waits_for_full_batch_or_timeout_before_sending 3105 Bump rust-toolchain to 1.45.0 3089 operations Docker debian packages systemd 3129 Add Rust 2018 edition tag to .rustfmt.toml 3120 operations Debian buster is stable, not sid 3131 deps bump lodash from 4.17.15 to 4.17.19 in /website 3098 tests Update benches 3107 Remove tokio 0.1 from shutdown 3106 operations Update cidr-utils 3139 operations Pin back Rust 1.45 until after 0.10.0 3142 operations Fixup clippy issues from 1.44 rollback 3143 Prepare v0.10.0 3022 operations Fixup release-homebrew 3158 operations Fixup version invocation to not have info 3153 operations Fixup release s3 job 3155 operations Fixup release commit 3163

What’s next

Kubernetes! ⎈
We’re doing this integration right. We’ve been alpha testing with a number of large clusters and we’ve honed in on a very high-quality approach that reflects the UX Vector is going after. You can learn more in our Kubernetes Integration RFC. Do you want to beta test? Tell us!
WASM plugins 🔌
This release included an experimental implementation of a WASM transform behind a wasm build flag. We plan to stabilize WASM transforms, as well as add WASM sinks and sources, enabling you to extend Vector in the language of your choice with little performance penalty. ❤️ WASM. Check out the WASM Plugin RFC for more info.
Dynamic HTTP Rate-Limiting (AIMD)
Fiddling with rate-limits is a frustrating endeavor. If you set them too high you risk overwhelming the service; too low and you’re unnecessarily limiting throughput. What happens if you deploy a few more Vector instances? Vector is planning to solve this by automatically detecting the optimal rate limits, taking inspiration from TCP congestion control algorithms. Check out the Dynamic HTTP Rate Limiting RFC.

Download Version 0.10.0

Linux
tar.gz
ARM64 ARMv7 x86_64
Linux (deb)
deb
ARM64 ARMv7 x86_64
Linux (rpm)
rpm
ARM64 ARMv7 x86_64
macOS
tar.gz
x86_64
Windows
zip
x86_64
Windows (MSI)
msi
x86_64
browse all files...