vectordotdev / vector
A high-performance observability data pipeline.
AI Architecture Analysis
This repository is indexed by RepoMind. By analyzing vectordotdev/vector in our AI interface, you can instantly generate complete architecture diagrams, visualize control flows, and perform automated security audits across the entire codebase.
Our Agentic Context Augmented Generation (Agentic CAG) engine loads full source files into context, avoiding the fragmentation of traditional RAG systems. Ask questions about the architecture, dependencies, or specific features to see it in action.
Embed this Badge
Showcase RepoMind's analysis directly in your repository's README.
Repository Summary (README)
Preview
What is Vector?
Vector is a high-performance, end-to-end (agent & aggregator) observability data pipeline that puts you in control of your observability data. Collect, transform, and route all your logs and metrics to any vendors you want today and any other vendors you may want tomorrow. Vector enables dramatic cost reduction, novel data enrichment, and data security where you need it, not where it is most convenient for your vendors. Additionally, it is open source and up to 10x faster than every alternative in the space.
To get started, follow our quickstart guide or install Vector.
Vector is maintained by Datadog's Community Open Source Engineering team.
Principles
- Reliable - Built in Rust, Vector's primary design goal is reliability.
- End-to-end - Deploys as an agent or aggregator. Vector is a complete platform.
- Unified - Logs, metrics (beta), and traces (coming soon). One tool for all of your data.
Use cases
- Reduce total observability costs.
- Transition vendors without disrupting workflows.
- Enhance data quality and improve insights.
- Consolidate agents and eliminate agent fatigue.
- Improve overall observability performance and reliability.
Community
- Vector is relied on by startups and enterprises like Atlassian, T-Mobile, Comcast, Zendesk, Discord, Fastly, CVS, Trivago, Tuple, Douban, Visa, Mambu, Blockfi, Claranet, Instacart, Forcepoint, and many more.
- Vector is downloaded over 100,000 times per day.
- Vector's largest user processes over 500TB daily.
- Vector has over 500 contributors and growing.
Documentation
All user documentation is available at vector.dev/docs.
Other Resources:
Comparisons
Performance
The following performance tests demonstrate baseline performance between common protocols with the exception of the Regex Parsing test.
| Test | Vector | Filebeat | FluentBit | FluentD | Logstash | SplunkUF | SplunkHF |
|---|---|---|---|---|---|---|---|
| TCP to Blackhole | 86mib/s | n/a | 64.4mib/s | 27.7mib/s | 40.6mib/s | n/a | n/a |
| File to TCP | 76.7mib/s | 7.8mib/s | 35mib/s | 26.1mib/s | 3.1mib/s | 40.1mib/s | 39mib/s |
| Regex Parsing | 13.2mib/s | n/a | 20.5mib/s | 2.6mib/s | 4.6mib/s | n/a | 7.8mib/s |
| TCP to HTTP | 26.7mib/s | n/a | 19.6mib/s | <1mib/s | 2.7mib/s | n/a | n/a |
| TCP to TCP | 69.9mib/s | 5mib/s | 67.1mib/s | 3.9mib/s | 10mib/s | 70.4mib/s | 7.6mib/s |
To learn more about our performance tests, please see the Vector test harness.
Correctness
The following correctness tests are not exhaustive, but they demonstrate fundamental differences in quality and attention to detail:
| Test | Vector | Filebeat | FluentBit | FluentD | Logstash | Splunk UF | Splunk HF |
|---|---|---|---|---|---|---|---|
| Disk Buffer Persistence | ✓ | ✓ | ⚠ | ✓ | ✓ | ||
| File Rotate (create) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| File Rotate (copytruncate) | ✓ | ✓ | ✓ | ||||
| File Truncation | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Process (SIGHUP) | ✓ | ⚠ | ✓ | ✓ | |||
| JSON (wrapped) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
To learn more about our correctness tests, please see the Vector test harness.
Features
Vector is an end-to-end, unified, open data platform.
| Vector | Beats | Fluentbit | Fluentd | Logstash | Splunk UF | Splunk HF | Telegraf | |
|---|---|---|---|---|---|---|---|---|
| End-to-end | ✓ | ✓ | ||||||
| Agent | ✓ | ✓ | ✓ | ✓ | ✓ | |||
| Aggregator | ✓ | ✓ | ✓ | ✓ | ✓ | |||
| Unified | ✓ | ✓ | ||||||
| Logs | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Metrics | ✓ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ | ✓ |
| Traces | 🚧 | |||||||
| Open | ✓ | ✓ | ✓ | ✓ | ||||
| Open-source | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ||
| Vendor-neutral | ✓ | ✓ | ✓ | ✓ | ||||
| Reliability | ✓ | |||||||
| Memory-safe | ✓ | ✓ | ||||||
| Delivery guarantees | ✓ | ✓ | ✓ | |||||
| Multi-core | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
⚠ = Not interoperable, metrics are represented as structured logs
<p align="center"> Developed with ❤️ by <strong><a href="https://datadoghq.com">Datadog</a></strong> - <a href="https://github.com/vectordotdev/vector/security/policy">Security Policy</a> - <a href="https://github.com/vectordotdev/vector/blob/master/PRIVACY.md">Privacy Policy</a> </p>