A high-performance observability data pipeline.
  • Rust 63.5%
  • CUE 34%
  • HTML 1.2%
  • TypeScript 0.7%
  • JavaScript 0.2%
  • Other 0.1%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Yoenn Burban c0806d3b31
Some checks failed
create_preview_sites.yml / fix(metrics): correct CUE metric catalog discrepancies (#25559) (push) Failing after 0s
Static Analysis / static-analysis (push) Has been cancelled
Check Generated VRL Docs Freshness / changes (push) Has been cancelled
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
Check Generated VRL Docs Freshness / run-check-generated-vrl-docs (push) Has been cancelled
Check Generated VRL Docs Freshness / check-generated-vrl-docs (push) Has been cancelled
fix(metrics): correct CUE metric catalog discrepancies (#25559)
* fix(docs): correct buffer byte metric names in CUE documentation

The buffer byte metrics were documented under the old names
`buffer_received_event_bytes_total` and `buffer_sent_event_bytes_total`,
but the code emits `buffer_received_bytes_total` and
`buffer_sent_bytes_total` as defined in
`lib/vector-common/src/internal_event/metric_name.rs`
(BufferReceivedBytesTotal / BufferSentBytesTotal).

Users querying the documented names would find no data.

Fixes the HIGH-severity discrepancy identified in the telemetry
correctness report (section F1).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(docs): remove stale metrics from CUE documentation

The following metrics are documented in the CUE catalog but no longer
emitted by the code (absent from CounterName/GaugeName enums in
lib/vector-common/src/internal_event/metric_name.rs and from all
emit!() / counter!() / gauge!() call sites):

- connection_read_errors_total (syslog.cue)
- utf8_convert_errors_total (syslog.cue)
- stdin_reads_failed_total (sources/stdin.cue, internal_metrics.cue)
- protobuf_decode_errors_total (sources/vector.cue, sinks/vector.cue,
    internal_metrics.cue); replaced by component_errors_total
- send_errors_total (sinks/nats.cue, sinks/redis.cue,
    internal_metrics.cue); replaced by component_errors_total
- streams_total (sinks/loki.cue, internal_metrics.cue)
- invalid_record_total (internal_metrics.cue); replaced by
    component_errors_total
- timestamp_parse_errors_total (internal_metrics.cue); replaced by
    component_errors_total
- config_reload_rejected (internal_metrics.cue); replaced by
    component_errors_total with error_code="reload"
- config_reloaded (internal_metrics.cue); replaced by reloaded_total

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(docs): add missing metrics to CUE documentation

These metrics are emitted by the code (registered in
lib/vector-common/src/internal_event/metric_name.rs) but were absent
from the CUE catalog:

Buffer metrics (lib/vector-buffers/src/internal_events.rs):
- buffer_discarded_bytes_total
- buffer_errors_total

Adaptive concurrency metrics (src/internal_events/adaptive_concurrency.rs):
- adaptive_concurrency_back_pressure
- adaptive_concurrency_past_rtt_mean
- adaptive_concurrency_reached_limit

HTTP client metrics (src/internal_events/http_client.rs):
- http_client_errors_total
- http_client_error_rtt_seconds

Infrastructure metrics:
- active_endpoints (src/internal_events/open.rs)
- sqs_message_defer_succeeded_total (src/internal_events/aws_sqs.rs)
- k8s_event_namespace_annotation_failures_total
    (src/internal_events/kubernetes_logs.rs)
- k8s_event_node_annotation_failures_total
    (src/internal_events/kubernetes_logs.rs)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(docs): correct open_files type and remove phantom output tag from transform_buffer_*

open_files: documented as type "counter" but the code emits it as a
gauge via gauge!(GaugeName::OpenFiles) in
src/internal_events/file.rs. GaugeName is defined in
lib/vector-common/src/internal_event/metric_name.rs.

transform_buffer_*: all seven transform buffer metrics were documented
with an "output" tag. In production, the topology builder always
creates transform channel metrics with output=None
(src/topology/builder.rs: ChannelMetricMetadata::new(Transform, None)),
so the "output" label is never present. The tag has been removed from
all seven transform_buffer_* entries.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(docs): document buffer metric tags in CUE

Add the missing buffer_id, stage, and intentional labels to the internal metrics CUE schema so the documented buffer telemetry matches the runtime emitters.

Co-authored-by: Cursor <cursoragent@cursor.com>

* group all buffer tags in a helper

* add missing error type per codex suggestion

* use proper tags per codex review comment

* fmt

* document _buffer_error_code

* fmt

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-03 14:24:14 +00:00
.cargo chore(internal docs): clarify JEMALLOC_SYS_WITH_LG_PAGE comment (#25435) 2026-05-14 17:33:09 +00:00
.config chore(ci): Reduce test timeout to 2 minutes (#19845) 2024-02-09 20:31:06 +00:00
.github chore(dev): harden merged-lcov artifact upload step (#25549) 2026-06-01 14:53:28 +00:00
benches chore(metrics): introduce enums for metric names (#25342) 2026-05-02 20:51:00 +00:00
changelog.d fix(observability): propagate component span into spawned tasks (#25521) 2026-06-02 17:30:11 +00:00
config feat(sinks): introduce RetryStrategy / Config for http based sinks (#25057) 2026-05-01 13:30:24 +00:00
distribution chore(deps): bump debian from cedb1ef to b6e2a15 in /distribution/docker/debian in the docker-images group (#25468) 2026-05-20 14:53:27 +00:00
docs chore(deps): drop gssapi from default cargo feature (#25256) 2026-05-14 20:04:48 +00:00
lib docs(schemas): improve log_namespace docs and warn on log_schema (#25560) 2026-06-02 19:31:35 +00:00
licenses chore(deps): bump aws crates and resolve RUSTSEC-2026-0002 (#24909) 2026-03-13 20:46:55 +00:00
proto enhancement(api)!: replace custom Health RPC with standard gRPC health service (#25139) 2026-04-13 16:26:51 +00:00
regression chore(ci): unify Dockerfile apt deps into one script (#25436) 2026-05-14 20:20:46 +00:00
rfcs chore(dev): retire the unused Docker dev environment (#25429) 2026-05-14 14:28:23 +00:00
rust-doc feat(external docs): Add VRL crate documentation (#24384) 2025-12-19 21:16:23 +00:00
scripts chore(ci): unify Dockerfile apt deps into one script (#25436) 2026-05-14 20:20:46 +00:00
src docs(schemas): improve log_namespace docs and warn on log_schema (#25560) 2026-06-02 19:31:35 +00:00
testing fix(log_to_metric transform): set not working in log-to-metric transform when all_metrics=true (#20228) 2024-04-09 14:45:35 +00:00
tests fix(observability): propagate component span into spawned tasks (#25521) 2026-06-02 17:30:11 +00:00
tilt chore(deps): upgrade mold linker from 1.2.1/2.0.0 to 2.40.4 (#24947) 2026-03-17 20:54:19 +00:00
vdev fix(vdev): emit breaking: true for .breaking.md changelog fragments (#25556) 2026-06-01 18:45:28 +00:00
website fix(metrics): correct CUE metric catalog discrepancies (#25559) 2026-06-03 14:24:14 +00:00
.dockerignore chore(ci): .dockerignore should exlcude target dirs (#24154) 2025-11-05 14:52:22 +00:00
.editorconfig chore(dev): Standardize on a common maximum line length (#13574) 2022-07-20 12:03:39 -06:00
.gitattributes feat(metrics): support Datadog metric origin metadata (#18405) 2023-09-20 13:22:06 +00:00
.gitignore feat(aws_s3 sink): add Parquet encoder with schema_file and auto infer schema support (#25156) 2026-04-14 22:53:04 +00:00
.markdownlint.jsonc chore(ci): migrate from markdownlint-cli to markdownlint-cli2 (#25081) 2026-03-31 14:54:02 -04:00
.prettierignore chore(dev): add repo-wide prettier config for YAML, JS, TS, and JSON (#25082) 2026-04-02 09:48:09 -04:00
.prettierrc.json chore(dev): add repo-wide prettier config for YAML, JS, TS, and JSON (#25082) 2026-04-02 09:48:09 -04:00
.rustfmt.toml chore(dev): add maxwidth format option (#23985) 2025-10-10 13:41:54 -04:00
AGENTS.md docs(website): convert TOML config examples to YAML (#25163) 2026-04-13 16:08:25 +00:00
AI_POLICY.md chore(internal docs): AI policy follow-ups (#25550) 2026-06-01 16:06:00 +00:00
buf.yaml chore(dev): move VRL-specific crates under lib/vector-vrl/ (#24854) 2026-03-05 21:14:01 +00:00
build.rs fix(dev): resolve build.rs HEAD path in worktrees (#25120) 2026-04-03 19:39:14 +00:00
Cargo.lock chore(deps): bump openssl from 0.10.79 to 0.10.80 in the cargo-security group across 1 directory (#25475) 2026-05-29 18:23:14 +00:00
Cargo.toml fix(observability): propagate component span into spawned tasks (#25521) 2026-06-02 17:30:11 +00:00
CHANGELOG.md chore: Add link to CHANGELOG.md (#8885) 2021-08-26 07:41:21 -06:00
CLAUDE.md feat(dev): introduce AGENTS.md (#23858) 2026-02-20 20:18:43 +00:00
clippy.toml fix(observability): do not hold span guards across await points (#25482) 2026-05-26 08:31:26 +00:00
CODE_OF_CONDUCT.md chore: Brush off markdown files (#12877) 2022-05-26 15:43:13 -07:00
CONTRIBUTING.md chore(internal docs): add AI contribution policy (#25511) 2026-05-27 21:40:46 +00:00
Cross.toml chore(ci): remove release-flags.sh (#24828) 2026-05-07 16:23:08 +00:00
deny.toml chore(deps): bump rand to 0.8.6 and fix RUSTSEC-2026-0097 (#25505) 2026-05-27 19:18:29 +00:00
LICENSE chore: Switch to the MPL 2.0 license (#3654) 2020-09-03 11:20:20 -04:00
LICENSE-3rdparty.csv chore(deps): update zerobus sdk to 2.0.1 (#25438) 2026-05-28 16:53:50 +00:00
license-tool.toml chore(deps): update mongodb crate to 3.3.0 (#24271) 2025-11-19 15:52:47 +00:00
Makefile chore(dev): use lazy variable for CONTAINER_TOOL detection (#25523) 2026-05-29 15:16:47 +00:00
Makefile.packaging chore(ci): use debug profile for k8s e2e build (#25397) 2026-05-08 19:03:05 +00:00
NOTICE chore: Update the NOTICE file (#17430) 2023-05-24 14:51:54 +00:00
PRIVACY.md chore(dev): enforce and fix markdownlint rules (#25038) 2026-03-25 19:11:04 +00:00
README.md docs(internal docs): README e2e badge (#24375) 2025-12-12 17:43:29 +00:00
RELEASES.md chore(dev): enforce and fix markdownlint rules (#25038) 2026-03-25 19:11:04 +00:00
rust-toolchain.toml chore(dev): Upgrade to rust 1.92.0 (#24376) 2025-12-19 18:02:51 +00:00
SECURITY.md chore(internal docs): update security policy structure and content (#25074) 2026-03-30 21:27:02 +00:00
STYLE.md chore(deps): Drop usage of once_cell::LazyLock (#21511) 2024-10-15 20:52:08 +00:00
Tiltfile chore(deps): Update Rust toolchain to 1.83.0 (#22068) 2025-01-02 21:18:37 +00:00
VERSIONING.md chore(dev): enforce and fix markdownlint rules (#25038) 2026-03-25 19:11:04 +00:00

Nightly Integration/E2E Test Suite Component Features

Vector

Quickstart  •   Docs  •   Guides  •   Integrations  •   Chat  •   Download  •   Rust Crate Docs

What is Vector?

Vector is a high-performance, end-to-end (agent & aggregator) observability data pipeline that puts you in control of your observability data. Collect, transform, and route all your logs and metrics to any vendors you want today and any other vendors you may want tomorrow. Vector enables dramatic cost reduction, novel data enrichment, and data security where you need it, not where it is most convenient for your vendors. Additionally, it is open source and up to 10x faster than every alternative in the space.

To get started, follow our quickstart guide or install Vector.

Vector is maintained by Datadog's Community Open Source Engineering team.

Principles

  • Reliable - Built in Rust, Vector's primary design goal is reliability.
  • End-to-end - Deploys as an agent or aggregator. Vector is a complete platform.
  • Unified - Logs, metrics (beta), and traces (coming soon). One tool for all of your data.

Use cases

  • Reduce total observability costs.
  • Transition vendors without disrupting workflows.
  • Enhance data quality and improve insights.
  • Consolidate agents and eliminate agent fatigue.
  • Improve overall observability performance and reliability.

Community

  • Vector is relied on by startups and enterprises like Atlassian, T-Mobile, Comcast, Zendesk, Discord, Fastly, CVS, Trivago, Tuple, Douban, Visa, Mambu, Blockfi, Claranet, Instacart, Forcepoint, and many more.
  • Vector is downloaded over 100,000 times per day.
  • Vector's largest user processes over 500TB daily.
  • Vector has over 500 contributors and growing.

Documentation

All user documentation is available at vector.dev/docs.

Other Resources:

Comparisons

Performance

The following performance tests demonstrate baseline performance between common protocols with the exception of the Regex Parsing test.

Test Vector Filebeat FluentBit FluentD Logstash SplunkUF SplunkHF
TCP to Blackhole 86mib/s n/a 64.4mib/s 27.7mib/s 40.6mib/s n/a n/a
File to TCP 76.7mib/s 7.8mib/s 35mib/s 26.1mib/s 3.1mib/s 40.1mib/s 39mib/s
Regex Parsing 13.2mib/s n/a 20.5mib/s 2.6mib/s 4.6mib/s n/a 7.8mib/s
TCP to HTTP 26.7mib/s n/a 19.6mib/s <1mib/s 2.7mib/s n/a n/a
TCP to TCP 69.9mib/s 5mib/s 67.1mib/s 3.9mib/s 10mib/s 70.4mib/s 7.6mib/s

To learn more about our performance tests, please see the Vector test harness.

Correctness

The following correctness tests are not exhaustive, but they demonstrate fundamental differences in quality and attention to detail:

Test Vector Filebeat FluentBit FluentD Logstash Splunk UF Splunk HF
Disk Buffer Persistence
File Rotate (create)
File Rotate (copytruncate)
File Truncation
Process (SIGHUP)
JSON (wrapped)

To learn more about our correctness tests, please see the Vector test harness.

Features

Vector is an end-to-end, unified, open data platform.

Vector Beats Fluentbit Fluentd Logstash Splunk UF Splunk HF Telegraf
End-to-end
Agent
Aggregator
Unified
Logs
Metrics
Traces 🚧
Open
Open-source
Vendor-neutral
Reliability
Memory-safe
Delivery guarantees
Multi-core

⚠ = Not interoperable, metrics are represented as structured logs


Developed with ❤️ by Datadog - Security Policy - Privacy Policy