Ingestion #

There are three ways to deliver data to VictoriaStack - pick the one that matches the pipeline you already have:

The paths combine: OpenTelemetry Collector or a third-party agent can push to vmagent/vlagent/vtagent, which buffer the data and forward it to the databases. That keeps the existing collection setup and still gets the delivery guarantees of the native agents.

OpenTelemetry #

See the example of the OpenTelemetry Collector configuration below:

      receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  otlp_http/vmetrics: { metrics_endpoint: http://victoriametrics:8428/opentelemetry/v1/metrics }
  otlp_http/vlogs:    { logs_endpoint: http://victorialogs:9428/insert/opentelemetry/v1/logs }
  otlp_http/vtraces:  { traces_endpoint: http://victoriatraces:10428/insert/opentelemetry/v1/traces }

service:
  pipelines:
    logs:    { receivers: [ otlp ], exporters: [ otlp_http/vlogs ] }
    traces:  { receivers: [ otlp ], exporters: [ otlp_http/vtraces ] }
    metrics: { receivers: [ otlp ], exporters: [ otlp_http/vmetrics ] }
    

Here, OpenTelemetry Collector receives signals via the gRPC protocol, and then forwards data to VictoriaStack via HTTP endpoints:

• http://victoriametrics:8428/opentelemetry/v1/metrics for metrics
• http://victorialogs:9428/insert/opentelemetry/v1/logs for logs
• http://victoriatraces:10428/insert/opentelemetry/v1/traces for traces

flowchart LR
    A["OpenTelemetry Collector"] --> |<a href="https://docs.victoriametrics.com/victoriametrics/integrations/opentelemetry/">metrics</a>| M[VictoriaMetrics]
    A --> |<a href="https://docs.victoriametrics.com/victorialogs/data-ingestion/opentelemetry/">logs</a>| L[VictoriaLogs]
    A --> |<a href="https://docs.victoriametrics.com/victoriatraces/data-ingestion/opentelemetry/">traces</a>| T[VictoriaTraces]

Each integration has tweaks you can apply. See the following docs for more details:

• VictoriaMetrics OpenTelemetry Setup
• VictoriaLogs OpenTelemetry Setup
• VictoriaTraces OpenTelemetry Setup

In the Quick Start we also use OpenTelemetry Collector as example.

Agents and protocols #

VictoriaStack supports a wide range of ingestion protocols and agents:

Keeping the agents you already run is the simplest way to start: nothing changes in how telemetry is collected, only where it is sent. Follow the docs linked above when reconfiguring them, since some agents can have their own recommended settings for endpoints, batching, compression, and retries.

For better delivery guarantees and ingestion pipeline control put a native agent between existing agents and stack: vmagent/vlagent/vtagent support the same protocols, buffer the data on disk while a database is unavailable, and replicate it to several installations.

Native ingestion #

Support of various protocols and agents makes it easy to integrate VictoriaStack with existing installations. For improving cost-efficiency, high throughput, reliable on-disk buffering, replication, and traffic compression we recommend using native agents instead or together with existing agents:

• vmagent for metrics
• vlagent for logs
• vtagent for traces

vmagent #

vmagent is an agent for collecting, receiving and forwarding metrics to the specified destinations. Its core features are the following:

• drop-in replacement for Prometheus : vmagent supports Prometheus service discovery , scraping configs , relabeling
• on-disk buffering : stores unsent metrics on the local disk until the remote destination is available
• replication and sharding : allows building highly available and horizontally scalable pipelines for metrics delivery
• uses fewer resources than other agents for collecting and forwarding metrics. In a benchmark scraping 1200 node exporters, vmagent needed 3.2x less CPU and 2.7x less memory than the OpenTelemetry Collector, and 1.6x less CPU and 3.0x less memory than Prometheus 3.x

In addition to the features above, vmagent supports all the same ingestion protocols for metrics as VictoriaMetrics, which makes it a perfect entry point in the delivery pipeline for metrics:

flowchart LR
    PRW["Prometheus<br/>Remote Write"]
    OTel["OpenTelemetry"]

    PRW -->|"metrics"| V
    OTel -->|"metrics"| V

    V["vmagent<br/><hr/>Buffer • Transform • Shard • Replicate"]

    subgraph Destinations
        direction TB
        VM["VictoriaMetrics × N"]
    end

    V -->|"replicated to N destinations"| VM

As an entry point, vmagent can filter , deduplicate , pre-aggregate , rate-limit ingested samples and limit cardinality .

The forwarded traffic is additionally compressed , resulting in 2x-5x network bandwidth reduction. See the blog post Save network costs with VictoriaMetrics remote write protocol .

Use vmagent to make metrics delivery highly available .

vlagent #

vlagent is an agent for collecting logs from various sources and forwarding them to the specified destinations. Its core features are the following:

• collecting logs from K8s pods : native support for collecting logs from Kubernetes Pods
• on-disk buffering and replication : allows building highly available and scalable pipelines for logs delivery
• uses fewer resources than other log collectors, which saves compute on collecting and delivering logs

In addition to the features above, vlagent supports all the same ingestion protocols for logs as VictoriaLogs, which makes it a perfect entry point in the delivery pipeline for logs:

flowchart LR
    FB["Fluent Bit"]
    OTel["OpenTelemetry"]

    FB -->|"logs"| V
    OTel -->|"logs"| V

    V["vlagent<br/><hr/>Buffer • Replicate"]

    subgraph Destinations
        direction TB
        VL["VictoriaLogs × N"]
    end

    V -->|"replicate to N destinations"| VL

Use vlagent to make log delivery highly available .

vtagent #

vtagent is an agent for accepting trace spans from OTLP exporters and forwarding them to the specified destinations. Its core feature is on-disk buffering and replication : that allows building highly available and scalable pipelines for spans delivery.

flowchart LR
    OTel1["OpenTelemetry collector"]
    OTel2["OpenTelemetry exporter"]

    OTel1 -->|"spans"| V
    OTel2 -->|"spans"| V

    V["vtagent<br/><hr/>Buffer • Replicate"]

    subgraph Destinations
        direction TB
        VT["VictoriaTraces × N"]
    end

    V -->|"replicate to N destinations"| VT

Cluster and multitenancy #

Clustered databases accept data on their insert components: vminsert, vlinsert, and vtinsert:

A clustered installation runs multiple replicas of every insert component. Put vmauth in front of them to control access, balance the load, and expose a single ingestion endpoint:

      unauthorized_user:
  url_map:
    # VictoriaMetrics writes
    - src_paths: ["/metrics/insert/.*"]
      drop_src_path_prefix_parts: 1
      url_prefix:
        - "http://vminsert-1:8480"
        - "http://vminsert-2:8480"

    # VictoriaLogs writes
    - src_paths: ["/logs/insert/.*"]
      drop_src_path_prefix_parts: 1
      url_prefix:
        - "http://vlinsert-1:9428"
        - "http://vlinsert-2:9428"

    # VictoriaTraces writes
    - src_paths: [ "/traces/insert/.*" ]
      drop_src_path_prefix_parts: 1
      url_prefix:
        - "http://vtinsert-1:10428"
        - "http://vtinsert-2:10428"
    

With the configuration above, vmauth accepts write requests on a single endpoint and balances them across the replicas:

• http://vmauth:8427/metrics/insert/* for metrics,
• http://vmauth:8427/logs/insert/* for logs,
• http://vmauth:8427/traces/insert/* for traces.

The clustered demo runs exactly this vmauth configuration, and its OpenTelemetry Collector writes to these endpoints.

Tenants are specified with HTTP headers in the same way in all three databases. Set AccountID and ProjectID on the write request, for example in an OpenTelemetry Collector exporter:

      exporters:
  otlp_http/vlogs: { logs_endpoint: http://vmauth:8427/logs/insert/opentelemetry/v1/logs, headers: { AccountID: "0", ProjectID: "0" } }
    

Requests without these headers write to the default tenant 0:0.

Once the data arrives, see Querying for the query languages and APIs of every signal, and UI integrations for building dashboards on top of them.