How to use OpenTelemetry metrics with VictoriaMetrics

VictoriaMetrics supports metrics ingestion with OpenTelemetry metrics format. This guide covers data ingestion via opentelemetry-collector and direct metrics push from application.


Install VictoriaMetrics single-server via helm chart#

Install single-server version:

helm repo add vm
helm repo update 
helm install victoria-metrics vm/victoria-metrics-single

Verify it’s up and running:

kubectl get pods
# victoria-metrics-victoria-metrics-single-server-0   1/1     Running   0          3m1s

Helm chart provides the following urls for reading and writing data:

Write url inside the kubernetes cluster:

Read Data:
 The following url can be used as the datasource url in Grafana:

Using opentelemetry-collector with VictoriaMetrics#

Deploy opentelemetry-collector and configure metrics forwarding#

helm repo add open-telemetry
helm repo update 

# add values
cat << EOF > values.yaml
mode: deployment
  repository: "otel/opentelemetry-collector-contrib"
    enabled: true
     compression: gzip
     encoding: proto
     endpoint: http://victoria-metrics-victoria-metrics-single-server.default.svc.cluster.local:8428/opentelemetry
        insecure: true
        receivers: [otlp]
        processors: []
        exporters: [otlphttp/victoriametrics]

# install helm chart
helm upgrade -i otl-collector open-telemetry/opentelemetry-collector -f values.yaml

# check if pod is healthy
kubectl get pod
NAME                                                     READY   STATUS    RESTARTS   AGE
otl-collector-opentelemetry-collector-7467bbb559-2pq2n   1/1     Running   0          23m

# forward port to local machine to verify metrics are ingested
kubectl port-forward service/victoria-metrics-victoria-metrics-single-server 8428

# check metric `k8s_container_ready` via browser http://localhost:8428/vmui/#/?g0.expr=k8s_container_ready

# forward port to local machine to setup opentelemetry-collector locally
kubectl port-forward otl-collector-opentelemetry-collector 4318

The full version of possible configuration options could be found in OpenTelemetry docs.

Sending to VictoriaMetrics via OpenTelemetry#

Metrics could be sent to VictoriaMetrics via OpenTelemetry instrumentation libraries. You can use any compatible OpenTelemetry instrumentation clients. In our example, we’ll create a WEB server in Golang and instrument it with metrics.

Building the Go application instrumented with metrics#

Copy the go file from here. This will give you a basic implementation of a dice roll WEB server with the urls for opentelemetry-collector pointing to localhost:4318. In the same directory run the following command to create the go.mod file:

go mod init vm/otel

For demo purposes, we’ll add the following dependencies to go.mod file:

require ( v0.52.0 v1.27.0 v1.27.0 v1.27.0 v1.27.0 v1.27.0 v1.27.0

require ( v4.3.0 // indirect v1.0.4 // indirect v1.4.1 // indirectdice.rolls v1.2.2 // indirect v2.20.0 // indirect v1.27.0 // indirect v1.27.0 // indirect v1.2.0 // indirect v0.25.0 // indirect v0.20.0 // indirect v0.15.0 // indirect v0.0.0-20240520151616-dc85e6b867a5 // indirect v0.0.0-20240515191416-fc5f0ca64291 // indirect v1.64.0 // indirect v1.34.1 // indirect

Once you have these in your go.mod file, you can run the following command to download the dependencies:

go mod tidy

Now you can run the application:

go run .

Test metrics ingestion#

By default, the application will be available at localhost:8080. You can start sending requests to /rolldice endpoint to generate metrics. The following command will send 20 requests to the /rolldice endpoint:

for i in `seq 1 20`; do curl http://localhost:8080/rolldice; done

After a few seconds you should start to see metrics sent over to the vmui interface by visiting http://localhost:8428/vmui/#/?g0.expr=dice.rolls in your browser or by querying the metric dice.rolls in the vmui interface.

Direct metrics push#

Metrics could be ingested into VictoriaMetrics directly with HTTP requests. You can use any compatible OpenTelemetry instrumentation clients. In our example, we’ll create a WEB server in Golang and instrument it with metrics.

Building the Go application instrumented with metrics#

See the full source code of the example here.

The list of OpenTelemetry dependencies for go.mod is the following:

go 1.20

require ( v1.7.0 v0.30.0 v0.30.0 v0.30.0 v1.7.0 v0.30.0

Let’s create a new file main.go with basic implementation of the WEB server:

package main

func main() {
  mux := http.NewServeMux()
  mux.HandleFunc("/api/fast", func(writer http.ResponseWriter, request *http.Request) {
    writer.Write([]byte(`fast ok`))
  mux.HandleFunc("/api/slow", func(writer http.ResponseWriter, request *http.Request) {
    time.Sleep(time.Second * 2)
    writer.Write([]byte(`slow ok`))

  mw, err := newMetricsMiddleware(mux)
  if err != nil {
    panic(fmt.Sprintf("cannot build metricMiddleWare: %q", err))

  go func() {
    http.ListenAndServe("localhost:8081", mw)

In the code above, we used newMetricsMiddleware function to create a handler for our server. Let’s define it below:

type metricMiddleWare struct {
	h               http.Handler
	requestsCount   syncint64.Counter
	requestsLatency syncfloat64.Histogram
	activeRequests  int64

func newMetricsMiddleware(h http.Handler) (*metricMiddleWare, error) {
	mw := &metricMiddleWare{h: h}
	mc, err := newMetricsController(ctx)
	if err != nil {
		return nil, fmt.Errorf("cannot build metrics collector: %w", err)

	prov := mc.Meter("")

	mw.requestsLatency, err = prov.SyncFloat64().Histogram("http_request_latency_seconds")
	if err != nil {
		return nil, fmt.Errorf("cannot create histogram: %w", err)
	mw.requestsCount, err = prov.SyncInt64().Counter("http_requests_total")
	if err != nil {
		return nil, fmt.Errorf("cannot create syncInt64 counter: %w", err)
	ar, err := prov.AsyncInt64().Gauge("http_active_requests")
	if err != nil {
		return nil, fmt.Errorf("cannot create AsyncInt64 gauge: %w", err)
	if err := prov.RegisterCallback([]instrument.Asynchronous{ar}, func(ctx context.Context) {
		ar.Observe(ctx, atomic.LoadInt64(&mw.activeRequests))
	}); err != nil {
		return nil, fmt.Errorf("cannot Register int64 gauge: %w", err)

	return mw, nil

The new type metricMiddleWare is instrumented with 3 metrics initialized in newMetricsMiddleware method:

  • counter http_requests_total
  • histogram http_request_latency_seconds
  • gauge http_active_requests

Let’s implement http.Handler interface for metricMiddleWare by adding ServeHTTP method:

func (m *metricMiddleWare) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	t := time.Now()
	path := r.URL.Path
	m.requestsCount.Add(nil, 1, attribute.String("path", path))
	atomic.AddInt64(&m.activeRequests, 1)
	defer func() {
		atomic.AddInt64(&m.activeRequests, -1)
		m.requestsLatency.Record(nil, time.Since(t).Seconds(), attribute.String("path", path))
	m.h.ServeHTTP(w, r)

In method above, our middleware processes received HTTP requests and updates metrics with each new request. But for these metrics to be shipped we need to add a new method newMetricsController to organize metrics collection:

func newMetricsController(ctx context.Context) (*controller.Controller, error) {
	options := []otlpmetrichttp.Option{
		otlpmetrichttp.WithEndpoint("<VictoriaMetrics endpoint - host:port>"),

	metricExporter, err := otlpmetrichttp.New(ctx, options...)
	if err != nil {
		return nil, fmt.Errorf("cannot create otlphttp exporter: %w", err)

	resourceConfig, err := resource.New(ctx, resource.WithAttributes(attribute.String("job", "otlp"), attribute.String("instance", "localhost")))
	if err != nil {
		return nil, fmt.Errorf("cannot create meter resource: %w", err)
	meterController := controller.New(
				histogram.WithExplicitBoundaries([]float64{0.01, 0.05, 0.1, 0.5, 0.9, 1.0, 5.0, 10.0, 100.0}),
		controller.WithCollectPeriod(time.Second * 10),
	if err := meterController.Start(ctx); err != nil {
		return nil, fmt.Errorf("cannot start meter controller: %w", err)
	return meterController, nil

This controller will collect and push collected metrics to VictoriaMetrics address with interval of 10s.

See the full source code of the example here.

Test metrics ingestion#

In order to push metrics of our WEB server to VictoriaMetrics it is necessary to ensure that VictoriaMetrics ingestion endpoint is available locally. In previous steps we already deployed a single-server VictoriaMetrics, so let’s make it available locally:

# port-forward victoriametrics to ingest metrics
kubectl port-forward victoria-metrics-victoria-metrics-single-server-0 8428

Now let’s run our WEB server and call its APIs:

# build and run the app
go run main.go 
2024/03/25 19:27:41 Starting web server...
2024/03/25 19:27:41 web server started at localhost:8081.

# execute few queries with curl
curl http://localhost:8081/api/fast
curl http://localhost:8081/api/slow

Open vmui and query http_requests_total or http_active_requests with metricsql.


  • VictoriaMetrics doesn’t support experimental JSON encoding format.
  • VictoriaMetrics supports only AggregationTemporalityCumulative type for histogram and summary