VictoriaTraces Agent

VictoriaTraces Agent - accepts trace spans via OpenTelemetry protocol and replicates them across multiple VictoriaTraces instances.

Prerequisites #

Before installing this chart, ensure your environment meets the following requirements:

Kubernetes cluster - A running Kubernetes cluster with sufficient resources
Helm - Helm package manager installed and configured

Additional requirements depend on your configuration:

Persistent storage - Required if you enable persistent volumes for data retention (enabled by default)
kubectl - Needed for cluster management and troubleshooting

For installation instructions, refer to the official documentation:

Quick start #

This Helm chart deploys vtagent as a StatefulSet. It receives trace spans via OpenTelemetry protocol from cluster workloads and forwards them to the configured VictoriaTraces destinations. If more than one destination is specified, collected trace spans are replicated to all configured destinations. When a destination is unavailable, trace spans are buffered on disk and sent as soon as the destination is back.

To install single-node version of VictoriaTraces, see these docs .
To install a VictoriaTraces cluster, see these docs .

Chart configuration #

The simplest working configuration includes specifying the remoteWrite array and setting CPU and memory resources for the chart.

Example of a minimal working configuration:

      remoteWrite:
  - url: http://victoria-traces:10428

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi
    

If the url has no path, trace spans are sent to the /insert/native endpoint. If multiple remoteWrite entries are defined, trace spans are replicated to all the specified destinations.

Accepting trace spans #

vtagent accepts trace spans over OTLP/HTTP on the primary HTTP listener (port 10429 by default) at the /insert/opentelemetry/v1/traces path. The chart always creates a Service for it, named vtagent-<release>. The Service is headless by default, so its DNS name resolves to the vtagent pod IPs. To get a regular load-balanced ClusterIP Service instead, unset service.clusterIP:

      service:
  clusterIP: ""
    

To additionally accept trace spans over OTLP/gRPC, set the listen address:

      otlpGRPC:
  listenAddr: :4317
    

The listener accepts plain-text connections unless otlpGRPC.tls is set. To enable TLS on the OTLP/gRPC listener:

      otlpGRPC:
  listenAddr: :4317
  tls: true
  tlsCertFile: /etc/tls/tls.crt
  tlsKeyFile: /etc/tls/tls.key

extraVolumes:
  - name: tls-certs
    secret:
      secretName: tls-secret

extraVolumeMounts:
  - name: tls-certs
    mountPath: /etc/tls
    readOnly: true
    

On-disk buffering #

Trace spans that can’t be delivered are buffered at the data volume, which is an emptyDir by default. The buffer of each remoteWrite destination is limited by maxDiskUsagePerURL. To keep the buffer across pod restarts, enable the persistent volume and size it for all the destinations:

      maxDiskUsagePerURL: 5GiB

persistentVolume:
  enabled: true
  size: 10Gi
    

Basic auth #

If you need to use basic auth, define the needed flags via environment variables as shown below:

      remoteWrite:
  - url: http://victoria-traces:10428

env:
  - name: VM_remoteWrite_basicAuth_password
    valueFrom:
      secretKeyRef:
        name: auth-secret
        key: VT_PASSWORD
  - name: VM_remoteWrite_basicAuth_username
    valueFrom:
      secretKeyRef:
        name: auth-secret
        key: VT_USERNAME
    

or mount secrets into pod and reference them in remoteWrite section:

      remoteWrite:
  - url: http://victoria-traces:10428
    basicAuth.username: <name>
    basicAuth.passwordFile: /path/to/password

extraVolumes:
  - name: basic-auth-secret
    secret:
      secretName: basic-auth-secret

extraVolumeMounts:
  - name: basic-auth-secret
    mountPath: /path/to
    readOnly: true
    

Multitenancy #

To define tenant , use ProjectID and AccountID headers as shown below:

      remoteWrite:
  - url: http://victoria-traces:10428
    headers:
      ProjectID: 12
      AccountID: 42
    

TLS #

To enable TLS verification for the remoteWrite target, you can specify the tls-prefixed flags inside each remoteWrite entry.

At a minimum, you should provide the tlsCAFile path so that the agent can verify the server’s TLS certificate. This is useful when the target endpoint uses a certificate signed by a custom or self-signed Certificate Authority (CA).

      remoteWrite:
  - url: https://victoria-traces:10428
    tlsCAFile: "/etc/tls/ca.crt"

extraVolumes:
  - name: tls-certs
    secret:
      secretName: tls-secret

extraVolumeMounts:
  - name: tls-certs
    mountPath: /etc/tls
    readOnly: true
    

For mutual TLS (mTLS), additionally specify the client certificate and key:

      remoteWrite:
  - url: https://victoria-traces:10428
    tlsCAFile: "/etc/tls/ca.crt"
    tlsCertFile: "/etc/tls/client.crt"
    tlsKeyFile: "/etc/tls/client.key"
    

If you want to disable TLS certificate verification (not recommended in production), you can set tlsInsecureSkipVerify to true.

      remoteWrite:
  - url: https://victoria-traces:10428
    tlsInsecureSkipVerify: true
    

Extra fields #

You can add custom fields to all the trace spans by including the VT-Extra-Fields header in headers section of remote write configuration. For example:

      remoteWrite:
  - url: http://victoria-traces:10428
    headers:
      VT-Extra-Fields:
        zone: us-east1-c
        source: victoria-traces-agent
    

HTTP listen address #

The chart configures the HTTP listen address via .Values.http list. Each entry in the list configures a separate HTTP listener. The primary listener (marked with primary: true) is the one service.servicePort, service.targetPort and the Service Monitor apply to.

To enable TLS on the HTTP listener:

      http:
  - name: https
    primary: true
    value: :10429
    tls: true
    tlsCertFile: /path/to/tls.crt
    tlsKeyFile: /path/to/tls.key
    

Resource naming #

Resources are named vtagent-<release>, which matches the naming used by the VictoriaMetrics operator. Legacy release-name based naming is not supported by this chart.

How to install #

Access a Kubernetes cluster.

Setup chart repository (can be omitted for OCI repositories) #

Add a chart helm repository with follow commands:

      helm repo add vm https://victoriametrics.github.io/helm-charts/

helm repo update

List versions of vm/victoria-traces-agent chart available to installation:

      helm search repo vm/victoria-traces-agent -l

Install `victoria-traces-agent` chart #

Export default values of victoria-traces-agent chart to file values.yaml:

For HTTPS repository

      helm show values vm/victoria-traces-agent > values.yaml

For OCI repository

      helm show values oci://ghcr.io/victoriametrics/helm-charts/victoria-traces-agent > values.yaml

Change the values according to the need of the environment in values.yaml file.

Consider setting .Values.nameOverride to a small value like vta to avoid hitting resource name limits of 63 characters

Test the installation with command:

For HTTPS repository

      helm install vta vm/victoria-traces-agent -f values.yaml -n NAMESPACE --debug --dry-run

For OCI repository

      helm install vta oci://ghcr.io/victoriametrics/helm-charts/victoria-traces-agent -f values.yaml -n NAMESPACE --debug --dry-run

Install chart with command:

For HTTPS repository

      helm install vta vm/victoria-traces-agent -f values.yaml -n NAMESPACE

For OCI repository

      helm install vta oci://ghcr.io/victoriametrics/helm-charts/victoria-traces-agent -f values.yaml -n NAMESPACE

Get the pods lists by running this commands:

      kubectl get pods -A | grep 'vta'

Get the application by running this command:

      helm list -f vta -n NAMESPACE

See the history of versions of vta application with command.

      helm history vta -n NAMESPACE

How to uninstall #

Remove application with command.

      helm uninstall vta -n NAMESPACE

Parameters #

The following tables lists the configurable parameters of the chart and their default values.

Change the values according to the need of the environment in victoria-traces-agent/values.yaml file.

Key	Description
`affinity: {}`	`(object)` Pod affinity
`annotations: {}`	`(object)` Annotations to be added to the deployment
`dnsConfig: {}`	`(object)` Custom DNS config for pod. Details are here
`env: []`	`(list)` Environment variables (ex.: secret tokens)
`extraArgs: envflag.enable: true envflag.prefix: VM_ loggerFormat: json`	`(object)` VTAgent extra command line arguments
`extraObjects: []`	`(list)` Add extra specs dynamically to this chart
`extraVolumeMounts: []`	`(list)` Extra Volume Mounts for the container
`extraVolumes: []`	`(list)` Extra Volumes for the pod
`fullnameOverride: ""`	`(string)` Override resources fullname
`global.cluster.dnsDomain: cluster.local.`	`(string)` K8s cluster domain suffix, used for building storage pods’ FQDN. Details are here
`global.compatibility: openshift: adaptSecurityContext: auto`	`(object)` Openshift security context compatibility configuration
`global.extraAnnotations: {}`	`(object)` Annotations added to all resources
`global.extraLabels: {}`	`(object)` Labels added to all resources
`global.image.registry: ""`	`(string)` Image registry, that can be shared across multiple helm charts
`global.imagePullSecrets: []`	`(list)` Image pull secrets, that can be shared across multiple helm charts
`horizontalPodAutoscaler: enabled: false maxReplicas: 10 metrics: [] minReplicas: 1`	`(object)` Horizontal Pod Autoscaler.
`horizontalPodAutoscaler.enabled: false`	`(bool)` Use HPA for vtagent
`horizontalPodAutoscaler.maxReplicas: 10`	`(int)` Maximum replicas for HPA to use to scale vtagent
`horizontalPodAutoscaler.metrics: []`	`(list)` Metric for HPA to use to scale vtagent
`horizontalPodAutoscaler.minReplicas: 1`	`(int)` Minimum replicas for HPA to use to scale vtagent
`http: - name: http primary: true value: :10429`	`(list)` HTTP listen addresses configuration. Each item configures an HTTP listen address with optional TLS settings. Items are used for Pod ports, command line arguments and Service port generation. OTLP/HTTP trace spans are accepted on the primary listener at `/insert/opentelemetry/v1/traces`.
`image.pullPolicy: IfNotPresent`	`(string)` Image pull policy
`image.registry: ""`	`(string)` Image registry
`image.repository: victoriametrics/vtagent`	`(string)` Image repository
`image.tag: ""`	`(string)` Image tag
`image.variant: ""`	`(string)` Image tag suffix, which is appended to `Chart.AppVersion` if no `image.tag` is defined
`imagePullSecrets: []`	`(list)` Image pull secrets. Overrides `global.imagePullSecrets`
`maxDiskUsagePerURL: 1GiB`	`(string)` Maximum disk usage per remoteWrite URL. When exceeded, old trace spans will be deleted.
`nameOverride: ""`	`(string)` Override chart name
`networkPolicy: annotations: {} egress: [] enabled: false ingress: [] labels: {} policyTypes: []`	`(object)` See `kubectl explain networkpolicy.spec` for more. Details are here
`networkPolicy.annotations: {}`	`(object)` Extra annotations for NetworkPolicy
`networkPolicy.egress: []`	`(list)` Egress rules
`networkPolicy.ingress: []`	`(list)` Ingress rules
`networkPolicy.labels: {}`	`(object)` Extra labels for NetworkPolicy
`networkPolicy.policyTypes: []`	`(list)` Policy types. When empty, they are derived from the configured `ingress` and `egress` rules. With no rules and no policy types Kubernetes treats the policy as `Ingress`, which denies all incoming traffic
`nodeSelector: {}`	`(object)` Pod’s node selector. Details are here
`otlpGRPC.listenAddr: ""`	`(string)` OTLP/gRPC listen address. The listener is disabled when empty. The recommended value is `:4317`
`otlpGRPC.tls: false`	`(bool)` Enable TLS for incoming OTLP/gRPC requests. Requires `tlsCertFile` and `tlsKeyFile`. vtagent enables it by default, so the chart always passes this value explicitly
`otlpGRPC.tlsCertFile: ""`	`(string)` Path to file with TLS certificate for the OTLP/gRPC listener
`otlpGRPC.tlsKeyFile: ""`	`(string)` Path to file with TLS key for the OTLP/gRPC listener
`otlpGRPC.tlsMinVersion: ""`	`(string)` Minimum TLS version for the OTLP/gRPC listener
`persistentVolume.accessModes: - ReadWriteOnce`	`(list)` Array of access modes. Must match those of existing PV or dynamic provisioner. Details are here
`persistentVolume.annotations: {}`	`(object)` Persistent volume annotations
`persistentVolume.enabled: false`	`(bool)` Create/use Persistent Volume Claim for vtagent. Empty dir if false
`persistentVolume.existingClaim: ""`	`(string)` Existing Claim name. If defined, PVC must be created manually before volume will be bound
`persistentVolume.extraLabels: {}`	`(object)` Persistent volume additional labels
`persistentVolume.matchLabels: {}`	`(object)` Bind Persistent Volume by labels. Must match all labels of targeted PV
`persistentVolume.size: 10Gi`	`(string)` Size of the volume. Should be big enough to keep buffers for all remoteWrite destinations, see `maxDiskUsagePerURL`
`persistentVolume.storageClassName: ""`	`(string)` StorageClass to use for persistent volume claim. Empty string means default StorageClass
`persistentVolume.volumeAttributesClassName: null`	`(string)` VolumeAttributesClass to use for persistent volume
`podAnnotations: {}`	`(object)` Annotations to be added to pod
`podLabels: {}`	`(object)` Extra labels for Pods only
`podSecurityContext: enabled: true`	`(object)` Security context to be added to pod
`priorityClassName: ""`	`(string)` Priority class to be assigned to the pod(s)
`remoteWrite: []`	`(list)` List of trace destinations. Trace spans will be replicated to all listed destinations. If the url path is not specified, the trace spans will be sent to the /insert/native endpoint.
`replicaCount: 1`	`(int)` Replica count
`resources: null`	`(string)`
`runtimeClassName: ""`	`(string)` Name of the RuntimeClass used to run the pod, e.g. “gvisor”
`securityContext: enabled: true`	`(object)` Security context to be added to pod’s containers
`service.annotations: {}`	`(object)` Service annotations
`service.clusterIP: None`	`(string)` Service ClusterIP. The Service is headless by default, which gives each vtagent pod a stable DNS name. Set it to an empty string to get a regular ClusterIP Service. It is ignored when `service.type` is not `ClusterIP`
`service.externalIPs: []`	`(list)` Service external IPs. Check here for details
`service.externalTrafficPolicy: ""`	`(string)` Service external traffic policy. Check here for details
`service.extraLabels: {}`	`(object)` Service labels
`service.extraPorts: []`	`(list)` Extra service ports
`service.healthCheckNodePort: ""`	`(string)` Health check node port for a service. Check here for details
`service.internalTrafficPolicy: ""`	`(string)` Service internal traffic policy. Check here for details
`service.ipFamilies: []`	`(list)` List of service IP families. Check here for details.
`service.ipFamilyPolicy: ""`	`(string)` Service IP family policy. Check here for details.
`service.loadBalancerIP: ""`	`(string)` Service load balancer IP
`service.loadBalancerSourceRanges: []`	`(list)` Load balancer source range
`service.selectorLabels: {}`	`(object)` Extra selector labels common for service only
`service.servicePort: ""`	`(string)` Service port
`service.targetPort: ""`	`(string)` Target port of the primary HTTP listener. Defaults to the primary `http` item name
`service.trafficDistribution: ""`	`(string)` Traffic Distribution. Check Traffic distribution
`service.type: ClusterIP`	`(string)` Service type
`serviceAccount.annotations: {}`	`(object)` Annotations to add to the service account
`serviceAccount.automountToken: false`	`(bool)` Mount API token to pod directly. vtagent doesn’t access Kubernetes API, so it is disabled by default
`serviceAccount.create: true`	`(bool)` Specifies whether a service account should be created
`serviceAccount.name: null`	`(string)` The name of the service account to use. If not set and create is true, a name is generated using the fullname template
`serviceMonitor.annotations: {}`	`(object)` Service Monitor annotations
`serviceMonitor.basicAuth: {}`	`(object)` Basic auth params for Service Monitor
`serviceMonitor.enabled: false`	`(bool)` Enable deployment of Service Monitor for vtagent. This is Prometheus operator object
`serviceMonitor.extraLabels: {}`	`(object)` Service Monitor labels
`serviceMonitor.metricRelabelings: []`	`(list)` Service Monitor metricRelabelings
`serviceMonitor.port: ""`	`(string)` Service Monitor port. Uses primary http item name by default
`serviceMonitor.relabelings: []`	`(list)` Service Monitor relabelings
`serviceMonitor.targetPort: ""`	`(string)` Service Monitor target port. Overrides port when set
`tolerations: []`	`(list)` Node tolerations for server scheduling to nodes with taints. Details are here
`topologySpreadConstraints: null`	`(string)` Pod topologySpreadConstraints
`verticalPodAutoscaler: enabled: false`	`(object)` Vertical Pod Autoscaler. Requires VPA CRD (`autoscaling.k8s.io/v1`) to be installed in the cluster. Note that VPA should not be used together with HPA on the same resource metrics (CPU/memory).
`verticalPodAutoscaler.enabled: false`	`(bool)` Use VPA for vtagent

VictoriaTraces Agent #

Prerequisites #

Quick start #

Chart configuration #

Accepting trace spans #

On-disk buffering #

Basic auth #

Multitenancy #

TLS #

Extra fields #

HTTP listen address #

Resource naming #

How to install #

Setup chart repository (can be omitted for OCI repositories) #

Install victoria-traces-agent chart #

How to uninstall #

Parameters #

VictoriaTraces Agent

Install `victoria-traces-agent` chart #