VictoriaMetrics Agent - collects metrics from various sources and stores them to VictoriaMetrics
Prerequisites #
- Install the follow packages:
git,kubectl,helm,helm-docs. See this tutorial .
How to install #
Access a Kubernetes cluster.
Setup chart repository (can be omitted for OCI repositories) #
Add a chart helm repository with follow commands:
helm repo add vm https://victoriametrics.github.io/helm-charts/
helm repo update
List versions of vm/victoria-metrics-agent chart available to installation:
helm search repo vm/victoria-metrics-agent -l
Install victoria-metrics-agent chart
#
Export default values of victoria-metrics-agent chart to file values.yaml:
For HTTPS repository
helm show values vm/victoria-metrics-agent > values.yamlFor OCI repository
helm show values oci://ghcr.io/victoriametrics/helm-charts/victoria-metrics-agent > values.yaml
Change the values according to the need of the environment in values.yaml file.
Test the installation with command:
For HTTPS repository
helm install vma vm/victoria-metrics-agent -f values.yaml -n NAMESPACE --debugFor OCI repository
helm install vma oci://ghcr.io/victoriametrics/helm-charts/victoria-metrics-agent -f values.yaml -n NAMESPACE --debug
Install chart with command:
For HTTPS repository
helm install vma vm/victoria-metrics-agent -f values.yaml -n NAMESPACEFor OCI repository
helm install vma oci://ghcr.io/victoriametrics/helm-charts/victoria-metrics-agent -f values.yaml -n NAMESPACE
Get the pods lists by running this commands:
kubectl get pods -A | grep 'vma'
Get the application by running this command:
helm list -f vma -n NAMESPACE
See the history of versions of vma application with command.
helm history vma -n NAMESPACE
Upgrade guide #
Upgrade to 0.13.0 #
- replace
remoteWriteUrlstoremoteWrite:
Given below config
remoteWriteUrls:
- http://address1/api/v1/write
- http://address2/api/v1/write
should be changed to
remoteWrite:
- url: http://address1/api/v1/write
- url: http://address2/api/v1/write
How to uninstall #
Remove application with command.
helm uninstall vma -n NAMESPACE
Documentation of Helm Chart #
Install helm-docs following the instructions on this tutorial
.
Generate docs with helm-docs command.
cd charts/victoria-metrics-agent
helm-docs
The markdown generation is entirely go template driven. The tool parses metadata from charts and generates a number of sub-templates that can be referenced in a template file (by default README.md.gotmpl). If no template file is provided, the tool has a default internal template that will generate a reasonably formatted README.
Examples #
Daemonset mode #
The vmagent can be deployed as a DaemonSet
, launching one pod per Kubernetes node.
This setup is a suitable alternative to using a Deployment or StatefulSet.
If you are using the VictoriaMetrics Operator
and deploying vmagent as a Custom Resource (CRD), refer to VMAgent - Daemon Set
documentation.
Key Benefits:
- Reduced network traffic for scraping metrics by collecting them locally from each node.
- Distributed scraping load across all nodes.
- Improved resilience. Scraping continues uninterrupted if a vmagent pod fails on one node.
To use DaemonSet mode effectively, the scraping configuration must be adjusted.
Use the spec.nodeName field selector to ensure each vmagent pod scrapes only targets local to its node.
In the kubernetes_sd_configs section use role: pod or role: node.
Using other roles (e.g., endpoints, service, etc.) may result in increased CPU and memory usage and overload of the Kubernetes API server.
Restrictions and Limitations:
- Sharding is not supported.
- PodDisruptionBudget is not supported.
- HorizontalPodAutoscaler is not supported.
- Persistent queue volume must be mounted using extraVolumes and extraVolumeMounts, and must use a hostPath volume source.
- Pod restarts may lead to small gaps in metrics collection, as only a single
vmagentpod is scheduled per node.
Below are three common scrape configurations typically used in DaemonSet mode.
Each configuration ensures metrics are collected only from the local node.
Scraping kubelet (node) metrics:
# values.yaml
mode: daemonSet
env:
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
remoteWrite:
# replace with your remote write url
- url: http://vmsingle-vms-victoria-metrics-k8s-stack:8428/api/v1/write
config:
global:
scrape_interval: 10s
scrape_configs:
- job_name: "kubernetes-nodes"
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
# filter node for local one
- action: keep
source_labels: [__meta_kubernetes_node_name]
regex: "^%{KUBE_NODE_NAME}$"
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
source_labels: [__address__]
regex: ([^:]+)(:[0-9]+)?
replacement: $1:10250
Scraping cAdvisor metrics:
# values.yaml
mode: daemonSet
env:
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
remoteWrite:
# replace with your remote write url
- url: http://vmsingle-vms-victoria-metrics-k8s-stack:8428/api/v1/write
config:
global:
scrape_interval: 10s
scrape_configs:
- job_name: "kubernetes-nodes-cadvisor"
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
metrics_path: /metrics/cadvisor
kubernetes_sd_configs:
- role: node
relabel_configs:
# filter node for local one
- action: keep
source_labels: [__meta_kubernetes_node_name]
regex: "^%{KUBE_NODE_NAME}$"
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
source_labels: [__address__]
regex: ([^:]+)(:[0-9]+)?
replacement: $1:10250
honor_timestamps: false
Scraping pod metrics:
# values.yaml
mode: daemonSet
env:
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
remoteWrite:
# replace with your remote write url
- url: http://vmsingle-vms-victoria-metrics-k8s-stack:8428/api/v1/write
config:
global:
scrape_interval: 10s
scrape_configs:
- job_name: "kubernetes-pods"
kubernetes_sd_configs:
- role: pod
selectors:
# use server side selector for pods
- role: pod
field: spec.nodeName=%{KUBE_NODE_NAME}
relabel_configs:
- action: drop
source_labels: [__meta_kubernetes_pod_container_init]
regex: true
- action: keep_if_equal
source_labels:
[__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number]
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
- source_labels: [__meta_kubernetes_pod_container_name]
target_label: container
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_node_name]
action: replace
target_label: node
Parameters #
The following tables lists the configurable parameters of the chart and their default values.
Change the values according to the need of the environment in victoria-metrics-agent/values.yaml file.