VictoriaMetrics Alert #

Version ArtifactHub License Slack X Reddit

VictoriaMetrics Alert - executes a list of given MetricsQL expressions (rules) and sends alerts to Alert Manager.

Prerequisites #

Before installing this chart, ensure your environment meets the following requirements:

  • Kubernetes cluster - A running Kubernetes cluster with sufficient resources
  • Helm - Helm package manager installed and configured

Additional requirements depend on your configuration:

  • Persistent storage - Required if you enable persistent volumes for data retention (enabled by default)
  • kubectl - Needed for cluster management and troubleshooting

For installation instructions, refer to the official documentation:

How to install #

Access a Kubernetes cluster.

Setup chart repository (can be omitted for OCI repositories) #

Add a chart helm repository with follow commands:

      helm repo add vm https://victoriametrics.github.io/helm-charts/

helm repo update

List versions of vm/victoria-metrics-alert chart available to installation:

      helm search repo vm/victoria-metrics-alert -l

Install victoria-metrics-alert chart #

Export default values of victoria-metrics-alert chart to file values.yaml:

  • For HTTPS repository

          helm show values vm/victoria-metrics-alert > values.yaml

  • For OCI repository

          helm show values oci://ghcr.io/victoriametrics/helm-charts/victoria-metrics-alert > values.yaml

Change the values according to the need of the environment in values.yaml file.

Note

Consider setting .Values.nameOverride to a small value like vma to avoid hitting resource name limits of 63 characters

Test the installation with command:

  • For HTTPS repository

          helm install vma vm/victoria-metrics-alert -f values.yaml -n NAMESPACE --debug

  • For OCI repository

          helm install vma oci://ghcr.io/victoriametrics/helm-charts/victoria-metrics-alert -f values.yaml -n NAMESPACE --debug

Install chart with command:

  • For HTTPS repository

          helm install vma vm/victoria-metrics-alert -f values.yaml -n NAMESPACE

  • For OCI repository

          helm install vma oci://ghcr.io/victoriametrics/helm-charts/victoria-metrics-alert -f values.yaml -n NAMESPACE

Get the pods lists by running this commands:

      kubectl get pods -A | grep 'vma'

Get the application by running this command:

      helm list -f vma -n NAMESPACE

See the history of versions of vma application with command.

      helm history vma -n NAMESPACE

HA configuration for Alertmanager #

There is no option on this chart to set up Alertmanager with HA mode . To enable the HA configuration, you can use:

How to uninstall #

Remove application with command.

      helm uninstall vma -n NAMESPACE

Documentation of Helm Chart #

Install helm-docs following the instructions on this tutorial .

Generate docs with helm-docs command.

      cd charts/victoria-metrics-alert

helm-docs

The markdown generation is entirely go template driven. The tool parses metadata from charts and generates a number of sub-templates that can be referenced in a template file (by default README.md.gotmpl). If no template file is provided, the tool has a default internal template that will generate a reasonably formatted README.

Parameters #

The following tables lists the configurable parameters of the chart and their default values.

Change the values according to the need of the environment in victoria-metrics-alert/values.yaml file.

KeyDescription
alertmanager.baseURL: ""
(string)

External URL, that alertmanager will expose to receivers

alertmanager.baseURLPrefix: ""
(string)

External URL Prefix, Prefix for the internal routes of web endpoints

alertmanager.cluster:
    gossipInterval: 200ms
    listenAddress: 0.0.0.0:9094
    peerTimeout: 15s
    pushPullInterval: 60s
    settleTimeout: 1m
(object)

Cluster configuration for alertmanager

alertmanager.cluster.gossipInterval: 200ms
(string)

Cluster gossip interval

alertmanager.cluster.listenAddress: 0.0.0.0:9094
(string)

Cluster listen address

alertmanager.cluster.peerTimeout: 15s
(string)

Cluster peer timeout

alertmanager.cluster.pushPullInterval: 60s
(string)

Cluster push/pull interval

alertmanager.cluster.settleTimeout: 1m
(string)

Cluster settle timeout

alertmanager.config:
    global:
        resolve_timeout: 5m
    receivers:
        - name: devnull
    route:
        group_by:
            - alertname
        group_interval: 10s
        group_wait: 30s
        receiver: devnull
        repeat_interval: 24h
(object)

Alertmanager configuration

alertmanager.configMap: ""
(string)

Use existing configmap if specified otherwise .config values will be used

alertmanager.deployment:
    spec:
        strategy:
            type: Recreate
(object)

K8s Deployment specific variables

alertmanager.emptyDir: {}
(object)

Empty dir configuration if persistence is disabled for Alertmanager

alertmanager.enabled: false
(bool)

Create alertmanager resources

alertmanager.envFrom: []
(list)

Specify alternative source for env variables

alertmanager.extraArgs: {}
(object)

Extra command line arguments for container of component

alertmanager.extraContainers: []
(list)

Extra containers to run in a pod with alertmanager

alertmanager.extraHostPathMounts: []
(list)

Additional hostPath mounts

alertmanager.extraVolumeMounts: []
(list)

Extra Volume Mounts for the container

alertmanager.extraVolumes: []
(list)

Extra Volumes for the pod

alertmanager.fullnameOverride: ""
(string)

Override Alertmanager resources fullname

alertmanager.image:
    registry: ""
    repository: prom/alertmanager
    tag: v0.27.0
(object)

Alertmanager image configuration

alertmanager.imagePullSecrets: []
(list)

Image pull secrets

alertmanager.ingress.annotations: {}
(object)

Ingress annotations

alertmanager.ingress.enabled: false
(bool)

Enable deployment of ingress for alertmanager component

alertmanager.ingress.extraLabels: {}
(object)

Ingress extra labels

alertmanager.ingress.hosts:
    - name: alertmanager.local
      path:
        - /
      port: web
(list)

Array of host objects

alertmanager.ingress.ingressClassName: ""
(string)

Ingress controller class name

alertmanager.ingress.pathType: Prefix
(string)

Ingress path type

alertmanager.ingress.tls: []
(list)

Array of TLS objects

alertmanager.initContainers: []
(list)

Additional initContainers to initialize the pod

alertmanager.listenAddress: 0.0.0.0:9093
(string)

Alertmanager listen address

alertmanager.mode: deployment
(string)

Alertmanager mode: deployment, statefulSet

alertmanager.nodeSelector: {}
(object)

Pod’s node selector. Details are here

alertmanager.persistentVolume.accessModes:
    - ReadWriteOnce
(list)

Array of access modes. Must match those of existing PV or dynamic provisioner. Details are here

alertmanager.persistentVolume.annotations: {}
(object)

Persistent volume annotations

alertmanager.persistentVolume.enabled: false
(bool)

Create/use Persistent Volume Claim for alertmanager component. Empty dir if false

alertmanager.persistentVolume.existingClaim: ""
(string)

Existing Claim name. If defined, PVC must be created manually before volume will be bound

alertmanager.persistentVolume.extraLabels: {}
(object)

PVC extra labels

alertmanager.persistentVolume.mountPath: /data
(string)

Mount path. Alertmanager data Persistent Volume mount root path.

alertmanager.persistentVolume.name: ""
(string)

Override Persistent Volume Claim name

alertmanager.persistentVolume.size: 50Mi
(string)

Size of the volume. Better to set the same as resource limit memory property.

alertmanager.persistentVolume.storageClassName: ""
(string)

StorageClass to use for persistent volume. Requires alertmanager.persistentVolume.enabled: true. If defined, PVC created automatically

alertmanager.persistentVolume.subPath: ""
(string)

Mount subpath

alertmanager.podAnnotations: {}
(object)

Alertmanager Pod annotations

alertmanager.podLabels: {}
(object)

Alertmanager Pod labels

alertmanager.podSecurityContext:
    enabled: false
(object)

Pod’s security context. Details are here

alertmanager.priorityClassName: ""
(string)

Name of Priority Class

alertmanager.probe.liveness:
    httpGet:
        path: '{{ ternary "" .app.baseURLPrefix (empty .app.baseURLPrefix) }}/-/healthy'
        port: web
(object)

Liveness probe

alertmanager.probe.readiness:
    httpGet:
        path: '{{ ternary "" .app.baseURLPrefix (empty .app.baseURLPrefix) }}/-/ready'
        port: web
(object)

Readiness probe

alertmanager.probe.startup:
    httpGet:
        path: '{{ ternary "" .app.baseURLPrefix (empty .app.baseURLPrefix) }}/-/ready'
        port: web
(object)

Startup probe

alertmanager.replicaCount: 1
(int)

Replica count

alertmanager.resources: {}
(object)

Resource object. Details are here

alertmanager.retention: 120h
(string)

Alertmanager retention

alertmanager.securityContext:
    enabled: false
(object)

Security context to be added to server pods

alertmanager.service.annotations: {}
(object)

Service annotations

alertmanager.service.clusterIP: ""
(string)

Service ClusterIP

alertmanager.service.externalIPs: []
(list)

Service external IPs. Check here for details

alertmanager.service.externalTrafficPolicy: ""
(string)

Service external traffic policy. Check here for details

alertmanager.service.healthCheckNodePort: ""
(string)

Health check node port for a service. Check here for details

alertmanager.service.ipFamilies: []
(list)

List of service IP families. Check here for details.

alertmanager.service.ipFamilyPolicy: ""
(string)

Service IP family policy. Check here for details.

alertmanager.service.labels: {}
(object)

Service labels

alertmanager.service.loadBalancerIP: ""
(string)

Service load balancer IP

alertmanager.service.loadBalancerSourceRanges: []
(list)

Load balancer source range

alertmanager.service.servicePort: 9093
(int)

Service port

alertmanager.service.type: ClusterIP
(string)

Service type

alertmanager.statefulSet:
    spec:
        podManagementPolicy: OrderedReady
        updateStrategy: {}
(object)

K8s StatefulSet specific variables

alertmanager.statefulSet.spec.podManagementPolicy: OrderedReady
(string)

Deploy order policy for StatefulSet pods

alertmanager.statefulSet.spec.updateStrategy: {}
(object)

StatefulSet update strategy. Check here for details.

alertmanager.templates: {}
(object)

Alertmanager extra templates

alertmanager.tolerations: []
(list)

Node tolerations for server scheduling to nodes with taints. Details are here

alertmanager.topologySpreadConstraints: []
(list)

Pod topologySpreadConstraints

alertmanager.webConfig: {}
(object)

Alertmanager web configuration

extraObjects: []
(list)

Add extra specs dynamically to this chart

global.cluster.dnsDomain: cluster.local.
(string)

K8s cluster domain suffix, uses for building storage pods’ FQDN. Details are here

global.compatibility:
    openshift:
        adaptSecurityContext: auto
(object)

Openshift security context compatibility configuration

global.image.registry: ""
(string)

Image registry, that can be shared across multiple helm charts

global.imagePullSecrets: []
(list)

Image pull secrets, that can be shared across multiple helm charts

license:
    key: ""
    secret:
        key: ""
        name: ""
(object)

Enterprise license key configuration for VictoriaMetrics enterprise. Required only for VictoriaMetrics enterprise. Check docs here, for more information, visit site. Request a trial license here Supported starting from VictoriaMetrics v1.94.0

license.key: ""
(string)

License key

license.secret:
    key: ""
    name: ""
(object)

Use existing secret with license key

license.secret.key: ""
(string)

Key in secret with license key

license.secret.name: ""
(string)

Existing secret name

nameOverride: ""
(string)

Override chart name

server.affinity: {}
(object)

Pod affinity

server.annotations: {}
(object)

Annotations to be added to the deployment

server.config:
    alerts:
        groups: []
(object)

VMAlert configuration

server.configMap: ""
(string)

VMAlert alert rules configuration. Use existing configmap if specified

server.datasource:
    basicAuth:
        password: ""
        username: ""
    bearer:
        token: ""
        tokenFile: ""
    url: ""
(object)

VMAlert reads metrics from source, next section represents its configuration. It can be any service which supports MetricsQL or PromQL.

server.datasource.basicAuth:
    password: ""
    username: ""
(object)

Basic auth for datasource

server.datasource.bearer.token: ""
(string)

Token with Bearer token. You can use one of token or tokenFile. You don’t need to add “Bearer” prefix string

server.datasource.bearer.tokenFile: ""
(string)

Token Auth file with Bearer token. You can use one of token or tokenFile

server.env: []
(list)

Additional environment variables (ex.: secret tokens, flags). Check here for details.

server.envFrom: []
(list)

Specify alternative source for env variables

server.extraArgs:
    envflag.enable: true
    envflag.prefix: VM_
    httpListenAddr: :8880
    loggerFormat: json
    rule:
        - /config/alert-rules.yaml
(object)

Extra command line arguments for container of component

server.extraContainers: []
(list)

Additional containers to run in the same pod

server.extraHostPathMounts: []
(list)

Additional hostPath mounts

server.extraVolumeMounts: []
(list)

Extra Volume Mounts for the container. Expects a lice of volume mounts

server.extraVolumes: []
(list)

Extra Volumes for the pod

server.fullnameOverride: ""
(string)

Override vmalert resources fullname

server.image:
    pullPolicy: IfNotPresent
    registry: ""
    repository: victoriametrics/vmalert
    tag: ""
    variant: ""
(object)

VMAlert image configuration

server.imagePullSecrets: []
(list)

Image pull secrets

server.ingress.annotations: {}
(object)

Ingress annotations

server.ingress.enabled: false
(bool)

Enable deployment of ingress for vmalert component

server.ingress.extraLabels: {}
(object)

Ingress extra labels

server.ingress.hosts:
    - name: vmalert.local
      path:
        - /
      port: http
(list)

Array of host objects

server.ingress.ingressClassName: ""
(string)

Ingress controller class name

server.ingress.pathType: Prefix
(string)

Ingress path type

server.ingress.tls: []
(list)

Array of TLS objects

server.initContainers: []
(list)

Additional initContainers to initialize the pod

server.labels: {}
(object)

Labels to be added to the deployment

server.minReadySeconds: 0
(int)

Specifies the minimum number of seconds for which a newly created Pod should be ready without any of its containers crashing/terminating 0 is the standard k8s default

server.name: ""
(string)

Override default app label name

server.nodeSelector: {}
(object)

Pod’s node selector. Details are here

server.notifier:
    alertmanager:
        basicAuth:
            password: ""
            username: ""
        bearer:
            token: ""
            tokenFile: ""
        url: ""
(object)

Notifier to use for alerts. Multiple notifiers can be enabled by using notifiers section

server.notifier.alertmanager.basicAuth:
    password: ""
    username: ""
(object)

Basic auth for alertmanager

server.notifier.alertmanager.bearer.token: ""
(string)

Token with Bearer token. You can use one of token or tokenFile. You don’t need to add “Bearer” prefix string

server.notifier.alertmanager.bearer.tokenFile: ""
(string)

Token Auth file with Bearer token. You can use one of token or tokenFile

server.notifiers: []
(list)

Additional notifiers to use for alerts

server.podAnnotations: {}
(object)

Annotations to be added to pod

server.podDisruptionBudget:
    enabled: false
    labels: {}
(object)

See kubectl explain poddisruptionbudget.spec for more. Or check docs

server.podLabels: {}
(object)

Pod’s additional labels

server.podSecurityContext:
    enabled: true
(object)

Pod’s security context. Details are here

server.priorityClassName: ""
(string)

Name of Priority Class

server.probe.liveness:
    failureThreshold: 3
    initialDelaySeconds: 5
    periodSeconds: 15
    tcpSocket: {}
    timeoutSeconds: 5
(object)

Liveness probe

server.probe.readiness:
    failureThreshold: 3
    httpGet: {}
    initialDelaySeconds: 5
    periodSeconds: 15
    timeoutSeconds: 5
(object)

Readiness probe

server.probe.startup: {}
(object)

Startup probe

server.remote.read.basicAuth:
    password: ""
    username: ""
(object)

Basic auth for remote read

server.remote.read.bearer:
    token: ""
    tokenFile: ""
(object)

Auth based on Bearer token for remote read

server.remote.read.bearer.token: ""
(string)

Token with Bearer token. You can use one of token or tokenFile. You don’t need to add “Bearer” prefix string

server.remote.read.bearer.tokenFile: ""
(string)

Token Auth file with Bearer token. You can use one of token or tokenFile

server.remote.read.url: ""
(string)

VMAlert remote read URL

server.remote.write.basicAuth:
    password: ""
    username: ""
(object)

Basic auth for remote write

server.remote.write.bearer:
    token: ""
    tokenFile: ""
(object)

Auth based on Bearer token for remote write

server.remote.write.bearer.token: ""
(string)

Token with Bearer token. You can use one of token or tokenFile. You don’t need to add “Bearer” prefix string

server.remote.write.bearer.tokenFile: ""
(string)

Token Auth file with Bearer token. You can use one of token or tokenFile

server.remote.write.url: ""
(string)

VMAlert remote write URL

server.replicaCount: 1
(int)

Replica count

server.resources: {}
(object)

Resource object. Details are here

server.securityContext:
    enabled: true
(object)

Security context to be added to server pods

server.service.annotations: {}
(object)

Service annotations

server.service.clusterIP: None
(string)

Service ClusterIP

server.service.externalIPs: []
(list)

Service external IPs. Check here for details

server.service.externalTrafficPolicy: ""
(string)

Service external traffic policy. Check here for details

server.service.healthCheckNodePort: ""
(string)

Health check node port for a service. Check here for details

server.service.ipFamilies: []
(list)

List of service IP families. Check here for details.

server.service.ipFamilyPolicy: ""
(string)

Service IP family policy. Check here for details.

server.service.labels: {}
(object)

Service labels

server.service.loadBalancerIP: ""
(string)

Service load balancer IP

server.service.loadBalancerSourceRanges: []
(list)

Load balancer source range

server.service.servicePort: 8880
(int)

Service port

server.service.type: ClusterIP
(string)

Service type

server.strategy:
    rollingUpdate:
        maxSurge: 25%
        maxUnavailable: 25%
    type: RollingUpdate
(object)

Deployment strategy, set to standard k8s default

server.tolerations: []
(list)

Node tolerations for server scheduling to nodes with taints. Details are here

server.topologySpreadConstraints: []
(list)

Pod topologySpreadConstraints

server.verticalPodAutoscaler:
    enabled: false
(object)

Vertical Pod Autoscaler

server.verticalPodAutoscaler.enabled: false
(bool)

Use VPA for vmalert

serviceAccount.annotations: {}
(object)

Annotations to add to the service account

serviceAccount.automountToken: true
(bool)

Mount API token to pod directly

serviceAccount.create: true
(bool)

Specifies whether a service account should be created

serviceAccount.name: null
(string)

The name of the service account to use. If not set and create is true, a name is generated using the fullname template

serviceMonitor.annotations: {}
(object)

Service Monitor annotations

serviceMonitor.basicAuth: {}
(object)

Basic auth params for Service Monitor

serviceMonitor.enabled: false
(bool)

Enable deployment of Service Monitor for server component. This is Prometheus operator object

serviceMonitor.extraLabels: {}
(object)

Service Monitor labels

serviceMonitor.metricRelabelings: []
(list)

Service Monitor metricRelabelings

serviceMonitor.relabelings: []
(list)

Service Monitor relabelings