VMAlertmanager #

VMAlertmanager - represents alertmanager configuration.

The VMAlertmanager CRD declaratively defines a desired Alertmanager setup to run in a Kubernetes cluster. It provides options to configure replication and persistent storage.

For each Alertmanager resource, the Operator deploys a properly configured StatefulSet in the same namespace. The Alertmanager pods are configured to include a Secret called <alertmanager-name> which holds the used configuration file in the key alertmanager.yaml.

When there are two or more configured replicas the Operator runs the Alertmanager instances in high availability mode.

Specification #

You can see the full actual specification of the VMAlertmanager resource in the API docs -> VMAlertManager.

If you can’t find necessary field in the specification of the custom resource, see Extra arguments section.

Also, you can check out the examples section.

Configuration #

The operator generates a configuration file for VMAlertmanager based on user input at the definition of CRD.

Generated config stored at Secret created by the operator, it has the following name template vmalertmanager-CRD_NAME-config.

This configuration file is mounted at VMAlertmanager Pod. A special side-car container tracks its changes and sends config-reload signals to alertmanager container.

The operator generates default configuration with blackhole root route. It needs to properly start alertmanager container with empty configuration.

Using secret #

Basically, you can use the global configuration defined at manually created Secret. This Secret must be created before VMAlertmanager.

Name of the Secret must be defined at VMAlertmanager spec.configSecret option:

apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-config
  labels:
    app: vm-operator
type: Opaque
stringData:
  alertmanager.yaml: |
    global:
      resolve_timeout: 5m
    route:
      receiver: 'webhook'
    receivers:
      - name: 'webhook'
        webhook_configs:
          - url: 'http://alertmanagerwh:30500/'    

---

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
  name: example-alertmanager
spec:
  replicaCount: 2
  configSecret: alertmanager-config

Using inline raw config #

Also, if there is no secret data at configuration, or you just want to redefine some global variables for alertmanager. You can define configuration at spec.configRawYaml section of VMAlertmanager configuration:

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
  name: example-alertmanager
spec:
  replicaCount: 2
  configRawYaml: |
    global:
       resolve_timeout: 5m
    route:
      receiver: 'default'
      group_interval: 5m
      repeat_interval: 12h
    receivers:
    - name: 'default'    

If both configSecret and configRawYaml are defined, only configuration from configRawYaml will be used. Values from configSecret will be ignored.

Using VMAlertmanagerConfig #

See details at VMAlertmanagerConfig.

The CRD specifies which VMAlertmanagerConfigs should be covered by the deployed VMAlertmanager instances based on label selection. The Operator then generates a configuration based on the included VMAlertmanagerConfigs and updates the Secret containing the configuration. It continuously does so for all changes that are made to VMAlertmanagerConfigs or to the VMAlertmanager resource itself.

Main goal of operator - generate safe configuration for alertmanager. In case of any of misconfiguration at VMAlertmanagerConfig operator skips it from config generation and updates VMAlertmanagerConfig Status field with error cause.

Configs are filtered by selectors configNamespaceSelector and configSelector in VMAlertmanager CRD definition. For selecting rules from all namespaces you must specify it to empty value:

spec:
  configSelector: {}
  configNamespaceSelector: {}

VMAlertmanagerConfig objects are generates part of VMAlertmanager configuration.

For filtering rules VMAlertmanager uses selectors configNamespaceSelector and configSelector. It allows configuring rules access control across namespaces and different environments. Specification of selectors you can see in this doc.

In addition to the above selectors, the filtering of objects in a cluster is affected by the field selectAllByDefault of VMAlertmanager spec and environment variable WATCH_NAMESPACE for operator.

Following rules are applied:

• If configNamespaceSelector and configSelector both undefined, then by default select nothing. With option set - spec.selectAllByDefault: true, select all vmalertmanagerconfigs.
• If configNamespaceSelector defined, configSelector undefined, then all vmalertmanagerconfigs are matching at namespaces for given configNamespaceSelector.
• If configNamespaceSelector undefined, configSelector defined, then all vmalertmanagerconfigs at VMAlertmanager’s namespaces are matching for given configSelector.
• If configNamespaceSelector and configSelector both defined, then only vmalertmanagerconfigs at namespaces matched configNamespaceSelector for given configSelector are matching.

Here’s a more visual and more detailed view:

More details about WATCH_NAMESPACE variable you can read in this doc.

Here are some examples of VMAlertmanager configuration with selectors:

# select all config objects in the cluster
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
  name: vmalertmanager-select-all
spec:
  # ...
  selectAllByDefault: true

---

# select all config objects in specific namespace (my-namespace)
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
  name: vmalertmanager-select-ns
spec:
  # ...
  configNamespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: my-namespace

Extra configuration files #

VMAlertmanager specification has the following fields, that can be used to configure without editing raw configuration file:

• spec.templates - list of keys in ConfigMaps, that contains template files for alertmanager, e.g.:

  apiVersion: operator.victoriametrics.com/v1beta1
  kind: VMAlertmanager
  metadata:
    name: example-alertmanager
  spec:
    replicaCount: 2
    templates:
      - name: alertmanager-templates
        key: my-template-1.tmpl
      - name: alertmanager-templates
        key: my-template-2.tmpl
  ---
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: alertmanager-templates
  data:
      my-template-1.tmpl: |
          {{ define "hello" -}}
          hello, Victoria!
          {{- end }}        
      my-template-2.tmpl: """
  

These templates will be automatically added to VMAlertmanager configuration and will be automatically reloaded on changes in source ConfigMap.

• spec.configMaps - list of ConfigMap names (in the same namespace) that will be mounted at VMAlertmanager workload and will be automatically reloaded on changes in source ConfigMap. Mount path is /etc/vm/configs/<configmap-name>.

Behavior without provided config #

If no configuration is provided, operator configures stub configuration with blackhole route.

High Availability #

The final step of the high availability scheme is Alertmanager, when an alert triggers, actually fire alerts against all instances of an Alertmanager cluster.

The Alertmanager, starting with the v0.5.0 release, ships with a high availability mode. It implements a gossip protocol to synchronize instances of an Alertmanager cluster regarding notifications that have been sent out, to prevent duplicate notifications. It is an AP (available and partition tolerant) system. Being an AP system means that notifications are guaranteed to be sent at least once.

The Victoria Metrics Operator ensures that Alertmanager clusters are properly configured to run highly available on Kubernetes.

Version management #

To set VMAlertmanager version add spec.image.tag name from releases

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
  name: example-vmalertmanager
spec:
  image:
    repository: prom/alertmanager
    tag: v0.25.0
    pullPolicy: Always
  # ...

Also, you can specify imagePullSecrets if you are pulling images from private repo:

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
  name: example-vmalertmanager
spec:
  image:
    repository: prom/alertmanager
    tag: v0.25.0
    pullPolicy: Always
  imagePullSecrets:
    - name: my-repo-secret
# ...

Resource management #

You can specify resources for each VMAlertManager resource in the spec section of the VMAlertManager CRD.

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertManager
metadata:
  name: vmalertmanager-resources-example
spec:
    # ...
    resources:
        requests:
          memory: "64Mi"
          cpu: "250m"
        limits:
          memory: "128Mi"
          cpu: "500m"
    # ...

If these parameters are not specified, then, by default all VMAlertManager pods have resource requests and limits from the default values of the following operator parameters:

• VM_VMALERTMANAGER_RESOURCE_LIMIT_MEM - default memory limit for VMAlertManager pods,
• VM_VMALERTMANAGER_RESOURCE_LIMIT_CPU - default memory limit for VMAlertManager pods,
• VM_VMALERTMANAGER_RESOURCE_REQUEST_MEM - default memory limit for VMAlertManager pods,
• VM_VMALERTMANAGER_RESOURCE_REQUEST_CPU - default memory limit for VMAlertManager pods.

These default parameters will be used if:

• VM_VMALERTMANAGER_USEDEFAULTRESOURCES is set to true (default value),
• VMAlertManager CR doesn’t have resources field in spec section.

Field resources in VMAlertManager spec have higher priority than operator parameters.

If you set VM_VMALERTMANAGER_USEDEFAULTRESOURCES to false and don’t specify resources in VMAlertManager CRD, then VMAlertManager pods will be created without resource requests and limits.

Also, you can specify requests without limits - in this case default values for limits will not be used.

Examples #

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
  name: vmalertmanager-example
spec:
  replicaCount: 1
  configRawYaml: |
        global:
          resolve_timeout: 5m
        route:
          group_wait: 30s
          group_interval: 5m
          repeat_interval: 12h
          receiver: 'webhook'
        receivers:
        - name: 'webhook'
          webhook_configs:
          - url: 'http://localhost:30502/'