VMAnomaly

VMAnomaly - represents VictoriaMetrics Anomaly Detection configuration Available from v0.60.0 .

The VMAnomaly CRD declaratively defines a desired Anomaly Detection setup to run in a Kubernetes cluster. It provides options to configure sharding , replication , persistent storage.

For each shard of VMAnomaly resource, the Operator deploys a properly configured StatefulSet in the same namespace. Anomaly Detection pods can either use a configuration Secret created from the value specified in spec.configRawYaml, or use an existing secret specified in spec.configSecret.

Configuration #

The operator generates a configuration file for VMAnomaly based on user input at the definition of CRD .

The generated configuration is stored in a Secret created by the operator, using the following naming template: config-vmanomaly-<CRD_NAME>.

This configuration file is mounted at VMAnomaly Pod.

VMAnomaly is enterprise only, license is required for this CRD. Trial license can be requested for VMAnomaly CRD evaluation.

The VMAnomaly CRD exposes connection and concurrency settings through typed spec.reader, spec.writer, and spec.monitoring fields. Query definitions and query-level business policies remain in spec.configRawYaml, spec.configSecret, or selected VMAnomalyConfig resources. Models, schedulers, and service settings are also defined in the raw configuration:

      apiVersion: operator.victoriametrics.com/v1
kind: VMAnomaly
metadata:
  name: example
spec:
  replicaCount: 2
  license:
    key: "xx"
  configRawYaml: |
    reader:
      queries:
        ingestion_rate:
          expr: 'sum(rate(vm_rows_inserted_total[5m])) by (type) > 0'
          step: '1m'
          data_range: [0, 'inf']
          detection_direction: above_expected
          min_rel_dev_from_expected: [0, 15]
    schedulers:
      scheduler_periodic_1m:
        class: "periodic"
        # class: "periodic" # or class: "scheduler.periodic.PeriodicScheduler" until v1.13.0 with class alias support)
        infer_every: "1m"
        fit_every: "1000d" # bootstrap-only; online models update from every inference
        fit_window: "3h"
    models:
      model_univariate_1:
        class: 'zscore_online'
        z_threshold: 2.5
        decay: 0.99 # forgetting factor; lower values adapt faster to recent data
    settings:
      n_workers: 8
      native_threads_per_worker: 1
  reader:
    datasourceURL: http://vmsingle-read-example:8428
    samplingPeriod: 10s
    workers: 0
  writer:
    datasourceURL: http://vmsingle-write-example:8428
    batchMaxSeries: 1000
    batchMaxBytes: 4194304
    metricPrefixCacheMaxEntries: 10000
  monitoring:
    push:
      url: http://vmsingle-monitoring-example:8428

Reader, Writer and Monitoring #

Anomaly Detection models , schedulers , settings , and reader.queries are defined in spec.configRawYaml or spec.configSecret. Reader connection settings and reader.workers are configured under spec.reader; writer and monitoring connection settings are configured under spec.writer and spec.monitoring.

Starting with vmanomaly v1.30.2, stable KPI policies - data_range, detection_direction, min_dev_from_expected, and min_rel_dev_from_expected - should be configured on each reader.queries.<alias>. Model-level placement remains compatible but is deprecated. settings.native_threads_per_worker remains in the raw settings section because VMAnomaly does not expose typed service settings. A value of 0 selects the automatic limit for both spec.reader.workers and settings.native_threads_per_worker.

Starting with vmanomaly v1.30.3, spec.writer.batchMaxSeries and spec.writer.batchMaxBytes bound VictoriaMetrics import requests, while spec.writer.metricPrefixCacheMaxEntries bounds prepared metric-label prefix caching. Set the cache limit to 0 to disable cross-cycle caching. Rendezvous sharding remains opt-in through spec.extraEnvs with VMANOMALY_SHARDING_STRATEGY=RENDEZVOUS; the existing Operator sharding fields and defaults are unchanged.

This was done to allow to use K8s secrets and configmaps as a source for TLS, basic and bearer secrets for reader, writer and monitoring endpoints. Also structure of this sections differ from Anomaly Detection configuration structure.

Given below reader, writer and monitoring configuration

      reader:
  class: vm
  datasource_url: http://localhost:8428
  sampling_period: 10s
  bearer_token: token
  queries:
    vmb:
      expr: avg(vm_blocks)
      step: 10s
      data_range: ['-inf', 'inf']
      tz: UTC
writer:
  class: vm
  datasource_url: http://localhost:8428/
  tenant_id: 0:0
  metric_format:
    __name__: vmanomaly_$VAR
    for: $QUERY_KEY
    run: test_metric_format
    config: io_vm_single.yaml
  import_json_path: /api/v1/import
  health_path: health
  user: foo
  password: bar
  tls_cert_file: cert-file
  tls_key_file: key-file
monitoring:
  pull:
    addr: 0.0.0.0
    port: 8080
  push:
    url: http://localhost:8480/
    tenant_id: 0:0
    user: USERNAME
    password: PASSWORD
    verify_tls: False
    timeout: 5s
    push_frequency: 15m
    extra_labels:
      job: vmanomaly-push
      test: test-1

maps into VMAnomaly CR

      apiVersion: operator.victoriametrics.com/v1
kind: VMAnomaly
metadata:
  name: example
spec:
  reader:
    datasourceURL: http://localhost:8428
    samplingPeriod: 10s
    bearer:
      bearerTokenSecret:
        name: k8s-secret-name
        key: token-secret-key
  writer:
    datasourceURL: http://localhost:8428/
    tenantID: 0:0
    metricFormat:
      name: vmanomaly_$VAR
      for: $QUERY_KEY
      extraLabels:
        run: test_metric_format
        config: io_vm_single.yaml
    importJsonPath: /api/v1/import
    healthPath: health
    basicAuth:
      username:
        name: k8s-basic-auth-secret-name
        key: username-secret-key
      password:
        name: k8s-basic-auth-secret-name
        key: password-secret-key
    tlsConfig:
      cert:
        name: k8s-tls-secret-name
        key: tls-cert-key
      keySecret:
        name: k8s-tls-secret-name
        key: tls-key
  monitoring:
    pull:
      addr: 0.0.0.0
      port: 8080
    push:
      url: http://localhost:8480/
      tenantID: 0:0
      username:
        name: k8s-basic-auth-secret-name
        key: username-secret-key
      password:
        name: k8s-basic-auth-secret-name
        key: password-secret-key
      tlsConfig:
        insecureSkipVerify: true
      timeout: 5s
      pushFrequency: 15m
      extraLabels:
        job: vmanomaly-push
        test: test-1
  configRawYaml: |
    reader:
      queries:
        vmb:
          expr: avg(vm_blocks)
          step: 10s
          data_range: ['-inf', 'inf']
          tz: UTC

Using secret #

Configuration can be defined in a manually created Secret, which must be created before VMAnomaly resource.

Secret selector must be defined at VMAnomaly spec.configSecret option:

      apiVersion: v1
kind: Secret
metadata:
  name: anomaly-config
  labels:
    app: vm-operator
type: Opaque
stringData:
  config.yaml: |
    reader:
      queries:
        ingestion_rate:
          expr: 'sum(rate(vm_rows_inserted_total[5m])) by (type) > 0'
          step: '1m'
    schedulers:
      scheduler_periodic_1m:
        class: "periodic"
        # class: "periodic" # or class: "scheduler.periodic.PeriodicScheduler" until v1.13.0 with class alias support)
        infer_every: "1m"
        fit_every: "1000d" # bootstrap-only; online models update from every inference
        fit_window: "3h"
    models:
      model_univariate_1:
        class: 'zscore_online'
        z_threshold: 2.5
        decay: 0.99 # forgetting factor; lower values adapt faster to recent data

---

apiVersion: operator.victoriametrics.com/v1
kind: VMAnomaly
metadata:
  name: example
spec:
  replicaCount: 2
  license:
    key: "xx"
  reader:
    datasourceURL: http://vmsingle-read-example:8428
    samplingPeriod: 10s
  writer:
    datasourceURL: http://vmsingle-write-example:8428
  configSecret:
    name: anomaly-config
    key: config.yaml

Using inline raw config #

Alternatively configuration can be defined as a spec.configRawYaml property value of VMAnomaly resource:

      apiVersion: operator.victoriametrics.com/v1
kind: VMAnomaly
metadata:
  name: example
spec:
  replicaCount: 2
  reader:
    datasourceURL: http://vmsingle-read-example:8428
    samplingPeriod: 10s
  writer:
    datasourceURL: http://vmsingle-write-example:8428
  license:
    key: "xx"
  configRawYaml: |
    reader:
      queries:
        ingestion_rate:
          expr: 'sum(rate(vm_rows_inserted_total[5m])) by (type) > 0'
          step: '1m'
    schedulers:
      scheduler_periodic_1m:
        class: "periodic"
        # class: "periodic" # or class: "scheduler.periodic.PeriodicScheduler" until v1.13.0 with class alias support)
        infer_every: "1m"
        fit_every: "1000d" # bootstrap-only; online models update from every inference
        fit_window: "3h"
    models:
      model_univariate_1:
        class: 'zscore_online'
        z_threshold: 2.5
        decay: 0.99 # forgetting factor; lower values adapt faster to recent data

If both configSecret and configRawYaml are defined, only configuration from configRawYaml will be used. Values from configSecret will be ignored.

High Availability and Sharding #

Anomaly Detection can be deployed in HA mode and scaled horizontally. Horizontal scaling is achieved by automatically splitting the configuration into shards using the spec.shardCount parameter. The spec.replicaCount parameter defines the number of replicas per shard. The Operator creates one StatefulSet per shard (equal to spec.shardCount), with each StatefulSet containing spec.replicaCount replicas.

      apiVersion: operator.victoriametrics.com/v1
kind: VMAnomaly
metadata:
  name: example
spec:
  replicaCount: 2
  shardCount: 2
  reader:
    datasourceURL: http://vmsingle-read-example:8428
    samplingPeriod: 10s
  writer:
    datasourceURL: http://vmsingle-write-example:8428
  license:
    key: "xx"
  configRawYaml: |
    reader:
      queries:
        ingestion_rate:
          expr: 'sum(rate(vm_rows_inserted_total[5m])) by (type) > 0'
          step: '1m'
        ingestion_failed:
          expr: 'sum(rate(vm_rows_failed[5m])) by (type) > 0'
          step: '1m'
    schedulers:
      scheduler_periodic_1m:
        class: "periodic"
        # class: "periodic" # or class: "scheduler.periodic.PeriodicScheduler" until v1.13.0 with class alias support)
        infer_every: "1m"
        fit_every: "1000d" # bootstrap-only; online models update from every inference
        fit_window: "3h"
    models:
      model_univariate_1:
        class: 'zscore_online'
        z_threshold: 2.5
        decay: 0.99 # forgetting factor; lower values adapt faster to recent data

Dynamic configuration #

VMAnomaly supports discovering additional models, schedulers, and queries using VMAnomalyConfig .

For filtering configuration objects, VMAnomaly uses configNamespaceSelector and configSelector selectors. This enables users to associate multiple VMAnomalyConfigs with multiple VMAnomaly instances (a many-to-many relationship).

It allows configuring object access control across namespaces and different environments. See this doc for the specification of selectors.

In addition to the above selectors, object filtering in a cluster is affected by the field selectAllByDefault of VMAnomaly spec and the WATCH_NAMESPACE environment variable for the operator.

The following rules are applied:

  • If configNamespaceSelector and configSelector are both undefined, then by default select nothing. With option set spec.selectAllByDefault: true, select all objects of the given type.
  • If configNamespaceSelector defined, configSelector is undefined, then all objects are matching at the namespaces for the given configNamespaceSelector.
  • If configNamespaceSelector undefined, configSelector is defined, then all objects in VMAnomaly’s namespaces are matched for the given configSelector.
  • If configNamespaceSelector and configSelector are both defined, then only objects in namespaces matched by configNamespaceSelector for the given configSelector are matching.

Here’s a more visual and more detailed view:

configNamespaceSelectorconfigSelectorselectAllByDefaultWATCH_NAMESPACESelected objects
undefinedundefinedfalseundefinednothing
undefinedundefinedtrueundefinedall objects of given type (...) in the cluster
definedundefinedanyundefinedall objects of given type (...) at namespaces for given configNamespaceSelector
undefineddefinedanyundefinedall objects of given type (...) only at VMAnomaly’s namespace are matching for given configSelector
defineddefinedanyundefinedall objects of given type (...) only at namespaces matched configNamespaceSelector for given configSelector
anyundefinedanydefinedall objects of given type (...) only at VMAnomaly’s namespace
anydefinedanydefinedall objects of given type (...) only at VMAnomaly’s namespace for given configSelector

See this doc for more details about the WATCH_NAMESPACE variable.

Version management #

To set VMAnomaly version add spec.image.tag name from releases

      apiVersion: operator.victoriametrics.com/v1
kind: VMAnomaly
metadata:
  name: example
spec:
  image:
    tag: v1.24.0
    pullPolicy: Always
  # ...

Also, you can specify imagePullSecrets if you are pulling images from private repo:

      apiVersion: operator.victoriametrics.com/v1
kind: VMAnomaly
metadata:
  name: example
spec:
  image:
    tag: v1.24.0
    pullPolicy: Always
  imagePullSecrets:
    - name: my-repo-secret
# ...

Resource management #

You can specify resources for each VMAnomaly resource in the spec section of the VMAnomaly CRD.

      apiVersion: operator.victoriametrics.com/v1
kind: VMAnomaly
metadata:
  name: resources-example
spec:
  # ...
  resources:
    requests:
      memory: 64Mi
      cpu: 250m
    limits:
      memory: 128Mi
      cpu: 500m
  # ...

If these parameters are not specified, then, by default all VMAnomaly pods have resource requests and limits from the default values of the following operator parameters :

  • VM_VMANOMALY_RESOURCE_LIMIT_MEM - default memory limit for VMAnomaly pods,
  • VM_VMANOMALY_RESOURCE_LIMIT_CPU - default memory limit for VMAnomaly pods,
  • VM_VMANOMALY_RESOURCE_REQUEST_MEM - default memory limit for VMAnomaly pods,
  • VM_VMANOMALY_RESOURCE_REQUEST_CPU - default memory limit for VMAnomaly pods.

These default parameters will be used if:

  • VM_VMANOMALY_USEDEFAULTRESOURCES is set to true (default value),
  • VMAnomaly CR doesn’t have resources field in spec section.

Field resources in VMAnomaly spec have higher priority than operator parameters.

If you set VM_VMANOMALY_USEDEFAULTRESOURCES to false and don’t specify resources in VMAnomaly CRD, then VMAnomaly pods will be created without resource requests and limits.

Also, you can specify requests without limits - in this case default values for limits will not be used.

Examples #

Below is an example of VMAnomaly setup with periodic scheduler , online z-score model , that is applied against data extracted using given ingestion_rate query from http://vmsingle-read-example:8428 endpoint

      apiVersion: operator.victoriametrics.com/v1
kind: VMAnomaly
metadata:
  name: example
spec:
  replicaCount: 1
  license:
    key: "xx"
  reader:
    datasourceURL: http://vmsingle-read-example:8428
    samplingPeriod: 10s
  writer:
    datasourceURL: http://vmsingle-write-example:8428
  configRawYaml: |-
    reader:
      queries:
        ingestion_rate:
          expr: 'sum(rate(vm_rows_inserted_total[5m])) by (type) > 0'
          step: '1m'
    schedulers:
      scheduler_periodic_1m:
        class: "periodic"
        # class: "periodic" # or class: "scheduler.periodic.PeriodicScheduler" until v1.13.0 with class alias support)
        infer_every: "1m"
        fit_every: "1000d" # bootstrap-only; online models update from every inference
        fit_window: "3h"
    models:
      model_univariate_1:
        class: 'zscore_online'
        z_threshold: 2.5
        decay: 0.99 # forgetting factor; lower values adapt faster to recent data
Previous VLDistributed Next VMAnomalyConfig