VMSingle

VMSingle represents database for storing metrics. The VMSingle CRD declaratively defines a single-node VM installation to run in a Kubernetes cluster.

For each VMSingle resource, the Operator deploys a properly configured Deployment in the same namespace. The VMSingle Pods are configured to mount an empty dir or PersistentVolumeClaimSpec for storing data. Deployment update strategy set to recreate . No more than one replica allowed.

For each VMSingle resource, the Operator adds Service and VMServiceScrape in the same namespace prefixed with name from VMSingle.metadata.name.

Specification #

You can see the full actual specification of the VMSingle resource in the API docs -> VMSingle .

If you can’t find necessary field in the specification of the custom resource, see Extra arguments section .

Also, you can check out the examples section.

Scraping #

VMSingle supports scraping targets with Available from v0.69.0 :

These objects specify which targets VMSingle should scrape and how to collect metrics, and generate part of VMSingle scrape configuration.

To enable scraping on VMSingle, set spec.ingestOnlyMode to false.

VMSingle uses selectors to filter scrape objects. Selectors are defined using the NamespaceSelector and Selector suffixes for each scrape object type in the VMSingle spec:

serviceScrapeNamespaceSelector and serviceScrapeSelector for VMServiceScrape objects,
podScrapeNamespaceSelector and podScrapeSelector for VMPodScrape objects,
probeNamespaceSelector and probeSelector for VMProbe objects,
staticScrapeNamespaceSelector and staticScrapeSelector for VMStaticScrape objects,
nodeScrapeNamespaceSelector and nodeScrapeSelector for VMNodeScrape objects.
scrapeConfigNamespaceSelector and scrapeConfigSelector for VMScrapeConfig objects.

This enables access control configuration for objects across namespaces. See this doc for selector specifications.

In addition to these selectors, object filtering in a cluster can be done by the selectAllByDefault VMSingle spec field and the operator’s WATCH_NAMESPACE environment variable.

Following rules are applied:

If ...NamespaceSelector and ...Selector are both undefined, nothing is selected. Setting spec.selectAllByDefault: true selects all objects of the given type.
If ...NamespaceSelector is defined and ...Selector is undefined, all objects of the given type in the namespaces matched by ...NamespaceSelector are selected.
If ...NamespaceSelector is undefined and ...Selector is defined, objects of the given type matching ...Selector in the VMSingle namespace are selected.
If ...NamespaceSelector and ...Selector are both defined, objects of the given type matching ...Selector in the namespaces matched by ...NamespaceSelector are selected.
selectAllByDefault has an effect only when both ...NamespaceSelector and ...Selector are undefined.
An empty ...NamespaceSelector ({}) matches all namespaces. If ...NamespaceSelector matches no namespaces, nothing is selected.
If the operator’s WATCH_NAMESPACE is set, ...NamespaceSelector is ignored and objects of the given type are selected from all watched namespaces, not only from the VMSingle namespace. This also applies when only ...Selector is defined.
If the operator’s WATCH_NAMESPACE is set and VMSingle uses the operator-managed ServiceAccount, VMNodeScrape objects are not selected, because node discovery requires cluster-wide access.

Below is a more visual and detailed view:

`...NamespaceSelector`	`...Selector`	`selectAllByDefault`	`WATCH_NAMESPACE`	Selected objects
undefined	undefined	false	any	nothing
undefined	undefined	true	undefined	all objects of the given type in the cluster
defined	undefined	any	undefined	all objects of the given type in the namespaces matched by `...NamespaceSelector`
undefined	defined	any	undefined	objects of the given type matching `...Selector` in the `VMSingle` namespace
defined	defined	any	undefined	objects of the given type matching `...Selector` in the namespaces matched by `...NamespaceSelector`
undefined	undefined	true	defined	all objects of the given type in the watched namespaces
defined	undefined	any	defined	all objects of the given type in the watched namespaces
any	defined	any	defined	objects of the given type matching `...Selector` in the watched namespaces

For more details about the WATCH_NAMESPACE variable, see this doc .

Here are some examples of VMSingle configuration with selectors:

      # select all scrape objects in the cluster
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
  name: select-all
spec:
  # ...
  ingestOnlyMode: false
  selectAllByDefault: true

---
# select all scrape objects in specific namespace (my-namespace)
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
  name: select-ns
spec:
  # ...
  serviceScrapeNamespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: my-namespace
  podScrapeNamespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: my-namespace
  nodeScrapeNamespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: my-namespace
  staticScrapeNamespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: my-namespace
  probeNamespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: my-namespace
  scrapeConfigNamespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: my-namespace
    

High availability #

VMSingle doesn’t support high availability by default, for such purpose use VMCluster instead or duplicate the setup.

Version management #

To set VMSingle version add spec.image.tag name from releases

      apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
  name: example
spec:
  image:
    repository: victoriametrics/victoria-metrics
    tag: v1.110.13
    pullPolicy: Always
  # ...
    

Also, you can specify imagePullSecrets if you are pulling images from private repo:

      apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
  name: example
spec:
  image:
    repository: victoriametrics/victoria-metrics
    tag: v1.110.13
    pullPolicy: Always
  imagePullSecrets:
    - name: my-repo-secret
# ...
    

Resource management #

You can specify resources for each VMSingle resource in the spec section of the VMSingle CRD.

      apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
  name: resources-example
spec:
    # ...
    resources:
        requests:
          memory: "64Mi"
          cpu: "250m"
        limits:
          memory: "128Mi"
          cpu: "500m"
    # ...
    

If these parameters are not specified, then, by default all VMSingle pods have resource requests and limits from the default values of the following operator parameters :

VM_VMSINGLEDEFAULT_RESOURCE_LIMIT_MEM - default memory limit for VMSingle pods,
VM_VMSINGLEDEFAULT_RESOURCE_LIMIT_CPU - default memory limit for VMSingle pods,
VM_VMSINGLEDEFAULT_RESOURCE_REQUEST_MEM - default memory limit for VMSingle pods,
VM_VMSINGLEDEFAULT_RESOURCE_REQUEST_CPU - default memory limit for VMSingle pods.

These default parameters will be used if:

VM_VMSINGLEDEFAULT_USEDEFAULTRESOURCES is set to true (default value),
VMSingle CR doesn’t have resources field in spec section.

Field resources in VMSingle spec have higher priority than operator parameters.

If you set VM_VMSINGLEDEFAULT_USEDEFAULTRESOURCES to false and don’t specify resources in VMSingle CRD, then VMSingle pods will be created without resource requests and limits.

Also, you can specify requests without limits - in this case default values for limits will not be used.

Enterprise features #

VMSingle supports features from VictoriaMetrics Enterprise :

For using Enterprise version of vmsingle you need to:

specify license at spec.license.key or at spec.license.keyRef .
change version of vmsingle to version with -enterprise suffix using Version management .

Downsampling #

Use spec.downsampling to configure Downsampling . Each rule requires offset (how far back to downsample) and interval (target resolution). An optional filter restricts the rule to matching time series. The optional dedupInterval sets -dedup.minScrapeInterval for the instance.

      apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
  name: ent-example
spec:
  license:
    keyRef:
      name: k8s-secret-that-contains-license
      key: key-in-a-secret-that-contains-license
  image:
    tag: v1.110.13-enterprise
  downsampling:
    dedupInterval: 1m
    rules:
      - periods:
          - offset: 30d
            interval: 5m
          - offset: 180d
            interval: 1h
          - offset: 1y
            interval: 6h
      - filter: '{env="prod"}'
        periods:
          - offset: 30d
            interval: 1m
          - offset: 180d
            interval: 10m

  # ...other fields...
    

You can read more about downsampling configuration on the VictoriaMetrics downsampling page .

Retention filters #

Use spec.retentionFilters to configure Retention filters . Each entry requires a MetricsQL label filter and a retention duration. The global spec.retentionPeriod applies to all series that don’t match any filter.

      apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
  name: ent-example
spec:
  license:
    keyRef:
      name: k8s-secret-that-contains-license
      key: key-in-a-secret-that-contains-license
  image:
    tag: v1.110.13-enterprise
  retentionPeriod: "12"
  retentionFilters:
    - filter: '{team="juniors"}'
      retention: 3d
    - filter: '{env=~"dev|staging"}'
      retention: 30d

  # ...other fields...
    

You can read more about retention filters configuration on the VictoriaMetrics retention filters page .

Backup automation #

You can check vmbackupmanager documentation for backup automation. It contains a description of the service and its features. This section covers vmbackupmanager integration in vmoperator.

VMSingle has built-in backup configuration, it uses vmbackupmanager - proprietary tool for backups. It supports incremental backups (hourly, daily, weekly, monthly) with popular object storages (aws s3, google cloud storage).

Here is a complete example for backup configuration:

      apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
  name: example
spec:
  # enabling enterprise features
  license:
    keyRef:
      name: k8s-secret-that-contains-license
      key: key-in-a-secret-that-contains-license
  image:
    tag: v1.110.13-enterprise
  vmBackup:
    # using enterprise features: Backup automation
    # more details about backup automation you can read on https://docs.victoriametrics.com/victoriametrics/vmbackupmanager/
    destination: "s3://your_bucket/folder"
    credentialsSecret:
      name: remote-storage-keys
      key: credentials

  # ...other fields...

---

apiVersion: v1
kind: Secret
metadata:
  name: remote-storage-keys
type: Opaque
stringData:
  credentials: |-
    [default]
    aws_access_key_id = your_access_key_id
    aws_secret_access_key = your_secret_access_key
    

You can read more about backup configuration options and mechanics here

Possible configuration options for backup crd can be found at link

Restoring backups #

There are several ways to restore with vmrestore or vmbackupmanager .

Manually mounting disk #

You have to stop VMSingle by scaling it replicas to zero and manually restore data to the database directory.

Steps:

Edit VMSingle CRD, set replicaCount: 0
Wait until database stops
SSH to some server, where you can mount VMSingle disk and mount it manually
Restore files with vmrestore
Umount disk
Edit VMSingle CRD, set replicaCount: 1
Wait database start

Using VMRestore init container #

Add init container with vmrestore command to VMSingle CRD, example:

      apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
  name: vmsingle
spec:
  # enabling enterprise features
  license:
    keyRef:
      name: k8s-secret-that-contains-license
      key: key-in-a-secret-that-contains-license
  image:
    tag: v1.110.13-enterprise
  vmBackup:
    # using enterprise features: Backup automation
    # more details about backup automation you can read https://docs.victoriametrics.com/victoriametrics/vmbackupmanager/
    destination: "s3://your_bucket/folder"
    credentialsSecret:
      name: remote-storage-keys
      key: credentials

  extraArgs:
    runOnStart: "true"

  initContainers:
    - name: vmrestore
      image: victoriametrics/vmrestore:latest
      volumeMounts:
        - mountPath: /victoria-metrics-data
          name: data
        - mountPath: /etc/vm/creds
          name: secret-remote-storage-keys
          readOnly: true
      args:
        - -storageDataPath=/victoria-metrics-data
        - -src=s3://your_bucket/folder/latest
        - -credsFilePath=/etc/vm/creds/credentials

  # ...other fields...
    

Apply it, and db will be restored from S3
Remove initContainers and apply CRD.

Note that using VMRestore will require adjusting src for each pod because restore will be handled per-pod.

Using VMBackupmanager init container #

Using VMBackupmanager restore in Kubernetes environment is described here .

Advantages of using VMBackupmanager include:

Automatic adjustment of src for each pod when backup is requested
Graceful handling of case when no restore is required - VMBackupmanager will exit with successful status code and won’t prevent pod from starting

Examples #

removePvcAfterDelete: true makes the operator keep an owner reference on the PersistentVolumeClaim, so it’s garbage-collected by Kubernetes when the VMSingle object is deleted. By default (false) the owner reference is stripped and the PVC is kept to preserve existing data.

      apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
  name: example
spec:
  retentionPeriod: "12"
  removePvcAfterDelete: true
  storage:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 50Gi
  extraArgs:
    dedup.minScrapeInterval: 60s
  resources:
    requests:
      memory: 500Mi
      cpu: 500m
    limits:
      memory: 10Gi
      cpu: 5
    

Previous VMServiceScrape Next VMStaticScrape

VMSingle #