#

Version ArtifactHub License Slack X Reddit

Victoria Metrics Operator

Prerequisites #

  • Install the follow packages: git, kubectl, helm, helm-docs. See this tutorial.
  • PV support on underlying infrastructure.

ArgoCD issues #

When running operator using ArgoCD without Cert Manager (.Values.admissionWebhooks.certManager.enabled: false) it will rerender webhook certificates on each sync since Helm lookup function is not respected by ArgoCD. To prevent this please update you operator Application spec.syncPolicy and spec.ignoreDifferences with a following:

              1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
            
              apiVersion: argoproj.io/v1alpha1
kind: Application
...
spec:
  ...
  destination:
    ...
    namespace: <operator-namespace>
  ...
  syncPolicy:
    syncOptions:
    # https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#respect-ignore-difference-configs
    # argocd must also ignore difference during apply stage
    # otherwise it ll silently override changes and cause a problem
    - RespectIgnoreDifferences=true
  ignoreDifferences:
    - group: ""
      kind: Secret
      name: <fullname>-validation
      namespace: <operator-namespace>
      jsonPointers:
        - /data
    - group: admissionregistration.k8s.io
      kind: ValidatingWebhookConfiguration
      name: <fullname>-admission
      jqPathExpressions:
      - '.webhooks[]?.clientConfig.caBundle'

where <fullname> is output of {{ include "vm-operator.fullname" }} for your setup

Upgrade guide #

During release an issue with helm CRD was discovered. So for upgrade from version less then 0.1.3 you have to two options:

  1. use helm management for CRD, enabled by default.
  2. use own management system, need to add variable: –set createCRD=false.

If you choose helm management, following steps must be done before upgrade:

  1. define namespace and helm release name variables
              1
2
            
              export NAMESPACE=default
export RELEASE_NAME=operator

execute kubectl commands:

              1
2
            
              kubectl get crd  | grep victoriametrics.com | awk '{print $1 }' | xargs -i kubectl label crd {} app.kubernetes.io/managed-by=Helm --overwrite
kubectl get crd  | grep victoriametrics.com | awk '{print $1 }' | xargs -i kubectl annotate crd {} meta.helm.sh/release-namespace="$NAMESPACE" meta.helm.sh/release-name="$RELEASE_NAME"  --overwrite

run helm upgrade command.

Chart Details #

This chart will do the following:

  • Rollout victoria metrics operator

How to install #

Access a Kubernetes cluster.

Setup chart repository (can be omitted for OCI repositories) #

Add a chart helm repository with follow commands:

              1
2
3
4
            
              helm repo add vm https://victoriametrics.github.io/helm-charts/

helm repo update

List versions of vm/victoria-metrics-operator chart available to installation:

              1
2
            
              helm search repo vm/victoria-metrics-operator -l

Install victoria-metrics-operator chart #

Export default values of victoria-metrics-operator chart to file values.yaml:

  • For HTTPS repository

                  1
2
            
                  helm show values vm/victoria-metrics-operator > values.yaml

  • For OCI repository

                  1
2
            
                  helm show values oci://ghcr.io/victoriametrics/helm-charts/victoria-metrics-operator > values.yaml

Change the values according to the need of the environment in values.yaml file.

Test the installation with command:

  • For HTTPS repository

                  1
2
            
                  helm install vmo vm/victoria-metrics-operator -f values.yaml -n NAMESPACE --debug --dry-run

  • For OCI repository

                  1
2
            
                  helm install vmo oci://ghcr.io/victoriametrics/helm-charts/victoria-metrics-operator -f values.yaml -n NAMESPACE --debug --dry-run

Install chart with command:

  • For HTTPS repository

                  1
2
            
                  helm install vmo vm/victoria-metrics-operator -f values.yaml -n NAMESPACE

  • For OCI repository

                  1
2
            
                  helm install vmo oci://ghcr.io/victoriametrics/helm-charts/victoria-metrics-operator -f values.yaml -n NAMESPACE

Get the pods lists by running this commands:

              1
2
            
              kubectl get pods -A | grep 'vmo'

Get the application by running this command:

              1
2
            
              helm list -f vmo -n NAMESPACE

See the history of versions of vmo application with command.

              1
2
            
              helm history vmo -n NAMESPACE

Validation webhook #

Its possible to use validation of created resources with operator. For now, you need cert-manager to easily certificate management https://cert-manager.io/docs/

              1
2
3
4
5
6
            
              admissionWebhooks:
  enabled: true
  # what to do in case, when operator not available to validate request.
  certManager:
    # enables cert creation and injection by cert-manager
    enabled: true

How to uninstall #

Remove application with command.

              1
2
            
              helm uninstall vmo -n NAMESPACE

Documentation of Helm Chart #

Install helm-docs following the instructions on this tutorial.

Generate docs with helm-docs command.

              1
2
3
            
              cd charts/victoria-metrics-operator

helm-docs

The markdown generation is entirely go template driven. The tool parses metadata from charts and generates a number of sub-templates that can be referenced in a template file (by default README.md.gotmpl). If no template file is provided, the tool has a default internal template that will generate a reasonably formatted README.

Disabling automatic ServiceAccount token mount #

There are cases when it is required to disable automatic ServiceAccount token mount due to hardening reasons. To disable it, set the following values:

              1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
            
              serviceAccount:
  automountServiceAccountToken: false

extraVolumes:
  - name: operator
    projected:
      sources:
        - downwardAPI:
            items:
              - fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
                path: namespace
        - configMap:
            name: kube-root-ca.crt
        - serviceAccountToken:
            expirationSeconds: 7200
            path: token

extraVolumeMounts:
  - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    name: operator

This configuration disables the automatic ServiceAccount token mount and mounts the token explicitly.

Enable hostNetwork on operator #

When running managed Kubernetes such as EKS with custom CNI solution like Cilium or Calico, EKS control plane cannot communicate with CNI’s pod CIDR. In that scenario, we need to run webhook service i.e operator with hostNetwork so that it can share node’s network namespace.

              1
            
              hostNetwork: true

Parameters #

The following tables lists the configurable parameters of the chart and their default values.

Change the values according to the need of the environment in victoria-metrics-operator/values.yaml file.

KeyDescription
admissionWebhooks:
    certManager:
        ca:
            commonName: ca.validation.victoriametrics
            duration: 63800h0m0s
            secretTemplate: {}
            subject: {}
        cert:
            commonName: ""
            duration: 45800h0m0s
            secretTemplate: {}
            subject: {}
        enabled: false
        issuer: {}
    enabled: true
    enabledCRDValidation:
        vlogs: true
        vmagent: true
        vmalert: true
        vmalertmanager: true
        vmalertmanagerconfig: true
        vmauth: true
        vmcluster: true
        vmrule: true
        vmsingle: true
        vmuser: true
    keepTLSSecret: true
    policy: Fail
    tls:
        caCert: null
        cert: null
        key: null
(object)

Configures resource validation

admissionWebhooks.certManager:
    ca:
        commonName: ca.validation.victoriametrics
        duration: 63800h0m0s
        secretTemplate: {}
        subject: {}
    cert:
        commonName: ""
        duration: 45800h0m0s
        secretTemplate: {}
        subject: {}
    enabled: false
    issuer: {}
(object)

Enables custom ca bundle, if you are not using cert-manager. In case of custom ca, you have to create secret - {chart-name}-validation with keys: tls.key, tls.crt, ca.crt

admissionWebhooks.certManager.ca:
    commonName: ca.validation.victoriametrics
    duration: 63800h0m0s
    secretTemplate: {}
    subject: {}
(object)

Certificate Authority parameters

admissionWebhooks.certManager.cert:
    commonName: ""
    duration: 45800h0m0s
    secretTemplate: {}
    subject: {}
(object)

Certificate parameters

admissionWebhooks.certManager.enabled: false
(bool)

Enables cert creation and injection by cert-manager.

admissionWebhooks.certManager.issuer: {}
(object)

If needed, provide own issuer. Operator will create self-signed if empty.

admissionWebhooks.enabled: true
(bool)

Enables validation webhook.

admissionWebhooks.policy: Fail
(string)

What to do in case, when operator not available to validate request.

affinity: {}
(object)

Pod affinity

allowedMetricsEndpoints[0]: /metrics
(string)
allowedMetricsEndpoints[1]: /metrics/resources
(string)
annotations: {}
(object)

Annotations to be added to the all resources

crds.annotations: {}
(object)

additional CRD annotations, when .Values.crds.plain: false

crds.cleanup.enabled: false
(bool)

Tells helm to clean up all the vm resources under this release’s namespace when uninstalling

crds.cleanup.image:
    pullPolicy: IfNotPresent
    repository: bitnami/kubectl
    tag: ""
(object)

Image configuration for CRD cleanup Job

crds.cleanup.resources:
    limits:
        cpu: 500m
        memory: 256Mi
    requests:
        cpu: 100m
        memory: 56Mi
(object)

Cleanup hook resources

crds.enabled: true
(bool)

manages CRD creation. Disables CRD creation only in combination with crds.plain: false due to helm dependency conditions limitation

crds.plain: false
(bool)

check if plain or templated CRDs should be created. with this option set to false, all CRDs will be rendered from templates. with this option set to true, all CRDs are immutable and require manual upgrade.

env: []
(list)

Extra settings for the operator deployment. Full list here

envFrom: []
(list)

Specify alternative source for env variables

extraArgs: {}
(object)

Operator container additional commandline arguments

extraContainers: []
(list)

Extra containers to run in a pod with operator

extraHostPathMounts: []
(list)

Additional hostPath mounts

extraLabels: {}
(object)

Labels to be added to the all resources

extraObjects: []
(list)

Add extra specs dynamically to this chart

extraVolumeMounts: []
(list)

Extra Volume Mounts for the container

extraVolumes: []
(list)

Extra Volumes for the pod

fullnameOverride: ""
(string)

Overrides the full name of server component resources

global.cluster.dnsDomain: cluster.local.
(string)

K8s cluster domain suffix, uses for building storage pods’ FQDN. Details are here

global.compatibility:
    openshift:
        adaptSecurityContext: auto
(object)

Openshift security context compatibility configuration

global.image.registry: ""
(string)

Image registry, that can be shared across multiple helm charts

global.imagePullSecrets: []
(list)

Image pull secrets, that can be shared across multiple helm charts

hostNetwork: false
(bool)

Enable hostNetwork on operator deployment

image:
    pullPolicy: IfNotPresent
    registry: ""
    repository: victoriametrics/operator
    tag: ""
    variant: ""
(object)

operator image configuration

image.pullPolicy: IfNotPresent
(string)

Image pull policy

image.registry: ""
(string)

Image registry

image.repository: victoriametrics/operator
(string)

Image repository

image.tag: ""
(string)

Image tag override Chart.AppVersion

imagePullSecrets: []
(list)

Secret to pull images

lifecycle: {}
(object)

Operator lifecycle. See this article for details.

logLevel: info
(string)

VM operator log level. Possible values: info and error.

nameOverride: ""
(string)

Override chart name

nodeSelector: {}
(object)

Pod’s node selector. Details are here

operator.disable_prometheus_converter: false
(bool)

By default, operator converts prometheus-operator objects.

operator.enable_converter_ownership: false
(bool)

Enables ownership reference for converted prometheus-operator objects, it will remove corresponding victoria-metrics objects in case of deletion prometheus one.

operator.prometheus_converter_add_argocd_ignore_annotations: false
(bool)

Compare-options and sync-options for prometheus objects converted by operator for properly use with ArgoCD

operator.useCustomConfigReloader: false
(bool)

Enables custom config-reloader, bundled with operator. It should reduce vmagent and vmauth config sync-time and make it predictable.

podDisruptionBudget:
    enabled: false
    labels: {}
(object)

See kubectl explain poddisruptionbudget.spec for more or check these docs

podLabels: {}
(object)

extra Labels for Pods only

podSecurityContext:
    enabled: true
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
(object)

Pod’s security context. Details are here

priorityClassName: ""
(string)

Name of Priority Class

probe.liveness:
    failureThreshold: 3
    initialDelaySeconds: 5
    periodSeconds: 15
    tcpSocket:
        port: probe
    timeoutSeconds: 5
(object)

Liveness probe

probe.readiness:
    failureThreshold: 3
    httpGet:
        port: probe
    initialDelaySeconds: 5
    periodSeconds: 15
    timeoutSeconds: 5
(object)

Readiness probe

probe.startup: {}
(object)

Startup probe

rbac.aggregatedClusterRoles:
    enabled: true
    labels:
        admin:
            rbac.authorization.k8s.io/aggregate-to-admin: "true"
        view:
            rbac.authorization.k8s.io/aggregate-to-view: "true"
(object)

Create aggregated clusterRoles for CRD readonly and admin permissions

rbac.aggregatedClusterRoles.labels:
    admin:
        rbac.authorization.k8s.io/aggregate-to-admin: "true"
    view:
        rbac.authorization.k8s.io/aggregate-to-view: "true"
(object)

Labels attached to according clusterRole

rbac.create: true
(bool)

Specifies whether the RBAC resources should be created

replicaCount: 1
(int)

Number of operator replicas

resources: {}
(object)

Resource object

securityContext:
    allowPrivilegeEscalation: false
    capabilities:
        drop:
            - ALL
    enabled: true
    readOnlyRootFilesystem: true
(object)

Security context to be added to server pods

service.annotations: {}
(object)

Service annotations

service.clusterIP: ""
(string)

Service ClusterIP

service.externalIPs: ""
(string)

Service external IPs. Check here for details

service.externalTrafficPolicy: ""
(string)

Service external traffic policy. Check here for details

service.healthCheckNodePort: ""
(string)

Health check node port for a service. Check here for details

service.ipFamilies: []
(list)

List of service IP families. Check here for details.

service.ipFamilyPolicy: ""
(string)

Service IP family policy. Check here for details.

service.labels: {}
(object)

Service labels

service.loadBalancerIP: ""
(string)

Service load balancer IP

service.loadBalancerSourceRanges: []
(list)

Load balancer source range

service.servicePort: 8080
(int)

Service port

service.type: ClusterIP
(string)

Service type

service.webhookPort: 9443
(int)

Service webhook port

serviceAccount.automountServiceAccountToken: true
(bool)

Whether to automount the service account token. Note that token needs to be mounted manually if this is disabled.

serviceAccount.create: true
(bool)

Specifies whether a service account should be created

serviceAccount.name: ""
(string)

The name of the service account to use. If not set and create is true, a name is generated using the fullname template

serviceMonitor:
    annotations: {}
    basicAuth: {}
    enabled: false
    extraLabels: {}
    interval: ""
    relabelings: []
    scheme: ""
    scrapeTimeout: ""
    tlsConfig: {}
    vm: true
(object)

Configures monitoring with serviceScrape using either VMServiceScrape or ServiceMonitor. For latter Prometheus Operator CRDs should be preinstalled

serviceMonitor.vm: true
(bool)

Creates VMServiceScrape if true and ServiceMonitor otherwise. Make sure Prometheus Operator CRDs are installed if it’s set to false

terminationGracePeriodSeconds: 30
(int)

Graceful pod termination timeout. See this article for details.

tolerations: []
(list)

Array of tolerations object. Spec is here

topologySpreadConstraints: []
(list)

Pod Topology Spread Constraints. Spec is here

watchNamespaces: []
(list)

By default, the operator will watch all the namespaces If you want to override this behavior, specify the namespace. Operator supports multiple namespaces for watching.

Previous