VictoriaMetrics K8s Stack

Kubernetes monitoring on VictoriaMetrics stack. Includes VictoriaMetrics Operator, Grafana dashboards, ServiceScrapes and VMRules

Overview #

This chart is an All-in-one solution to start monitoring kubernetes cluster. It installs multiple dependency charts like grafana , node-exporter , kube-state-metrics and victoria-metrics-operator . Also it installs Custom Resources like VMSingle , VMCluster , VMAgent , VMAlert .

By default, the operator converts all existing prometheus-operator API objects into corresponding VictoriaMetrics Operator objects.

To enable metrics collection for kubernetes this chart installs multiple scrape configurations for kubernetes components like kubelet and kube-proxy, etc. Metrics collection is done by VMAgent . So if want to ship metrics to external VictoriaMetrics database you can disable VMSingle installation by setting vmsingle.enabled to false and setting vmagent.vmagentSpec.remoteWrite.url to your external VictoriaMetrics database.

This chart also installs bunch of dashboards and recording rules from kube-prometheus project.

Configuration #

Configuration of this chart is done through helm values.

Dependencies #

Dependencies can be enabled or disabled by setting enabled to true or false in values.yaml file.

!Important: for dependency charts anything that you can find in values.yaml of dependency chart can be configured in this chart under key for that dependency. For example if you want to configure grafana you can find all possible configuration options in values.yaml and you should set them in values for this chart under grafana: key. For example if you want to configure grafana.persistence.enabled you should set it in values.yaml like this:

      #################################################
###              dependencies               #####
#################################################
# Grafana dependency chart configuration. For possible values refer to https://github.com/grafana/helm-charts/tree/main/charts/grafana#configuration
grafana:
  enabled: true
  persistence:
    type: pvc
    enabled: false
    

VictoriaMetrics components #

This chart installs multiple VictoriaMetrics components using Custom Resources that are managed by victoria-metrics-operator Each resource can be configured using spec of that resource from API docs of victoria-metrics-operator . For example if you want to configure VMAgent you can find all possible configuration options in API docs and you should set them in values for this chart under vmagent.spec key. For example if you want to configure remoteWrite.url you should set it in values.yaml like this:

      vmagent:
  spec:
    remoteWrite:
      - url: "https://insert.vmcluster.domain.com/insert/0/prometheus/api/v1/write"
    

ArgoCD issues #

Operator self signed certificates #

When deploying K8s stack using ArgoCD without Cert Manager (.Values.victoria-metrics-operator.admissionWebhooks.certManager.enabled: false) it will rerender operator’s webhook certificates on each sync since Helm lookup function is not respected by ArgoCD. To prevent this please update you K8s stack Application spec.syncPolicy and spec.ignoreDifferences with a following:

      apiVersion: argoproj.io/v1alpha1
kind: Application
...
spec:
  ...
  destination:
    ...
    namespace: <k8s-stack-namespace>
  ...
  syncPolicy:
    syncOptions:
    # https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#respect-ignore-difference-configs
    # argocd must also ignore difference during apply stage
    # otherwise it ll silently override changes and cause a problem
    - RespectIgnoreDifferences=true
  ignoreDifferences:
    - group: ""
      kind: Secret
      name: <fullname>-validation
      namespace: <k8s-stack-namespace>
      jsonPointers:
        - /data
    - group: admissionregistration.k8s.io
      kind: ValidatingWebhookConfiguration
      name: <fullname>-admission
      jqPathExpressions:
      - '.webhooks[]?.clientConfig.caBundle'
    

where <fullname> is output of {{ include "vm-operator.fullname" }} for your setup

Grafana password changes constantly #

When deploying K8s stack using ArgoCD with Grafana subchart enabled it will update Grafana password on each sync since Helm lookup function is not respected by ArgoCD. To prevent this please update you K8s stack Application spec.syncPolicy and spec.ignoreDifferences with a following:

      apiVersion: argoproj.io/v1alpha1
kind: Application
...
spec:
  ...
  destination:
    ...
    namespace: <k8s-stack-namespace>
  ...
  syncPolicy:
    syncOptions:
    # https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#respect-ignore-difference-configs
    # argocd must also ignore difference during apply stage
    # otherwise it ll silently override changes and cause a problem
    - RespectIgnoreDifferences=true
  ignoreDifferences:
    - kind: "Secret"
      jsonPointers:
        - /data/admin-password
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/template/metadata/annotations/checksum~1secret
    

`metadata.annotations: Too long: must have at most 262144 bytes` on dashboards #

If one of dashboards ConfigMap is failing with error Too long: must have at most 262144 bytes, please make sure you’ve added argocd.argoproj.io/sync-options: ServerSideApply=true annotation to your dashboards:

      defaultDashboards:
  annotations:
    argocd.argoproj.io/sync-options: ServerSideApply=true
    

Resources are not completely removed after chart uninstallation #

This chart uses pre-delete Helm hook to cleanup resources managed by operator, but it’s not supported in ArgoCD and this hook is ignored. To have a control over resources removal please consider using either ArgoCD sync phases and waves or installing operator chart separately

Rules and dashboards #

This chart by default install multiple dashboards and recording rules from kube-prometheus you can disable dashboards with defaultDashboards.enabled: false and experimentalDashboardsEnabled: false and rules can be configured under defaultRules

Adding external dashboards #

By default, this chart uses sidecar in order to provision default dashboards. If you want to add you own dashboards there are two ways to do it:

Add dashboards by creating a ConfigMap. An example ConfigMap:

      apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    grafana_dashboard: "1"
  name: grafana-dashboard
data:
  dashboard.json: |-
      {...}
    

Use init container provisioning. Note that this option requires disabling sidecar and will remove all default dashboards provided with this chart. An example configuration:

      grafana:
  sidecar:
    dashboards:
      enabled: false
  dashboards:
    default:
      vmcluster:
        gnetId: 11176
        revision: 45
        datasource: VictoriaMetrics
    

When using this approach, you can find dashboards for VictoriaMetrics components published here .

Prometheus scrape configs #

This chart installs multiple scrape configurations for kubernetes monitoring. They are configured under #ServiceMonitors section in values.yaml file. For example if you want to configure scrape config for kubelet you should set it in values.yaml like this:

      kubelet:
  enabled: true
  # spec for VMNodeScrape crd
  # https://docs.victoriametrics.com/operator/api/#vmnodescrapespec
  spec:
    interval: "30s"
    

Using externally managed Grafana #

If you want to use an externally managed Grafana instance but still want to use the dashboards provided by this chart you can set .Values.grafana.enabled to false and set .Values.defaultDashboards.enabled to true. This will install the dashboards but will not install Grafana.

For example:

      defaultDashboards:
  enabled: true

grafana:
  enabled: false
    

This will create ConfigMaps with dashboards to be imported into Grafana.

If additional configuration for labels or annotations is needed in order to import dashboard to an existing Grafana you can set .Values.grafana.sidecar.dashboards.additionalDashboardLabels or .Values.grafana.sidecar.dashboards.additionalDashboardAnnotations in values.yaml:

For example:

      defaultDashboards:
  enabled: true
  labels:
    key: value
  annotations:
    key: value
    

Using additional Grafana datasources #

To add additional datasources set them using .Values.defaultDatasources.extra list property. If datasource type is not prometheus or alertmanager please also add it to .Values.grafana.plugins list (even if it belongs to a list of Grafana built-in datasources ):

      defaultDatasources:
  extra:
    - name: Loki
      type: loki
      url: http://loki-gateway.loki.svc.cluster.local
      uid: Loki
      access: proxy
      version: 1
      orgId: 1
      isDefault: false
      editable: false
      jsonData:
        httpMethod: GET
grafana:
  plugins:
    - loki
    

Using alternative image registry #

All images of VictoriaMetrics components are available on Docker Hub and Quay . It is possible to override default image registry for all components deployed by operator and operator itself by using the following values:

      global:
  image:
    registry: quay.io
victoria-metrics-operator:
  env:
    - name: VM_VMALERTMANAGER_ALERTMANAGERDEFAULTBASEIMAGE
      value: prometheus/alertmanager
    

Prerequisites #

Before installing this chart, ensure your environment meets the following requirements:

Kubernetes cluster - A running Kubernetes cluster with sufficient resources
Helm - Helm package manager installed and configured

Additional requirements depend on your configuration:

Persistent storage - Required if you enable persistent volumes for data retention (enabled by default)
kubectl - Needed for cluster management and troubleshooting

For installation instructions, refer to the official documentation:

How to install #

Access a Kubernetes cluster.

Setup chart repository (can be omitted for OCI repositories) #

Add a chart helm repository with follow commands:

      helm repo add vm https://victoriametrics.github.io/helm-charts/

helm repo update

List versions of vm/victoria-metrics-k8s-stack chart available to installation:

      helm search repo vm/victoria-metrics-k8s-stack -l

Install `victoria-metrics-k8s-stack` chart #

Export default values of victoria-metrics-k8s-stack chart to file values.yaml:

For HTTPS repository

      helm show values vm/victoria-metrics-k8s-stack > values.yaml

For OCI repository

      helm show values oci://ghcr.io/victoriametrics/helm-charts/victoria-metrics-k8s-stack > values.yaml

Change the values according to the need of the environment in values.yaml file.

Consider setting .Values.nameOverride to a small value like vmks to avoid hitting resource name limits of 63 characters

Test the installation with command:

For HTTPS repository

      helm install vmks vm/victoria-metrics-k8s-stack -f values.yaml -n NAMESPACE --debug --dry-run

For OCI repository

      helm install vmks oci://ghcr.io/victoriametrics/helm-charts/victoria-metrics-k8s-stack -f values.yaml -n NAMESPACE --debug --dry-run

Install chart with command:

For HTTPS repository

      helm install vmks vm/victoria-metrics-k8s-stack -f values.yaml -n NAMESPACE

For OCI repository

      helm install vmks oci://ghcr.io/victoriametrics/helm-charts/victoria-metrics-k8s-stack -f values.yaml -n NAMESPACE

Get the pods lists by running this commands:

      kubectl get pods -A | grep 'vmks'

Get the application by running this command:

      helm list -f vmks -n NAMESPACE

See the history of versions of vmks application with command.

      helm history vmks -n NAMESPACE

Install operator separately #

To have control over an order of managed resources removal or to be able to remove a whole namespace with managed resources it’s recommended to disable operator in k8s-stack chart (victoria-metrics-operator.enabled: false) and install it separately. To move operator from existing k8s-stack release to a separate one please follow the steps below:

disable cleanup webhook (victoria-metrics-operator.crds.cleanup.enabled: false) and apply changes
disable operator (victoria-metrics-operator.enabled: false) and apply changes
deploy operator separately with crds.plain: true

If you’re planning to delete k8s-stack by a whole namespace removal please consider deploying operator in a separate namespace as due to uncontrollable removal order process can hang if operator is removed before at least one resource it manages.

Install locally (Minikube) #

To run VictoriaMetrics stack locally it’s possible to use Minikube . To avoid dashboards and alert rules issues please follow the steps below:

Run Minikube cluster

      minikube start --container-runtime=containerd --extra-config=scheduler.bind-address=0.0.0.0 --extra-config=controller-manager.bind-address=0.0.0.0 --extra-config=etcd.listen-metrics-urls=http://0.0.0.0:2381
    

Install helm chart

      helm install [RELEASE_NAME] vm/victoria-metrics-k8s-stack -f values.yaml -f values.minikube.yaml -n NAMESPACE --debug --dry-run
    

How to uninstall #

Remove application with command.

      helm uninstall vmks -n NAMESPACE

CRDs created by this chart are not removed by default and should be manually cleaned up:

      kubectl get crd | grep victoriametrics.com | awk '{print $1 }' | xargs -i kubectl delete crd {}
    

Troubleshooting #

If you cannot install helm chart with error configmap already exist. It could happen because of name collisions, if you set too long release name. Kubernetes by default, allows only 63 symbols at resource names and all resource names are trimmed by helm to 63 symbols. To mitigate it, use shorter name for helm chart release name, like:

      # stack - is short enough
helm upgrade -i stack vm/victoria-metrics-k8s-stack
    

Or use override for helm chart release name:

      helm upgrade -i some-very-long-name vm/victoria-metrics-k8s-stack --set fullnameOverride=stack
    

Upgrade guide #

Usually, helm upgrade doesn’t requires manual actions. Just execute command:

      $ helm upgrade [RELEASE_NAME] vm/victoria-metrics-k8s-stack
    

But release with CRD update can only be patched manually with kubectl. Since helm does not perform a CRD update, we recommend that you always perform this when updating the helm-charts version:

      # 1. check the changes in CRD
$ helm show crds vm/victoria-metrics-k8s-stack --version [YOUR_CHART_VERSION] | kubectl diff -f -

# 2. apply the changes (update CRD)
$ helm show crds vm/victoria-metrics-k8s-stack --version [YOUR_CHART_VERSION] | kubectl apply -f - --server-side
    

All other manual actions upgrades listed below:

Upgrade to 0.29.0 #

To provide more flexibility for VMAuth configuration all <component>.vmauth params were moved to vmauth.spec. Also .vm.write and .vm.read variables are available in vmauth.spec, which represent vmsingle, vminsert, externalVM.write and vmsingle, vmselect, externalVM.read parsed URLs respectively.

If your configuration in version < 0.29.0 looked like below:

      vmcluster:
  vmauth:
    vmselect:
      - src_paths:
          - /select/.*
        url_prefix:
          - /
    vminsert:
      - src_paths:
          - /insert/.*
        url_prefix:
          - /
    

In 0.29.0 it should look like:

      vmauth:
  spec:
    unauthorizedAccessConfig:
      - src_paths:
          - '{{ .vm.read.path }}/.*'
        url_prefix:
          - '{{ urlJoin (omit .vm.read "path") }}/'
      - src_paths:
          - '{{ .vm.write.path }}/.*'
        url_prefix:
          - '{{ urlJoin (omit .vm.write "path") }}/'
    

Upgrade to 0.13.0 #

node-exporter starting from version 4.0.0 is using the Kubernetes recommended labels. Therefore you have to delete the daemonset before you upgrade.

      kubectl delete daemonset -l app=prometheus-node-exporter
    

scrape configuration for kubernetes components was moved from vmServiceScrape.spec section to spec section. If you previously modified scrape configuration you need to update your values.yaml
grafana.defaultDashboardsEnabled was renamed to defaultDashboardsEnabled (moved to top level). You may need to update it in your values.yaml

Upgrade to 0.6.0 #

All CRD must be update to the latest version with command:

      kubectl apply -f https://raw.githubusercontent.com/VictoriaMetrics/helm-charts/master/charts/victoria-metrics-k8s-stack/crds/crd.yaml

Upgrade to 0.4.0 #

All CRD must be update to v1 version with command:

      kubectl apply -f https://raw.githubusercontent.com/VictoriaMetrics/helm-charts/master/charts/victoria-metrics-k8s-stack/crds/crd.yaml

Upgrade from 0.2.8 to 0.2.9 #

Update VMAgent crd

command:

      kubectl apply -f https://raw.githubusercontent.com/VictoriaMetrics/operator/v0.16.0/config/crd/bases/operator.victoriametrics.com_vmagents.yaml

Upgrade from 0.2.5 to 0.2.6 #

New CRD added to operator - VMUser and VMAuth, new fields added to exist crd. Manual commands:

      kubectl apply -f https://raw.githubusercontent.com/VictoriaMetrics/operator/v0.15.0/config/crd/bases/operator.victoriametrics.com_vmusers.yaml
kubectl apply -f https://raw.githubusercontent.com/VictoriaMetrics/operator/v0.15.0/config/crd/bases/operator.victoriametrics.com_vmauths.yaml
kubectl apply -f https://raw.githubusercontent.com/VictoriaMetrics/operator/v0.15.0/config/crd/bases/operator.victoriametrics.com_vmalerts.yaml
kubectl apply -f https://raw.githubusercontent.com/VictoriaMetrics/operator/v0.15.0/config/crd/bases/operator.victoriametrics.com_vmagents.yaml
kubectl apply -f https://raw.githubusercontent.com/VictoriaMetrics/operator/v0.15.0/config/crd/bases/operator.victoriametrics.com_vmsingles.yaml
kubectl apply -f https://raw.githubusercontent.com/VictoriaMetrics/operator/v0.15.0/config/crd/bases/operator.victoriametrics.com_vmclusters.yaml
    

Parameters #

The following tables lists the configurable parameters of the chart and their default values.

Change the values according to the need of the environment in victoria-metrics-k8s-stack/values.yaml file.

Key	Description
`additionalVictoriaMetricsMap: {}`	`(object)` deprecated. use extraRules instead
`alertmanager.annotations: {}`	`(object)` VMAlertmanager annotations
`alertmanager.config: receivers: - name: blackhole route: receiver: blackhole`	`(object)` VMAlertmanager configuration
`alertmanager.enabled: true`	`(bool)` Create VMAlertmanager CR
`alertmanager.ingress: annotations: {} enabled: false extraPaths: [] hosts: - alertmanager.domain.com labels: {} path: '{{ .Values.alertmanager.spec.routePrefix \| default "/" }}' pathType: Prefix tls: []`	`(object)` VMAlertmanager ingress configuration
`alertmanager.ingress.extraPaths: []`	`(list)` Extra paths to prepend to every host configuration. This is useful when working with annotation based services.
`alertmanager.labels: {}`	`(object)` VMAlertmanager labels
`alertmanager.monzoTemplate: enabled: true`	`(object)` Better alert templates for slack source
`alertmanager.route: annotations: {} enabled: false extraRules: [] filters: [] hostnames: [] labels: {} matches: - path: type: PathPrefix value: '{{ .Values.alertmanager.spec.routePrefix \| default "/" }}' parentRefs: [] port: '{{ .Values.alertmanager.spec.port }}'`	`(object)` VMAlertmanager route configuration
`alertmanager.route.annotations: {}`	`(object)` HTTPRoute annotations
`alertmanager.route.enabled: false`	`(bool)` Enable deployment of HTTPRoute for alertmanager component
`alertmanager.route.extraRules: []`	`(list)` Extra rules to prepend to route. This is useful when working with annotation based services.
`alertmanager.route.filters: []`	`(list)` Filters for a default rule in HTTPRoute
`alertmanager.route.hostnames: []`	`(list)` Array of hostnames
`alertmanager.route.labels: {}`	`(object)` HTTPRoute extra labels
`alertmanager.route.matches: - path: type: PathPrefix value: '{{ .Values.alertmanager.spec.routePrefix \| default "/" }}'`	`(list)` Matches for a default rule in HTTPRoute
`alertmanager.route.parentRefs: []`	`(list)` HTTPGateway objects refs
`alertmanager.route.port: '{{ .Values.alertmanager.spec.port }}'`	`(string)` Route port
`alertmanager.spec: configSecret: "" externalURL: "" image: tag: v0.32.1 port: "9093" replicaCount: 1 routePrefix: / selectAllByDefault: true`	`(object)` Full spec for VMAlertmanager CRD. Allowed values described here
`alertmanager.spec.configSecret: ""`	`(string)` If this one defined, it will be used for alertmanager configuration and config parameter will be ignored
`alertmanager.templateFiles: {}`	`(object)` Extra alert templates
`alertmanager.useManagedConfig: false`	`(bool)` enable storing .Values.alertmanager.config in VMAlertmanagerConfig instead of k8s Secret. Note: VMAlertmanagerConfig and plain VMAlertmanager config structures are not equal. If you’re migrating existing config, please make sure that `.Values.alertmanager.config`: with `useManagedConfig: false` has structure described here. with `useManagedConfig: true` has structure described here.
`argocdReleaseOverride: ""`	`(string)` If this chart is used in “Argocd” with “releaseName” field then VMServiceScrapes couldn’t select the proper services. For correct working need set value ‘argocdReleaseOverride=$ARGOCD_APP_NAME’
`coreDns.enabled: true`	`(bool)` Enabled CoreDNS metrics scraping
`coreDns.service.enabled: true`	`(bool)` Create service for CoreDNS metrics
`coreDns.service.labels: {}`	`(object)` CoreDNS service labels
`coreDns.service.port: 9153`	`(int)` CoreDNS service port
`coreDns.service.selector: k8s-app: kube-dns`	`(object)` CoreDNS service pod selector
`coreDns.service.targetPort: 9153`	`(int)` CoreDNS service target port
`coreDns.vmScrape: spec: endpoints: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token port: http-metrics jobLabel: app.kubernetes.io/component namespaceSelector: matchNames: []`	`(object)` Spec for VMServiceScrape CRD is here
`defaultDashboards.annotations: {}`	`(object)` Additional annotations for dashboard resources
defaultDashboards.dashboards: alertmanager-overview: enabled: '{{ .Values.alertmanager.enabled }}' controller-manager: enabled: '{{ .Values.kubeControllerManager.enabled }}' grafana-overview: enabled: '{{ .Values.grafana.enabled }}' kubelet: enabled: '{{ .Values.kubelet.enabled }}' node-exporter-full: clusterMetric: node_uname_info enabled: true proxy: enabled: '{{ .Values.kubeProxy.enabled }}' scheduler: enabled: '{{ .Values.kubeScheduler.enabled }}' victoriametrics-cluster: clusterMetric: vm_app_version victoriametrics-operator: clusterMetric: vm_app_version enabled: true victoriametrics-single-node: clusterMetric: vm_app_version victoriametrics-vmagent: clusterMetric: vm_app_version victoriametrics-vmalert: clusterMetric: vm_app_version enabled: true	`(object)` Per-dashboard configuration. Keys match dashboard names from JSON content. Set enabled: false to skip a dashboard. Set clusterMetric to patch the cluster variable query.
`defaultDashboards.enabled: true`	`(bool)` Enable dashboard sync via sync-job
`defaultDashboards.grafanaOperator.enabled: false`	`(bool)` Create GrafanaDashboard CRDs instead of ConfigMaps (requires grafana-operator)
`defaultDashboards.grafanaOperator.spec.allowCrossNamespaceImport: false`	`(bool)`
`defaultDashboards.grafanaOperator.spec.instanceSelector.matchLabels.dashboards: grafana`	`(string)`
`defaultDashboards.labels: {}`	`(object)` Additional labels for dashboard resources (ConfigMaps or GrafanaDashboard CRDs)
defaultDashboards.sources: etcd: enabled: '{{ .Values.kubeEtcd.enabled }}' url: https://raw.githubusercontent.com/monitoring-mixins/website/master/assets/etcd/dashboards/etcd.json kube-prometheus: url: https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/grafana-dashboardDefinitions.yaml kubernetes-system-api-server: enabled: '{{ .Values.kubeApiServer.enabled }}' url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-system-api-server.json kubernetes-system-coredns: enabled: '{{ .Values.coreDns.enabled }}' url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-system-coredns.json kubernetes-views-global: url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-global.json kubernetes-views-namespaces: url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-namespaces.json kubernetes-views-nodes: url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-nodes.json kubernetes-views-pods: url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-pods.json node-exporter-full: enabled: '{{ index .Values "prometheus-node-exporter" "enabled" }}' url: https://raw.githubusercontent.com/rfmoz/grafana-dashboards/master/prometheus/node-exporter-full.json victoriametrics-backupmanager: enabled: '{{ or (not (empty (dig "spec" "vmBackup" "destination" "" .Values.vmsingle))) (not (empty (dig "spec" "storage" "vmBackup" "destination" "" .Values.vmcluster))) }}' url: https://raw.githubusercontent.com/VictoriaMetrics/VictoriaMetrics/master/dashboards/backupmanager.json victoriametrics-cluster: enabled: '{{ .Values.vmcluster.enabled }}' url: https://raw.githubusercontent.com/VictoriaMetrics/VictoriaMetrics/master/dashboards/victoriametrics-cluster.json victoriametrics-operator: enabled: '{{ index .Values "victoria-metrics-operator" "enabled" }}' url: https://raw.githubusercontent.com/VictoriaMetrics/VictoriaMetrics/master/dashboards/operator.json victoriametrics-single-node: enabled: '{{ .Values.vmsingle.enabled }}' url: https://raw.githubusercontent.com/VictoriaMetrics/VictoriaMetrics/master/dashboards/victoriametrics.json victoriametrics-vmagent: enabled: '{{ .Values.vmagent.enabled }}' url: https://raw.githubusercontent.com/VictoriaMetrics/VictoriaMetrics/master/dashboards/vmagent.json victoriametrics-vmalert: enabled: '{{ .Values.vmalert.enabled }}' url: https://raw.githubusercontent.com/VictoriaMetrics/VictoriaMetrics/master/dashboards/vmalert.json	`(object)` Dashboard sources. Set enabled: false on any entry to disable it. Add new keys to extend.
`defaultDatasources.alertmanager: datasources: - access: proxy jsonData: implementation: prometheus name: Alertmanager uid: Alertmanager`	`(object)` List of alertmanager datasources. VMAlertmanager generated `url` will be added to each datasource in template if alertmanager is enabled
`defaultDatasources.extra: []`	`(list)` Configure additional grafana datasources (passed through tpl). Check here for details
`defaultDatasources.grafanaOperator.annotations: {}`	`(object)`
`defaultDatasources.grafanaOperator.enabled: false`	`(bool)` Create datasources as CRDs (requires grafana-operator to be installed)
`defaultDatasources.grafanaOperator.spec.allowCrossNamespaceImport: false`	`(bool)`
`defaultDatasources.grafanaOperator.spec.instanceSelector.matchLabels.dashboards: grafana`	`(string)`
`defaultDatasources.victoriametrics.datasources: - access: proxy isDefault: true name: VictoriaMetrics type: prometheus uid: VictoriaMetrics - access: proxy isDefault: false name: VictoriaMetrics (DS) type: victoriametrics-metrics-datasource uid: VictoriaMetricsDS`	`(list)` List of prometheus compatible datasource configurations. VM `url` will be added to each of them in templates.
defaultRules: alerting: spec: annotations: {} labels: {} annotations: {} enabled: true extraGroupByLabels: [] group: spec: {} groups: alertmanager.rules: rules: {} etcd: alerting: {} enabled: true recording: {} rules: {} general.rules: rules: {} k8s.rules.container_cpu_limits: rules: {} k8s.rules.container_cpu_requests: rules: {} k8s.rules.container_cpu_usage_seconds_total: rules: {} k8s.rules.container_memory_cache: rules: {} k8s.rules.container_memory_limits: rules: {} k8s.rules.container_memory_requests: rules: {} k8s.rules.container_memory_rss: rules: {} k8s.rules.container_memory_swap: rules: {} k8s.rules.container_memory_working_set_bytes: rules: {} k8s.rules.pod_owner: rules: {} kube-apiserver-burnrate.rules: rules: {} kube-apiserver-histogram.rules: rules: {} kube-apiserver-slos: rules: {} kube-prometheus-general.rules: rules: {} kube-prometheus-node-recording.rules: rules: {} kube-scheduler.rules: rules: {} kube-state-metrics: rules: {} kubelet.rules: rules: {} kubernetes-apps: jobNamespaces: kube-state-metrics: .* rules: {} kubernetes-resources: rules: {} kubernetes-storage: jobNamespaces: kubelet: .* rules: {} kubernetes-system: rules: {} kubernetes-system-apiserver: rules: {} kubernetes-system-controller-manager: rules: {} kubernetes-system-kubelet: rules: {} kubernetes-system-scheduler: rules: {} node-exporter: rules: {} node-exporter.rules: rules: {} node-network: rules: {} node.rules: rules: {} vm-health: rules: {} vmagent: rules: {} vmalert: rules: {} vmcluster: rules: {} vmoperator: rules: {} vmsingle: rules: {} jobNamespaces: {} labels: {} recording: spec: annotations: {} labels: {} rule: spec: annotations: {} labels: {} rules: {} runbookUrl: https://runbooks.prometheus-operator.dev/runbooks sources: alertmanager: enabled: '{{ .Values.alertmanager.enabled }}' url: https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/alertmanager-prometheusRule.yaml etcd: enabled: '{{ .Values.kubeEtcd.enabled }}' url: https://raw.githubusercontent.com/monitoring-mixins/website/master/assets/etcd/alerts.yaml kube-state-metrics: enabled: '{{ index .Values "kube-state-metrics" "enabled" }}' url: https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/kubeStateMetrics-prometheusRule.yaml kubernetes: url: https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/kubePrometheus-prometheusRule.yaml kubernetes-control-plane: url: https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/kubernetesControlPlane-prometheusRule.yaml node-exporter: enabled: '{{ index .Values "prometheus-node-exporter" "enabled" }}' url: https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/nodeExporter-prometheusRule.yaml vmagent: enabled: '{{ .Values.vmagent.enabled }}' url: https://raw.githubusercontent.com/VictoriaMetrics/VictoriaMetrics/master/deployment/docker/rules/alerts-vmagent.yml vmalert: enabled: '{{ .Values.vmalert.enabled }}' url: https://raw.githubusercontent.com/VictoriaMetrics/VictoriaMetrics/master/deployment/docker/rules/alerts-vmalert.yml vmcluster: enabled: '{{ .Values.vmcluster.enabled }}' url: https://raw.githubusercontent.com/VictoriaMetrics/VictoriaMetrics/master/deployment/docker/rules/alerts-cluster.yml vmhealth: enabled: '{{ or .Values.vmsingle.enabled .Values.vmcluster.enabled }}' url: https://raw.githubusercontent.com/VictoriaMetrics/VictoriaMetrics/master/deployment/docker/rules/alerts-health.yml vmoperator: enabled: '{{ index .Values "victoria-metrics-operator" "enabled" }}' url: https://raw.githubusercontent.com/VictoriaMetrics/operator/master/config/alerting/vmoperator-rules.yaml vmsingle: enabled: '{{ .Values.vmsingle.enabled }}' url: https://raw.githubusercontent.com/VictoriaMetrics/VictoriaMetrics/master/deployment/docker/rules/alerts-single-node.yml	`(object)` Create default rules for monitoring the cluster
`defaultRules.alerting: spec: annotations: {} labels: {}`	`(object)` Common properties for VMRules alerts
`defaultRules.alerting.spec.annotations: {}`	`(object)` Additional annotations for VMRule alerts
`defaultRules.alerting.spec.labels: {}`	`(object)` Additional labels for VMRule alerts
`defaultRules.annotations: {}`	`(object)` Annotations for default rules
`defaultRules.extraGroupByLabels: []`	`(list)` Labels, which are used for grouping results of the queries. Note that these labels are joined with `.Values.global.clusterLabel`
`defaultRules.group: spec: {}`	`(object)` Common properties for VMRule groups
defaultRules.groups: alertmanager.rules: rules: {} etcd: alerting: {} enabled: true recording: {} rules: {} general.rules: rules: {} k8s.rules.container_cpu_limits: rules: {} k8s.rules.container_cpu_requests: rules: {} k8s.rules.container_cpu_usage_seconds_total: rules: {} k8s.rules.container_memory_cache: rules: {} k8s.rules.container_memory_limits: rules: {} k8s.rules.container_memory_requests: rules: {} k8s.rules.container_memory_rss: rules: {} k8s.rules.container_memory_swap: rules: {} k8s.rules.container_memory_working_set_bytes: rules: {} k8s.rules.pod_owner: rules: {} kube-apiserver-burnrate.rules: rules: {} kube-apiserver-histogram.rules: rules: {} kube-apiserver-slos: rules: {} kube-prometheus-general.rules: rules: {} kube-prometheus-node-recording.rules: rules: {} kube-scheduler.rules: rules: {} kube-state-metrics: rules: {} kubelet.rules: rules: {} kubernetes-apps: jobNamespaces: kube-state-metrics: .* rules: {} kubernetes-resources: rules: {} kubernetes-storage: jobNamespaces: kubelet: .* rules: {} kubernetes-system: rules: {} kubernetes-system-apiserver: rules: {} kubernetes-system-controller-manager: rules: {} kubernetes-system-kubelet: rules: {} kubernetes-system-scheduler: rules: {} node-exporter: rules: {} node-exporter.rules: rules: {} node-network: rules: {} node.rules: rules: {} vm-health: rules: {} vmagent: rules: {} vmalert: rules: {} vmcluster: rules: {} vmoperator: rules: {} vmsingle: rules: {}	`(object)` Rule group properties. Keys must match upstream group names exactly.
`defaultRules.groups.etcd.alerting: {}`	`(object)` Common properties for all alerting rules in a group
`defaultRules.groups.etcd.enabled: true`	`(bool)` Enable/disable rule group
`defaultRules.groups.etcd.recording: {}`	`(object)` Common properties for all recording rules in a group
`defaultRules.groups.etcd.rules: {}`	`(object)` Common properties for all rules in a group
`defaultRules.jobNamespaces: {}`	`(object)` Inject namespace=~”” filter into metric selectors for metrics scraped by the given job. Merged with per-group jobNamespaces (per-group wins on conflict). Only use for jobs whose metrics are always namespace-scoped.
`defaultRules.labels: {}`	`(object)` Labels for default rules
`defaultRules.recording: spec: annotations: {} labels: {}`	`(object)` Common properties for VMRules recording rules
`defaultRules.recording.spec.annotations: {}`	`(object)` Additional annotations for VMRule recording rules
`defaultRules.recording.spec.labels: {}`	`(object)` Additional labels for VMRule recording rules
`defaultRules.rule: spec: annotations: {} labels: {}`	`(object)` Common properties for all VMRules
`defaultRules.rule.spec.annotations: {}`	`(object)` Additional annotations for all VMRules
`defaultRules.rule.spec.labels: {}`	`(object)` Additional labels for all VMRules
`defaultRules.rules: {}`	`(object)` Per rule properties
`defaultRules.runbookUrl: https://runbooks.prometheus-operator.dev/runbooks`	`(string)` Runbook url prefix for default rules
defaultRules.sources: alertmanager: enabled: '{{ .Values.alertmanager.enabled }}' url: https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/alertmanager-prometheusRule.yaml etcd: enabled: '{{ .Values.kubeEtcd.enabled }}' url: https://raw.githubusercontent.com/monitoring-mixins/website/master/assets/etcd/alerts.yaml kube-state-metrics: enabled: '{{ index .Values "kube-state-metrics" "enabled" }}' url: https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/kubeStateMetrics-prometheusRule.yaml kubernetes: url: https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/kubePrometheus-prometheusRule.yaml kubernetes-control-plane: url: https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/kubernetesControlPlane-prometheusRule.yaml node-exporter: enabled: '{{ index .Values "prometheus-node-exporter" "enabled" }}' url: https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/nodeExporter-prometheusRule.yaml vmagent: enabled: '{{ .Values.vmagent.enabled }}' url: https://raw.githubusercontent.com/VictoriaMetrics/VictoriaMetrics/master/deployment/docker/rules/alerts-vmagent.yml vmalert: enabled: '{{ .Values.vmalert.enabled }}' url: https://raw.githubusercontent.com/VictoriaMetrics/VictoriaMetrics/master/deployment/docker/rules/alerts-vmalert.yml vmcluster: enabled: '{{ .Values.vmcluster.enabled }}' url: https://raw.githubusercontent.com/VictoriaMetrics/VictoriaMetrics/master/deployment/docker/rules/alerts-cluster.yml vmhealth: enabled: '{{ or .Values.vmsingle.enabled .Values.vmcluster.enabled }}' url: https://raw.githubusercontent.com/VictoriaMetrics/VictoriaMetrics/master/deployment/docker/rules/alerts-health.yml vmoperator: enabled: '{{ index .Values "victoria-metrics-operator" "enabled" }}' url: https://raw.githubusercontent.com/VictoriaMetrics/operator/master/config/alerting/vmoperator-rules.yaml vmsingle: enabled: '{{ .Values.vmsingle.enabled }}' url: https://raw.githubusercontent.com/VictoriaMetrics/VictoriaMetrics/master/deployment/docker/rules/alerts-single-node.yml	`(object)` Rule sources. Set enabled: false on any entry to disable it. Add new keys to extend.
`defaultScrapeService.namespace: kube-system`	`(string)`
`external.grafana.datasource: VictoriaMetrics`	`(string)` External Grafana datasource name
`external.grafana.host: ""`	`(string)` External Grafana host
`external.vm: read: url: "" write: url: ""`	`(object)` External VM read and write URLs
`extraObjects: []`	`(list)` Add extra objects dynamically to this chart
`extraRules: {}`	`(object)` Provide custom recording or alerting rules to be deployed into the cluster.
`fullnameOverride: ""`	`(string)` Resource full name override
`global.cluster.dnsDomain: cluster.local.`	`(string)` K8s cluster domain suffix, uses for building storage pods’ FQDN. Details are here
`global.clusterLabel: cluster`	`(string)` Cluster label to use for dashboards and rules
`global.license: key: "" keyRef: {}`	`(object)` Global license configuration
grafana: enabled: true forceDeployDatasource: false ingress: annotations: {} enabled: false extraPaths: [] hosts: - grafana.domain.com labels: {} path: / pathType: Prefix tls: [] sidecar: dashboards: defaultFolderName: default enabled: true folder: /var/lib/grafana/dashboards label: grafana_dashboard labelValue: "1" multicluster: false provider: name: default orgid: 1 datasources: enabled: true label: grafana_datasource labelValue: "1" vmScrape: enabled: true spec: endpoints: - port: '{{ .Values.grafana.service.portName }}' selector: matchLabels: app.kubernetes.io/name: '{{ include "grafana.name" .Subcharts.grafana }}'	`(object)` Grafana dependency chart configuration. For possible values refer here
`grafana.forceDeployDatasource: false`	`(bool)` Create datasource configmap even if grafana deployment has been disabled
`grafana.ingress.extraPaths: []`	`(list)` Extra paths to prepend to every host configuration. This is useful when working with annotation based services.
`grafana.vmScrape: enabled: true spec: endpoints: - port: '{{ .Values.grafana.service.portName }}' selector: matchLabels: app.kubernetes.io/name: '{{ include "grafana.name" .Subcharts.grafana }}'`	`(object)` Grafana VM scrape config
`grafana.vmScrape.spec: endpoints: - port: '{{ .Values.grafana.service.portName }}' selector: matchLabels: app.kubernetes.io/name: '{{ include "grafana.name" .Subcharts.grafana }}'`	`(object)` Scrape configuration for Grafana
`kube-state-metrics: enabled: true vmScrape: enabled: true spec: endpoints: - honorLabels: true metricRelabelConfigs: - action: labeldrop regex: (uid\|container_id\|image_id) port: http jobLabel: app.kubernetes.io/name selector: matchLabels: app.kubernetes.io/instance: '{{ include "vm.release" . }}' app.kubernetes.io/name: '{{ include "kube-state-metrics.name" (index .Subcharts "kube-state-metrics") }}'`	`(object)` kube-state-metrics dependency chart configuration. For possible values check here
`kube-state-metrics.vmScrape: enabled: true spec: endpoints: - honorLabels: true metricRelabelConfigs: - action: labeldrop regex: (uid\|container_id\|image_id) port: http jobLabel: app.kubernetes.io/name selector: matchLabels: app.kubernetes.io/instance: '{{ include "vm.release" . }}' app.kubernetes.io/name: '{{ include "kube-state-metrics.name" (index .Subcharts "kube-state-metrics") }}'`	`(object)` Scrape configuration for Kube State Metrics
`kubeApiServer.enabled: true`	`(bool)` Enable Kube Api Server metrics scraping
`kubeApiServer.vmScrape: spec: endpoints: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token port: https scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt serverName: kubernetes jobLabel: component namespaceSelector: matchNames: - default selector: matchLabels: component: apiserver provider: kubernetes`	`(object)` Spec for VMServiceScrape CRD is here
`kubeControllerManager.enabled: true`	`(bool)` Enable kube controller manager metrics scraping
`kubeControllerManager.endpoints: []`	`(list)` If your kube controller manager is not deployed as a pod, specify IPs it can be found on
`kubeControllerManager.service.enabled: true`	`(bool)` Create service for kube controller manager metrics scraping
`kubeControllerManager.service.labels: {}`	`(object)` Kube controller manager service labels
`kubeControllerManager.service.port: 10257`	`(int)` Kube controller manager service port
`kubeControllerManager.service.selector: component: kube-controller-manager`	`(object)` Kube controller manager service pod selector
`kubeControllerManager.service.targetPort: 10257`	`(int)` Kube controller manager service target port
`kubeControllerManager.vmScrape: spec: endpoints: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token port: http-metrics scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt serverName: kubernetes jobLabel: app.kubernetes.io/component namespaceSelector: matchNames: []`	`(object)` Spec for VMServiceScrape CRD is here
`kubeDns.enabled: false`	`(bool)` Enabled KubeDNS metrics scraping
`kubeDns.service.enabled: false`	`(bool)` Create Service for KubeDNS metrics
`kubeDns.service.labels: {}`	`(object)` KubeDNS service labels
`kubeDns.service.ports: dnsmasq: port: 10054 targetPort: 10054 skydns: port: 10055 targetPort: 10055`	`(object)` KubeDNS service ports
`kubeDns.service.selector: k8s-app: kube-dns`	`(object)` KubeDNS service pods selector
`kubeDns.vmScrape: spec: endpoints: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token port: http-metrics-dnsmasq - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token port: http-metrics-skydns jobLabel: app.kubernetes.io/component namespaceSelector: matchNames: []`	`(object)` Spec for VMServiceScrape CRD is here
`kubeEtcd.enabled: true`	`(bool)` Enabled KubeETCD metrics scraping
`kubeEtcd.endpoints: []`	`(list)` If your etcd is not deployed as a pod, specify IPs it can be found on
`kubeEtcd.service.enabled: true`	`(bool)` Enable service for ETCD metrics scraping
`kubeEtcd.service.labels: {}`	`(object)` ETCD service labels
`kubeEtcd.service.port: 2379`	`(int)` ETCD service port
`kubeEtcd.service.selector: component: etcd`	`(object)` ETCD service pods selector
`kubeEtcd.service.targetPort: 2379`	`(int)` ETCD service target port
`kubeEtcd.vmScrape: spec: endpoints: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token port: http-metrics scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt jobLabel: app.kubernetes.io/component namespaceSelector: matchNames: []`	`(object)` Spec for VMServiceScrape CRD is here
`kubeProxy.enabled: false`	`(bool)` Enable kube proxy metrics scraping
`kubeProxy.endpoints: []`	`(list)` If your kube proxy is not deployed as a pod, specify IPs it can be found on
`kubeProxy.service.enabled: true`	`(bool)` Enable service for kube proxy metrics scraping
`kubeProxy.service.labels: {}`	`(object)` Kube proxy service labels
`kubeProxy.service.port: 10249`	`(int)` Kube proxy service port
`kubeProxy.service.selector: k8s-app: kube-proxy`	`(object)` Kube proxy service pod selector
`kubeProxy.service.targetPort: 10249`	`(int)` Kube proxy service target port
`kubeProxy.vmScrape: spec: endpoints: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token port: http-metrics scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt jobLabel: app.kubernetes.io/component namespaceSelector: matchNames: []`	`(object)` Spec for VMServiceScrape CRD is here
`kubeScheduler.enabled: true`	`(bool)` Enable KubeScheduler metrics scraping
`kubeScheduler.endpoints: []`	`(list)` If your kube scheduler is not deployed as a pod, specify IPs it can be found on
`kubeScheduler.service.enabled: true`	`(bool)` Enable service for KubeScheduler metrics scrape
`kubeScheduler.service.labels: {}`	`(object)` KubeScheduler service labels
`kubeScheduler.service.port: 10259`	`(int)` KubeScheduler service port
`kubeScheduler.service.selector: component: kube-scheduler`	`(object)` KubeScheduler service pod selector
`kubeScheduler.service.targetPort: 10259`	`(int)` KubeScheduler service target port
`kubeScheduler.vmScrape: spec: endpoints: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token port: http-metrics scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt jobLabel: app.kubernetes.io/component namespaceSelector: matchNames: []`	`(object)` Spec for VMServiceScrape CRD is here
kubelet: enabled: true vmScrape: kind: VMNodeScrape spec: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: false interval: 30s metricRelabelConfigs: - action: labeldrop regex: (uid) - action: labeldrop regex: (id\|name) - action: drop regex: (rest_client_request_duration_seconds_bucket\|rest_client_request_duration_seconds_sum\|rest_client_request_duration_seconds_count) source_labels: - __name__ relabelConfigs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - sourceLabels: - __metrics_path__ targetLabel: metrics_path - replacement: kubelet targetLabel: job scheme: https scrapeTimeout: 5s tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true vmScrapes: cadvisor: enabled: true spec: path: /metrics/cadvisor kubelet: spec: {} probes: enabled: true spec: path: /metrics/probes resources: enabled: true spec: path: /metrics/resource	`(object)` Component scraping the kubelets
kubelet.vmScrape: kind: VMNodeScrape spec: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true honorTimestamps: false interval: 30s metricRelabelConfigs: - action: labeldrop regex: (uid) - action: labeldrop regex: (id\|name) - action: drop regex: (rest_client_request_duration_seconds_bucket\|rest_client_request_duration_seconds_sum\|rest_client_request_duration_seconds_count) source_labels: - __name__ relabelConfigs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - sourceLabels: - __metrics_path__ targetLabel: metrics_path - replacement: kubelet targetLabel: job scheme: https scrapeTimeout: 5s tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true	`(object)` Spec for VMNodeScrape CRD is here
`kubelet.vmScrapes.cadvisor: enabled: true spec: path: /metrics/cadvisor`	`(object)` Enable scraping /metrics/cadvisor from kubelet’s service
`kubelet.vmScrapes.probes: enabled: true spec: path: /metrics/probes`	`(object)` Enable scraping /metrics/probes from kubelet’s service
`kubelet.vmScrapes.resources: enabled: true spec: path: /metrics/resource`	`(object)` Enabled scraping /metrics/resource from kubelet’s service
`nameOverride: ""`	`(string)` Override chart name
prometheus-node-exporter: enabled: true extraArgs: - --collector.filesystem.mount-points-exclude=^/(dev\|proc\|sys\|var/lib/docker/.+\|var/lib/kubelet/.+)($\|/) - --collector.filesystem.fs-types-exclude=^(autofs\|binfmt_misc\|bpf\|cgroup2?\|configfs\|debugfs\|devpts\|devtmpfs\|fusectl\|hugetlbfs\|iso9660\|mqueue\|nsfs\|overlay\|proc\|procfs\|pstore\|rpc_pipefs\|securityfs\|selinuxfs\|squashfs\|erofs\|sysfs\|tracefs)$ service: labels: jobLabel: node-exporter vmScrape: enabled: true spec: endpoints: - metricRelabelConfigs: - action: drop regex: /var/lib/kubelet/pods.+ source_labels: - mountpoint port: metrics jobLabel: jobLabel selector: matchLabels: app.kubernetes.io/name: '{{ include "prometheus-node-exporter.name" (index .Subcharts "prometheus-node-exporter") }}'	`(object)` prometheus-node-exporter dependency chart configuration. For possible values check here
`prometheus-node-exporter.vmScrape: enabled: true spec: endpoints: - metricRelabelConfigs: - action: drop regex: /var/lib/kubelet/pods.+ source_labels: - mountpoint port: metrics jobLabel: jobLabel selector: matchLabels: app.kubernetes.io/name: '{{ include "prometheus-node-exporter.name" (index .Subcharts "prometheus-node-exporter") }}'`	`(object)` Node Exporter VM scrape config
`prometheus-node-exporter.vmScrape.spec: endpoints: - metricRelabelConfigs: - action: drop regex: /var/lib/kubelet/pods.+ source_labels: - mountpoint port: metrics jobLabel: jobLabel selector: matchLabels: app.kubernetes.io/name: '{{ include "prometheus-node-exporter.name" (index .Subcharts "prometheus-node-exporter") }}'`	`(object)` Scrape configuration for Node Exporter
`syncJob.affinity: {}`	`(object)` Affinity for sync-job Pod scheduling
`syncJob.backoffLimit: 3`	`(int)` Maximum number of retries before the Job is considered failed
`syncJob.containerSecurityContext: {}`	`(object)` Security context for the sync-job container
`syncJob.enabled: true`	`(bool)` Fetch dashboards and rules from upstream sources at deploy time. Creates ConfigMaps (for Grafana sidecar) or GrafanaDashboard CRDs and VMRules directly in the cluster. A new Job runs automatically whenever this config changes.
`syncJob.image.pullPolicy: IfNotPresent`	`(string)`
`syncJob.image.repository: ghcr.io/victoriametrics/sync-job`	`(string)`
`syncJob.image.tag: latest`	`(string)`
`syncJob.nodeSelector: {}`	`(object)`
`syncJob.podAnnotations: {}`	`(object)` Annotations to add to the sync-job Pod
`syncJob.podSecurityContext: {}`	`(object)` Security context for the sync-job Pod
`syncJob.prune: true`	`(bool)` Delete managed resources that are no longer in the current config
`syncJob.resourcePrefix: ""`	`(string)` Prefix for managed resource names (VMRules, ConfigMaps, GrafanaDashboards). Defaults to the Helm release name when empty.
`syncJob.resources: {}`	`(object)` Override multicluster mode for dashboard and rule patching. Falls back to grafana.sidecar.dashboards.multicluster when not set. multicluster: false
`syncJob.tolerations: []`	`(list)`
`syncJob.ttlSecondsAfterFinished: 600`	`(int)` Seconds to keep a finished job before automatic deletion
`tenant: "0"`	`(string)` Tenant to use for Grafana datasources and remote write
`victoria-metrics-operator: admissionWebhooks: policy: Ignore crds: cleanup: enabled: true image: pullPolicy: IfNotPresent repository: registry.k8s.io/kubectl plain: true enabled: true operator: disable_prometheus_converter: false serviceMonitor: enabled: true`	`(object)` VictoriaMetrics Operator dependency chart configuration. More values can be found here. Also checkout here possible ENV variables to configure operator behaviour
`victoria-metrics-operator.admissionWebhooks.policy: Ignore`	`(string)` Default the validating webhook `failurePolicy` to `Ignore` so the stack can be installed/upgraded in a single pass without races against the operator’s webhook server. Override to `Fail` for strict validation once the operator is in steady state.
`victoria-metrics-operator.operator.disable_prometheus_converter: false`	`(bool)` By default, operator converts prometheus-operator objects.
`vmagent.additionalRemoteWrites: []`	`(list)` Remote write configuration of VMAgent, allowed parameters defined in a spec
`vmagent.annotations: {}`	`(object)` VMAgent annotations
`vmagent.enabled: true`	`(bool)` Create VMAgent CR
`vmagent.ingress: annotations: {} enabled: false extraPaths: [] hosts: - vmagent.domain.com labels: {} path: "" pathType: Prefix tls: []`	`(object)` VMAgent ingress configuration
`vmagent.labels: {}`	`(object)` VMAgent labels
`vmagent.rbac.annotations: {}`	`(object)` Role/RoleBinding annotations
`vmagent.rbac.extraLabels: {}`	`(object)` Role/RoleBinding labels
`vmagent.rbac.namespaced: true`	`(bool)` Defines if ClusterRole or Role with respective bindings should be created
`vmagent.rbac.rules: []`	`(list)` additional rules for a role
`vmagent.route: annotations: {} enabled: false extraRules: [] filters: [] hostnames: [] labels: {} matches: - path: type: PathPrefix value: '{{ dig "spec" "extraArgs" "http.pathPrefix" "/" .Values.vmagent }}' parentRefs: [] port: '{{ .Values.vmagent.spec.port }}'`	`(object)` VMAgent route configuration
`vmagent.route.annotations: {}`	`(object)` HTTPRoute annotations
`vmagent.route.enabled: false`	`(bool)` Enable deployment of HTTPRoute for vmagent component
`vmagent.route.extraRules: []`	`(list)` Extra rules to prepend to route. This is useful when working with annotation based services.
`vmagent.route.filters: []`	`(list)` Filters for a default rule in HTTPRoute
`vmagent.route.hostnames: []`	`(list)` Array of hostnames
`vmagent.route.labels: {}`	`(object)` HTTPRoute extra labels
`vmagent.route.matches: - path: type: PathPrefix value: '{{ dig "spec" "extraArgs" "http.pathPrefix" "/" .Values.vmagent }}'`	`(list)` Matches for a default rule in HTTPRoute
`vmagent.route.parentRefs: []`	`(list)` HTTPGateway objects refs
`vmagent.route.port: '{{ .Values.vmagent.spec.port }}'`	`(string)` Route port
`vmagent.spec: externalLabels: {} extraArgs: promscrape.streamParse: "true" port: "8429" scrapeInterval: 20s selectAllByDefault: true`	`(object)` Full spec for VMAgent CRD. Allowed values described here
`vmalert.additionalNotifierConfigs: {}`	`(object)` Allows to configure static notifiers, discover notifiers via Consul and DNS, see specification here. This configuration will be created as separate secret and mounted to VMAlert pod.
`vmalert.annotations: {}`	`(object)` VMAlert annotations
`vmalert.enabled: true`	`(bool)` Create VMAlert CR
`vmalert.ingress: annotations: {} enabled: false extraPaths: [] hosts: - vmalert.domain.com labels: {} path: "" pathType: Prefix tls: []`	`(object)` VMAlert ingress config
`vmalert.ingress.extraPaths: []`	`(list)` Extra paths to prepend to every host configuration. This is useful when working with annotation based services.
`vmalert.labels: {}`	`(object)` VMAlert labels
`vmalert.remoteWriteVMAgent: false`	`(bool)` Controls whether VMAlert should use VMAgent or VMInsert as a target for remotewrite
`vmalert.route: annotations: {} enabled: false extraRules: [] filters: [] hostnames: [] labels: {} matches: - path: type: PathPrefix value: '{{ dig "spec" "extraArgs" "http.pathPrefix" "/" .Values.vmalert }}' parentRefs: [] port: '{{ .Values.vmalert.spec.port }}'`	`(object)` VMAlert route configuration
`vmalert.route.annotations: {}`	`(object)` HTTPRoute annotations
`vmalert.route.enabled: false`	`(bool)` Enable deployment of HTTPRoute for vmalert component
`vmalert.route.extraRules: []`	`(list)` Extra rules to prepend to route. This is useful when working with annotation based services.
`vmalert.route.filters: []`	`(list)` Filters for a default rule in HTTPRoute
`vmalert.route.hostnames: []`	`(list)` Array of hostnames
`vmalert.route.labels: {}`	`(object)` HTTPRoute extra labels
`vmalert.route.matches: - path: type: PathPrefix value: '{{ dig "spec" "extraArgs" "http.pathPrefix" "/" .Values.vmalert }}'`	`(list)` Matches for a default rule in HTTPRoute
`vmalert.route.parentRefs: []`	`(list)` HTTPGateway objects refs
`vmalert.route.port: '{{ .Values.vmalert.spec.port }}'`	`(string)` Route port
`vmalert.spec: evaluationInterval: 20s externalLabels: {} extraArgs: http.pathPrefix: / port: "8080" selectAllByDefault: true`	`(object)` Full spec for VMAlert CRD. Allowed values described here
`vmalert.templateFiles: {}`	`(object)` Extra VMAlert annotation templates
`vmauth.annotations: {}`	`(object)` VMAuth annotations
`vmauth.enabled: false`	`(bool)` Enable VMAuth CR
`vmauth.labels: {}`	`(object)` VMAuth labels
`vmauth.spec: port: "8427" unauthorizedUserAccessSpec: disabled: false discover_backend_ips: true url_map: - src_paths: - '{{ .vm.read.path }}/.' url_prefix: - '{{ urlJoin (omit .vm.read "path") }}/' - src_paths: - '{{ .vm.write.path }}/.' url_prefix: - '{{ urlJoin (omit .vm.write "path") }}/'`	`(object)` Full spec for VMAuth CRD. Allowed values described here It’s possible to use given below predefined variables in spec: * `{{ .vm.read }}` - parsed vmselect, vmsingle or external.vm.read URL * `{{ .vm.write }}` - parsed vminsert, vmsingle or external.vm.write URL
`vmauth.spec.unauthorizedUserAccessSpec.disabled: false`	`(bool)` Flag, that allows to disable default VMAuth unauthorized user access config
`vmcluster.annotations: {}`	`(object)` VMCluster annotations
`vmcluster.enabled: false`	`(bool)` Create VMCluster CR
`vmcluster.ingress.insert.annotations: {}`	`(object)` Ingress annotations
`vmcluster.ingress.insert.enabled: false`	`(bool)` Enable deployment of ingress for server component
`vmcluster.ingress.insert.extraPaths: []`	`(list)` Extra paths to prepend to every host configuration. This is useful when working with annotation based services.
`vmcluster.ingress.insert.hosts: []`	`(list)` Array of host objects
`vmcluster.ingress.insert.ingressClassName: ""`	`(string)` Ingress controller class name
`vmcluster.ingress.insert.labels: {}`	`(object)` Ingress extra labels
`vmcluster.ingress.insert.path: '{{ dig "extraArgs" "http.pathPrefix" "/" .Values.vmcluster.spec.vminsert }}'`	`(string)` Ingress default path
`vmcluster.ingress.insert.pathType: Prefix`	`(string)` Ingress path type
`vmcluster.ingress.insert.tls: []`	`(list)` Array of TLS objects
`vmcluster.ingress.select.annotations: {}`	`(object)` Ingress annotations
`vmcluster.ingress.select.enabled: false`	`(bool)` Enable deployment of ingress for server component
`vmcluster.ingress.select.extraPaths: []`	`(list)` Extra paths to prepend to every host configuration. This is useful when working with annotation based services.
`vmcluster.ingress.select.hosts: []`	`(list)` Array of host objects
`vmcluster.ingress.select.ingressClassName: ""`	`(string)` Ingress controller class name
`vmcluster.ingress.select.labels: {}`	`(object)` Ingress extra labels
`vmcluster.ingress.select.path: '{{ dig "extraArgs" "http.pathPrefix" "/" .Values.vmcluster.spec.vmselect }}'`	`(string)` Ingress default path
`vmcluster.ingress.select.pathType: Prefix`	`(string)` Ingress path type
`vmcluster.ingress.select.tls: []`	`(list)` Array of TLS objects
`vmcluster.ingress.storage.annotations: {}`	`(object)` Ingress annotations
`vmcluster.ingress.storage.enabled: false`	`(bool)` Enable deployment of ingress for server component
`vmcluster.ingress.storage.extraPaths: []`	`(list)` Extra paths to prepend to every host configuration. This is useful when working with annotation based services.
`vmcluster.ingress.storage.hosts: []`	`(list)` Array of host objects
`vmcluster.ingress.storage.ingressClassName: ""`	`(string)` Ingress controller class name
`vmcluster.ingress.storage.labels: {}`	`(object)` Ingress extra labels
`vmcluster.ingress.storage.path: ""`	`(string)` Ingress default path
`vmcluster.ingress.storage.pathType: Prefix`	`(string)` Ingress path type
`vmcluster.ingress.storage.tls: []`	`(list)` Array of TLS objects
`vmcluster.labels: {}`	`(object)` VMCluster labels
`vmcluster.route.insert.annotations: {}`	`(object)` HTTPRoute annotations
`vmcluster.route.insert.enabled: false`	`(bool)` Enable deployment of HTTPRoute for insert component
`vmcluster.route.insert.extraRules: []`	`(list)` Extra rules to prepend to route. This is useful when working with annotation based services.
`vmcluster.route.insert.filters: []`	`(list)` Filters for a default rule in HTTPRoute
`vmcluster.route.insert.hostnames: []`	`(list)` Array of hostnames
`vmcluster.route.insert.labels: {}`	`(object)` HTTPRoute extra labels
`vmcluster.route.insert.matches: - path: type: PathPrefix value: '{{ dig "extraArgs" "http.pathPrefix" "/" .Values.vmcluster.spec.vminsert }}'`	`(list)` Matches for a default rule in HTTPRoute
`vmcluster.route.insert.parentRefs: []`	`(list)` HTTPGateway objects refs
`vmcluster.route.insert.port: '{{ .Values.vmcluster.spec.vminsert.port }}'`	`(string)` Route port
`vmcluster.route.select.annotations: {}`	`(object)` HTTPRoute annotations
`vmcluster.route.select.enabled: false`	`(bool)` Enable deployment of HTTPRoute for select component
`vmcluster.route.select.extraRules: []`	`(list)` Extra rules to prepend to route. This is useful when working with annotation based services.
`vmcluster.route.select.filters: []`	`(list)` Filters for a default rule in HTTPRoute
`vmcluster.route.select.hostnames: []`	`(list)` Array of hostnames
`vmcluster.route.select.labels: {}`	`(object)` HTTPRoute extra labels
`vmcluster.route.select.matches: - path: type: PathPrefix value: '{{ dig "extraArgs" "http.pathPrefix" "/" .Values.vmcluster.spec.vmselect }}'`	`(list)` Matches for a default rule in HTTPRoute
`vmcluster.route.select.parentRefs: []`	`(list)` HTTPGateway objects refs
`vmcluster.route.select.port: '{{ .Values.vmcluster.spec.vmselect.port }}'`	`(string)` Route port
`vmcluster.route.storage.annotations: {}`	`(object)` HTTPRoute annotations
`vmcluster.route.storage.enabled: false`	`(bool)` Enable deployment of HTTPRoute for storage component
`vmcluster.route.storage.extraRules: []`	`(list)` Extra rules to prepend to route. This is useful when working with annotation based services.
`vmcluster.route.storage.filters: []`	`(list)` Filters for a default rule in HTTPRoute
`vmcluster.route.storage.hostnames: []`	`(list)` Array of hostnames
`vmcluster.route.storage.labels: {}`	`(object)` HTTPRoute extra labels
`vmcluster.route.storage.matches: - path: type: PathPrefix value: '{{ dig "extraArgs" "http.pathPrefix" "/" .Values.vmcluster.spec.vmstorage }}'`	`(list)` Matches for a default rule in HTTPRoute
`vmcluster.route.storage.parentRefs: []`	`(list)` HTTPGateway objects refs
`vmcluster.route.storage.port: '{{ .Values.vmcluster.spec.vmstorage.port }}'`	`(string)` Route port
`vmcluster.spec: replicationFactor: 2 retentionPeriod: "1" vminsert: enabled: true extraArgs: {} port: "8480" replicaCount: 2 resources: {} vmselect: cacheMountPath: /select-cache enabled: true extraArgs: {} port: "8481" replicaCount: 2 resources: {} storage: volumeClaimTemplate: spec: resources: requests: storage: 2Gi vmstorage: replicaCount: 2 resources: {} storage: volumeClaimTemplate: spec: resources: requests: storage: 10Gi storageDataPath: /vm-data`	`(object)` Full spec for VMCluster CRD. Allowed values described here
`vmcluster.spec.retentionPeriod: "1"`	`(string)` Data retention period. Possible units character: h(ours), d(ays), w(eeks), y(ears), if no unit character specified - month. The minimum retention period is 24h. See these docs
`vmcluster.spec.vminsert.enabled: true`	`(bool)` Set this value to false to disable VMInsert
`vmcluster.spec.vmselect.enabled: true`	`(bool)` Set this value to false to disable VMSelect
`vmdistributed.annotations: {}`	`(object)` VMDistributed annotations
`vmdistributed.enabled: false`	`(bool)` Create VMDistributed CR
`vmdistributed.labels: {}`	`(object)` VMDistributed labels
`vmdistributed.spec: vmauth: spec: port: "8427" zoneCommon: vmcluster: spec: vminsert: replicaCount: 2 vmselect: cacheMountPath: /select-cache replicaCount: 2 storage: volumeClaimTemplate: spec: resources: requests: storage: 2Gi vmstorage: replicaCount: 2 storage: volumeClaimTemplate: spec: resources: requests: storage: 10Gi storageDataPath: /vm-data zones: - name: us-east-1`	`(object)` Full spec for VMDistributed CRD. Allowed values described here
`vmsingle.annotations: {}`	`(object)` VMSingle annotations
`vmsingle.enabled: true`	`(bool)` Create VMSingle CR
`vmsingle.ingress.annotations: {}`	`(object)` Ingress annotations
`vmsingle.ingress.enabled: false`	`(bool)` Enable deployment of ingress for server component
`vmsingle.ingress.extraPaths: []`	`(list)` Extra paths to prepend to every host configuration. This is useful when working with annotation based services.
`vmsingle.ingress.hosts: []`	`(list)` Array of host objects
`vmsingle.ingress.ingressClassName: ""`	`(string)` Ingress controller class name
`vmsingle.ingress.labels: {}`	`(object)` Ingress extra labels
`vmsingle.ingress.path: ""`	`(string)` Ingress default path
`vmsingle.ingress.pathType: Prefix`	`(string)` Ingress path type
`vmsingle.ingress.tls: []`	`(list)` Array of TLS objects
`vmsingle.labels: {}`	`(object)` VMSingle labels
`vmsingle.route.annotations: {}`	`(object)` HTTPRoute annotations
`vmsingle.route.enabled: false`	`(bool)` Enable deployment of HTTPRoute for server component
`vmsingle.route.extraRules: []`	`(list)` Extra rules to prepend to route. This is useful when working with annotation based services.
`vmsingle.route.filters: []`	`(list)` Filters for a default rule in HTTPRoute
`vmsingle.route.hostnames: []`	`(list)` Array of hostnames
`vmsingle.route.labels: {}`	`(object)` HTTPRoute extra labels
`vmsingle.route.matches: - path: type: PathPrefix value: '{{ dig "spec" "extraArgs" "http.pathPrefix" "/" .Values.vmsingle }}'`	`(list)` Matches for a default rule in HTTPRoute
`vmsingle.route.parentRefs: []`	`(list)` HTTPGateway objects refs
`vmsingle.route.port: '{{ .Values.vmsingle.spec.port }}'`	`(string)` Route port
`vmsingle.spec: extraArgs: {} port: "8428" replicaCount: 1 retentionPeriod: "1" storage: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi`	`(object)` Full spec for VMSingle CRD. Allowed values describe here
`vmsingle.spec.retentionPeriod: "1"`	`(string)` Data retention period. Possible units character: h(ours), d(ays), w(eeks), y(ears), if no unit character specified - month. The minimum retention period is 24h. See these docs

VictoriaMetrics K8s Stack #

Overview #

Configuration #

Dependencies #

VictoriaMetrics components #

ArgoCD issues #

Operator self signed certificates #

Grafana password changes constantly #

metadata.annotations: Too long: must have at most 262144 bytes on dashboards #

Resources are not completely removed after chart uninstallation #

Rules and dashboards #

Adding external dashboards #

Prometheus scrape configs #

Using externally managed Grafana #

Using additional Grafana datasources #

Using alternative image registry #

Prerequisites #

How to install #

Setup chart repository (can be omitted for OCI repositories) #

Install victoria-metrics-k8s-stack chart #

Install operator separately #

Install locally (Minikube) #

How to uninstall #

Troubleshooting #

Upgrade guide #

Upgrade to 0.29.0 #

Upgrade to 0.13.0 #

Upgrade to 0.6.0 #

Upgrade to 0.4.0 #

Upgrade from 0.2.8 to 0.2.9 #

Upgrade from 0.2.5 to 0.2.6 #

Parameters #

VictoriaMetrics K8s Stack

`metadata.annotations: Too long: must have at most 262144 bytes` on dashboards #

Install `victoria-metrics-k8s-stack` chart #