VictoriaMetrics K8s Stack #

null

{}

receivers:
    - name: blackhole
route:
    receiver: blackhole
templates:
    - /etc/vm/configs/**/*.tmpl

alertmanager configuration

true

annotations: {}
enabled: false
extraPaths: []
hosts:
    - alertmanager.domain.com
labels: {}
path: '{{ .Values.alertmanager.spec.routePrefix | default "/" }}'
pathType: Prefix
tls: []

alertmanager ingress configuration

true

configSecret: ""
externalURL: ""
image:
    tag: v0.25.0
port: "9093"
routePrefix: /
selectAllByDefault: true

full spec for VMAlertmanager CRD. Allowed values described here

""

if this one defined, it will be used for alertmanager configuration and config parameter will be ignored

{}

extra alert templates

""

For correct working need set value ‘argocdReleaseOverride=$ARGOCD_APP_NAME’

true

true

9153

kube-dns

9153

spec:
    endpoints:
        - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
          port: http-metrics
    jobLabel: jobLabel
    namespaceSelector:
        matchNames:
            - kube-system

spec for VMServiceScrape crd https://docs.victoriametrics.com/operator/api.html#vmservicescrapespec

true

node-exporter-full: true
operator: false
vmalert: false

Enable dashboards despite it’s dependency is not installed

true

in ArgoCD using client-side apply this dashboard reaches annotations size limit and causes k8s issues without server side apply See this issue

true

Create default dashboards

alerting:
    spec:
        annotations: {}
        labels: {}
annotations: {}
create: true
group:
    spec:
        params: {}
groups:
    alertmanager:
        create: true
        rules: {}
    etcd:
        create: true
        rules: {}
    general:
        create: true
        rules: {}
    k8sContainerCpuUsageSecondsTotal:
        create: true
        rules: {}
    k8sContainerMemoryCache:
        create: true
        rules: {}
    k8sContainerMemoryRss:
        create: true
        rules: {}
    k8sContainerMemorySwap:
        create: true
        rules: {}
    k8sContainerMemoryWorkingSetBytes:
        create: true
        rules: {}
    k8sContainerResource:
        create: true
        rules: {}
    k8sPodOwner:
        create: true
        rules: {}
    kubeApiserver:
        create: true
        rules: {}
    kubeApiserverAvailability:
        create: true
        rules: {}
    kubeApiserverBurnrate:
        create: true
        rules: {}
    kubeApiserverHistogram:
        create: true
        rules: {}
    kubeApiserverSlos:
        create: true
        rules: {}
    kubePrometheusGeneral:
        create: true
        rules: {}
    kubePrometheusNodeRecording:
        create: true
        rules: {}
    kubeScheduler:
        create: true
        rules: {}
    kubeStateMetrics:
        create: true
        rules: {}
    kubelet:
        create: true
        rules: {}
    kubernetesApps:
        create: true
        rules: {}
        targetNamespace: .*
    kubernetesResources:
        create: true
        rules: {}
    kubernetesStorage:
        create: true
        rules: {}
        targetNamespace: .*
    kubernetesSystem:
        create: true
        rules: {}
    kubernetesSystemApiserver:
        create: true
        rules: {}
    kubernetesSystemControllerManager:
        create: true
        rules: {}
    kubernetesSystemKubelet:
        create: true
        rules: {}
    kubernetesSystemScheduler:
        create: true
        rules: {}
    node:
        create: true
        rules: {}
    nodeNetwork:
        create: true
        rules: {}
    vmHealth:
        create: true
        rules: {}
    vmagent:
        create: true
        rules: {}
    vmcluster:
        create: true
        rules: {}
    vmoperator:
        create: true
        rules: {}
    vmsingle:
        create: true
        rules: {}
labels: {}
recording:
    spec:
        annotations: {}
        labels: {}
rule:
    spec:
        annotations: {}
        labels: {}
rules: {}
runbookUrl: https://runbooks.prometheus-operator.dev/runbooks

Create default rules for monitoring the cluster

spec:
    annotations: {}
    labels: {}

Common properties for VMRules alerts

{}

Additional annotations for VMRule alerts

{}

Additional labels for VMRule alerts

{}

Annotations for default rules

spec:
    params: {}

Common properties for VMRule groups

{}

Optional HTTP URL parameters added to each rule request

{}

Common properties for all rules in a group

{}

Labels for default rules

spec:
    annotations: {}
    labels: {}

Common properties for VMRules recording rules

{}

Additional annotations for VMRule recording rules

{}

Additional labels for VMRule recording rules

spec:
    annotations: {}
    labels: {}

Common properties for all VMRules

{}

Additional annotations for all VMRules

{}

Additional labels for all VMRules

{}

Per rule properties

https://runbooks.prometheus-operator.dev/runbooks

Runbook url prefix for default rules

true

Create experimental dashboards

""

""

[]

Add extra objects dynamically to this chart

""

cluster

""

{}

[]

utc

prometheus

true

false

{}

false

[]

grafana.domain.com

{}

/

Prefix

[]

{}

{}

default

true

/var/lib/grafana/dashboards

false

default

1

false

- isDefault: true
  name: VictoriaMetrics
- isDefault: false
  name: VictoriaMetrics (DS)
  type: victoriametrics-datasource

list of default prometheus compatible datasource configurations. VM url will be added to each of them in templates and type will be set to defaultDatasourceType if not defined

true

true

enabled: true
spec:
    endpoints:
        - port: '{{ .Values.grafana.service.portName }}'
    selector:
        matchLabels:
            app.kubernetes.io/name: '{{ include "grafana.name" .Subcharts.grafana }}'

grafana VM scrape config

endpoints:
    - port: '{{ .Values.grafana.service.portName }}'
selector:
    matchLabels:
        app.kubernetes.io/name: '{{ include "grafana.name" .Subcharts.grafana }}'

Scrape configuration for Grafana

allowCrossNamespaceImport: false
enabled: false
instanceSelector:
    matchLabels:
        dashboards: grafana

Create dashboards as CRDs (reuqires grafana-operator to be installed)

true

enabled: true
spec:
    endpoints:
        - honorLabels: true
          metricRelabelConfigs:
            - action: labeldrop
              regex: (uid|container_id|image_id)
          port: http
    jobLabel: app.kubernetes.io/name
    selector:
        matchLabels:
            app.kubernetes.io/instance: '{{ include "vm.release" . }}'
            app.kubernetes.io/name: '{{ include "kube-state-metrics.name" (index .Subcharts "kube-state-metrics") }}'

Scrape configuration for Kube State Metrics

true

spec:
    endpoints:
        - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
          port: https
          scheme: https
          tlsConfig:
            caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            serverName: kubernetes
    jobLabel: component
    namespaceSelector:
        matchNames:
            - default
    selector:
        matchLabels:
            component: apiserver
            provider: kubernetes

spec for VMServiceScrape crd https://docs.victoriametrics.com/operator/api.html#vmservicescrapespec

true

[]

true

10257

kube-controller-manager

10257

spec:
    endpoints:
        - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
          port: http-metrics
          scheme: https
          tlsConfig:
            caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            serverName: kubernetes
    jobLabel: jobLabel
    namespaceSelector:
        matchNames:
            - kube-system

spec for VMServiceScrape crd https://docs.victoriametrics.com/operator/api.html#vmservicescrapespec

false

false

10054

10054

10055

10055

kube-dns

spec:
    endpoints:
        - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
          port: http-metrics-dnsmasq
        - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
          port: http-metrics-skydns
    jobLabel: jobLabel
    namespaceSelector:
        matchNames:
            - kube-system

spec for VMServiceScrape crd https://docs.victoriametrics.com/operator/api.html#vmservicescrapespec

true

[]

true

2379

etcd

2379

spec:
    endpoints:
        - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
          port: http-metrics
          scheme: https
          tlsConfig:
            caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    jobLabel: jobLabel
    namespaceSelector:
        matchNames:
            - kube-system

spec for VMServiceScrape crd https://docs.victoriametrics.com/operator/api.html#vmservicescrapespec

false

[]

true

10249

kube-proxy

10249

spec:
    endpoints:
        - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
          port: http-metrics
          scheme: https
          tlsConfig:
            caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    jobLabel: jobLabel
    namespaceSelector:
        matchNames:
            - kube-system

spec for VMServiceScrape crd https://docs.victoriametrics.com/operator/api.html#vmservicescrapespec

true

[]

true

10259

kube-scheduler

10259

spec:
    endpoints:
        - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
          port: http-metrics
          scheme: https
          tlsConfig:
            caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    jobLabel: jobLabel
    namespaceSelector:
        matchNames:
            - kube-system

spec for VMServiceScrape crd https://docs.victoriametrics.com/operator/api.html#vmservicescrapespec

true

kind: VMNodeScrape
spec:
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    honorLabels: true
    honorTimestamps: false
    interval: 30s
    metricRelabelConfigs:
        - action: labeldrop
          regex: (uid)
        - action: labeldrop
          regex: (id|name)
        - action: drop
          regex: (rest_client_request_duration_seconds_bucket|rest_client_request_duration_seconds_sum|rest_client_request_duration_seconds_count)
          source_labels:
            - __name__
    relabelConfigs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - sourceLabels:
            - __metrics_path__
          targetLabel: metrics_path
        - replacement: kubelet
          targetLabel: job
    scheme: https
    scrapeTimeout: 5s
    tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true

spec for VMNodeScrape crd https://docs.victoriametrics.com/operator/api.html#vmnodescrapespec

enabled: true
spec:
    path: /metrics/cadvisor

Enable scraping /metrics/cadvisor from kubelet’s service

{}

enabled: true
spec:
    path: /metrics/probes

Enable scraping /metrics/probes from kubelet’s service

""

true

--collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)

--collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$

node-exporter

enabled: true
spec:
    endpoints:
        - metricRelabelConfigs:
            - action: drop
              regex: /var/lib/kubelet/pods.+
              source_labels:
                - mountpoint
          port: metrics
    jobLabel: jobLabel
    selector:
        matchLabels:
            app.kubernetes.io/name: '{{ include "prometheus-node-exporter.name" (index .Subcharts "prometheus-node-exporter") }}'

node exporter VM scrape config

endpoints:
    - metricRelabelConfigs:
        - action: drop
          regex: /var/lib/kubelet/pods.+
          source_labels:
            - mountpoint
      port: metrics
jobLabel: jobLabel
selector:
    matchLabels:
        app.kubernetes.io/name: '{{ include "prometheus-node-exporter.name" (index .Subcharts "prometheus-node-exporter") }}'

Scrape configuration for Node Exporter

false

{}

Annotations to add to the service account

true

Specifies whether a service account should be created

""

If not set and create is true, a name is generated using the fullname template

"0"

crd:
    cleanup:
        enabled: true
        image:
            pullPolicy: IfNotPresent
            repository: bitnami/kubectl
    create: false
enabled: true
operator:
    disable_prometheus_converter: false
serviceMonitor:
    enabled: true

also checkout here possible ENV variables to configure operator behaviour https://docs.victoriametrics.com/operator/vars

enabled: true
image:
    pullPolicy: IfNotPresent
    repository: bitnami/kubectl

tells helm to clean up vm cr resources when uninstalling

false

we disable crd creation by operator chart as we create them in this chart

false

By default, operator converts prometheus-operator objects.

[]

remoteWrite configuration of VMAgent, allowed parameters defined in a spec

{}

true

annotations: {}
enabled: false
extraPaths: []
hosts:
    - vmagent.domain.com
labels: {}
path: ""
pathType: Prefix
tls: []

vmagent ingress configuration

[]

Extra paths to prepend to every host configuration. This is useful when working with annotation based services.

externalLabels: {}
extraArgs:
    promscrape.dropOriginalLabels: "true"
    promscrape.streamParse: "true"
image:
    tag: v1.103.0
port: "8429"
scrapeInterval: 20s
selectAllByDefault: true

full spec for VMAgent CRD. Allowed values described here

{}

{}

true

annotations: {}
enabled: false
extraPaths: []
hosts:
    - vmalert.domain.com
labels: {}
path: ""
pathType: Prefix
tls: []

vmalert ingress config

false

evaluationInterval: 15s
externalLabels: {}
extraArgs:
    http.pathPrefix: /
image:
    tag: v1.103.0
port: "8080"
selectAllByDefault: true

full spec for VMAlert CRD. Allowed values described here

{}

extra vmalert annotation templates

{}

false

discover_backend_ips: true
port: "8427"

full spec for VMAuth CRD. Allowed values described here

{}

false

{}

false

[]

vminsert.domain.com

{}

'{{ dig "extraArgs" "http.pathPrefix" "/" .Values.vmcluster.spec.vminsert }}'

Prefix

[]

{}

false

[]

vmselect.domain.com

{}

'{{ dig "extraArgs" "http.pathPrefix" "/" .Values.vmcluster.spec.vmselect }}'

Prefix

[]

{}

false

[]

vmstorage.domain.com

{}

""

Prefix

[]

replicationFactor: 2
retentionPeriod: "1"
vminsert:
    extraArgs: {}
    image:
        tag: v1.103.0-cluster
    port: "8480"
    replicaCount: 2
    resources: {}
vmselect:
    cacheMountPath: /select-cache
    extraArgs: {}
    image:
        tag: v1.103.0-cluster
    port: "8481"
    replicaCount: 2
    resources: {}
    storage:
        volumeClaimTemplate:
            spec:
                resources:
                    requests:
                        storage: 2Gi
vmstorage:
    image:
        tag: v1.103.0-cluster
    replicaCount: 2
    resources: {}
    storage:
        volumeClaimTemplate:
            spec:
                resources:
                    requests:
                        storage: 10Gi
    storageDataPath: /vm-data

full spec for VMCluster CRD. Allowed values described here

"1"

Data retention period. Possible units character: h(ours), d(ays), w(eeks), y(ears), if no unit character specified - month. The minimum retention period is 24h. See these docs

{}

true

{}

false

[]

vmsingle.domain.com

{}

""

Prefix

[]

extraArgs: {}
image:
    tag: v1.103.0
port: "8429"
replicaCount: 1
retentionPeriod: "1"
storage:
    accessModes:
        - ReadWriteOnce
    resources:
        requests:
            storage: 20Gi

full spec for VMSingle CRD. Allowed values describe here

"1"

Data retention period. Possible units character: h(ours), d(ays), w(eeks), y(ears), if no unit character specified - month. The minimum retention period is 24h. See these docs