The operator is set up using environment variables and command-line flags . Most environment variables control settings related to Resources , like CPU and memory defaults, image versions. Command-line flags configure the operator itself, like leader election, TLS, webhook validation, and rate limits.

Environment variables #

Run this command Available from v0.57.0 to see all environment variables your operator supports:

      OPERATOR_POD_NAME=$(kubectl get pod -l "app.kubernetes.io/name=victoria-metrics-operator"  -n vm -o jsonpath="{.items[0].metadata.name}");
kubectl exec -n vm "$OPERATOR_POD_NAME" -- /app --printDefaults 2>&1

# Output:
# KEY                   DEFAULT        REQUIRED    DESCRIPTION
# VM_METRICS_VERSION    v1.117.0       false       
# VM_LOGS_VERSION       v1.21.0        false 
# ... 
    

These are the latest operator environment variables:

Environment variables
VM_METRICS_VERSION: v1.135.0 #
VM_LOGS_VERSION: v1.45.0 #
VM_ANOMALY_VERSION: v1.28.5 #
VM_TRACES_VERSION: v0.7.0 #
VM_OPERATOR_VERSION: v0.66.1 #
VM_GATEWAY_API_ENABLED: false #
WATCH_NAMESPACE: - #
Defines a list of namespaces to be watched by operator. Operator don’t perform any cluster wide API calls if namespaces not empty. In case of empty list it performs only clusterwide api calls.
VM_CONTAINERREGISTRY: - #
container registry name prefix, e.g. docker.io
VM_CUSTOMCONFIGRELOADERIMAGE: - #
Deprecated: use VM_CONFIG_RELOADER_IMAGE instead
VM_PSPAUTOCREATEENABLED: false #
VM_ENABLETCP6: false #
VM_CONFIG_RELOADER_IMAGE: victoriametrics/operator:config-reloader-${VM_OPERATOR_VERSION} #
default image for all config-reloader containers
VM_CONFIG_RELOADER_LIMIT_MEMORY: unlimited #
defines global resource.limits.memory for all config-reloader containers
VM_CONFIG_RELOADER_LIMIT_CPU: unlimited #
defines global resource.limits.cpu for all config-reloader containers
VM_CONFIG_RELOADER_LIMIT_EPHEMERAL_STORAGE: unlimited #
defines global resource.limits.ephemeral-storage for all config-reloader containers
VM_CONFIG_RELOADER_REQUEST_MEMORY: 25Mi #
defines global resource.requests.memory for all config-reloader containers
VM_CONFIG_RELOADER_REQUEST_CPU: 10m #
defines global resource.requests.cpu for all config-reloader containers
VM_CONFIG_RELOADER_REQUEST_EPHEMERAL_STORAGE: unlimited #
defines global resource.requests.ephemeral-storage for all config-reloader containers
VM_VLOGSDEFAULT_IMAGE: victoriametrics/victoria-logs #
VM_VLOGSDEFAULT_VERSION: ${VM_LOGS_VERSION} #
VM_VLOGSDEFAULT_PORT: 9428 #
VM_VLOGSDEFAULT_USEDEFAULTRESOURCES: true #
VM_VLOGSDEFAULT_RESOURCE_LIMIT_MEM: 1500Mi #
VM_VLOGSDEFAULT_RESOURCE_LIMIT_CPU: 1200m #
VM_VLOGSDEFAULT_RESOURCE_LIMIT_EPHEMERAL_STORAGE: unlimited #
VM_VLOGSDEFAULT_RESOURCE_REQUEST_MEM: 500Mi #
VM_VLOGSDEFAULT_RESOURCE_REQUEST_CPU: 150m #
VM_VLOGSDEFAULT_RESOURCE_REQUEST_EPHEMERAL_STORAGE: unlimited #
VM_VLAGENTDEFAULT_IMAGE: victoriametrics/vlagent #
VM_VLAGENTDEFAULT_VERSION: ${VM_LOGS_VERSION} #
VM_VLAGENTDEFAULT_PORT: 9429 #
VM_VLAGENTDEFAULT_USEDEFAULTRESOURCES: true #
VM_VLAGENTDEFAULT_RESOURCE_LIMIT_MEM: 500Mi #
VM_VLAGENTDEFAULT_RESOURCE_LIMIT_CPU: 200m #
VM_VLAGENTDEFAULT_RESOURCE_LIMIT_EPHEMERAL_STORAGE: unlimited #
VM_VLAGENTDEFAULT_RESOURCE_REQUEST_MEM: 200Mi #
VM_VLAGENTDEFAULT_RESOURCE_REQUEST_CPU: 50m #
VM_VLAGENTDEFAULT_RESOURCE_REQUEST_EPHEMERAL_STORAGE: unlimited #
VM_VLSINGLEDEFAULT_IMAGE: victoriametrics/victoria-logs #
VM_VLSINGLEDEFAULT_VERSION: ${VM_LOGS_VERSION} #
VM_VLSINGLEDEFAULT_PORT: 9428 #
VM_VLSINGLEDEFAULT_USEDEFAULTRESOURCES: true #
VM_VLSINGLEDEFAULT_RESOURCE_LIMIT_MEM: 1500Mi #
VM_VLSINGLEDEFAULT_RESOURCE_LIMIT_CPU: 1200m #
VM_VLSINGLEDEFAULT_RESOURCE_LIMIT_EPHEMERAL_STORAGE: unlimited #
VM_VLSINGLEDEFAULT_RESOURCE_REQUEST_MEM: 500Mi #
VM_VLSINGLEDEFAULT_RESOURCE_REQUEST_CPU: 150m #
VM_VLSINGLEDEFAULT_RESOURCE_REQUEST_EPHEMERAL_STORAGE: unlimited #
VM_VTSINGLEDEFAULT_IMAGE: victoriametrics/victoria-traces #
VM_VTSINGLEDEFAULT_VERSION: ${VM_TRACES_VERSION} #
VM_VTSINGLEDEFAULT_PORT: 10428 #
VM_VTSINGLEDEFAULT_USEDEFAULTRESOURCES: true #
VM_VTSINGLEDEFAULT_RESOURCE_LIMIT_MEM: 1500Mi #
VM_VTSINGLEDEFAULT_RESOURCE_LIMIT_CPU: 1200m #
VM_VTSINGLEDEFAULT_RESOURCE_LIMIT_EPHEMERAL_STORAGE: unlimited #
VM_VTSINGLEDEFAULT_RESOURCE_REQUEST_MEM: 500Mi #
VM_VTSINGLEDEFAULT_RESOURCE_REQUEST_CPU: 150m #
VM_VTSINGLEDEFAULT_RESOURCE_REQUEST_EPHEMERAL_STORAGE: unlimited #
VM_VMALERTDEFAULT_IMAGE: victoriametrics/vmalert #
VM_VMALERTDEFAULT_VERSION: ${VM_METRICS_VERSION} #
VM_VMALERTDEFAULT_PORT: 8080 #
VM_VMALERTDEFAULT_USEDEFAULTRESOURCES: true #
VM_VMALERTDEFAULT_RESOURCE_LIMIT_MEM: 500Mi #
VM_VMALERTDEFAULT_RESOURCE_LIMIT_CPU: 200m #
VM_VMALERTDEFAULT_RESOURCE_LIMIT_EPHEMERAL_STORAGE: unlimited #
VM_VMALERTDEFAULT_RESOURCE_REQUEST_MEM: 200Mi #
VM_VMALERTDEFAULT_RESOURCE_REQUEST_CPU: 50m #
VM_VMALERTDEFAULT_RESOURCE_REQUEST_EPHEMERAL_STORAGE: unlimited #
VM_VMSERVICESCRAPEDEFAULT_ENFORCEENDPOINTSLICES: false #
Use endpointslices instead of endpoints as discovery role for vmservicescrape when generate scrape config for vmagent.
VM_VMAGENTDEFAULT_IMAGE: victoriametrics/vmagent #
VM_VMAGENTDEFAULT_VERSION: ${VM_METRICS_VERSION} #
VM_VMAGENTDEFAULT_PORT: 8429 #
VM_VMAGENTDEFAULT_USEDEFAULTRESOURCES: true #
VM_VMAGENTDEFAULT_RESOURCE_LIMIT_MEM: 500Mi #
VM_VMAGENTDEFAULT_RESOURCE_LIMIT_CPU: 200m #
VM_VMAGENTDEFAULT_RESOURCE_LIMIT_EPHEMERAL_STORAGE: unlimited #
VM_VMAGENTDEFAULT_RESOURCE_REQUEST_MEM: 200Mi #
VM_VMAGENTDEFAULT_RESOURCE_REQUEST_CPU: 50m #
VM_VMAGENTDEFAULT_RESOURCE_REQUEST_EPHEMERAL_STORAGE: unlimited #
VM_VMANOMALYDEFAULT_IMAGE: victoriametrics/vmanomaly #
VM_VMANOMALYDEFAULT_VERSION: ${VM_ANOMALY_VERSION} #
VM_VMANOMALYDEFAULT_PORT: 8490 #
VM_VMANOMALYDEFAULT_USEDEFAULTRESOURCES: true #
VM_VMANOMALYDEFAULT_RESOURCE_LIMIT_MEM: 500Mi #
VM_VMANOMALYDEFAULT_RESOURCE_LIMIT_CPU: 200m #
VM_VMANOMALYDEFAULT_RESOURCE_LIMIT_EPHEMERAL_STORAGE: unlimited #
VM_VMANOMALYDEFAULT_RESOURCE_REQUEST_MEM: 200Mi #
VM_VMANOMALYDEFAULT_RESOURCE_REQUEST_CPU: 50m #
VM_VMANOMALYDEFAULT_RESOURCE_REQUEST_EPHEMERAL_STORAGE: unlimited #
VM_VMSINGLEDEFAULT_IMAGE: victoriametrics/victoria-metrics #
VM_VMSINGLEDEFAULT_VERSION: ${VM_METRICS_VERSION} #
VM_VMSINGLEDEFAULT_PORT: 8429 #
VM_VMSINGLEDEFAULT_USEDEFAULTRESOURCES: true #
VM_VMSINGLEDEFAULT_RESOURCE_LIMIT_MEM: 1500Mi #
VM_VMSINGLEDEFAULT_RESOURCE_LIMIT_CPU: 1200m #
VM_VMSINGLEDEFAULT_RESOURCE_LIMIT_EPHEMERAL_STORAGE: unlimited #
VM_VMSINGLEDEFAULT_RESOURCE_REQUEST_MEM: 500Mi #
VM_VMSINGLEDEFAULT_RESOURCE_REQUEST_CPU: 150m #
VM_VMSINGLEDEFAULT_RESOURCE_REQUEST_EPHEMERAL_STORAGE: unlimited #
VM_VMCLUSTERDEFAULT_USEDEFAULTRESOURCES: true #
VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_IMAGE: victoriametrics/vmselect #
VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_VERSION: ${VM_METRICS_VERSION}-cluster #
VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_PORT: 8481 #
VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_RESOURCE_LIMIT_MEM: 1000Mi #
VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_RESOURCE_LIMIT_CPU: 500m #
VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_RESOURCE_LIMIT_EPHEMERAL_STORAGE: unlimited #
VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_RESOURCE_REQUEST_MEM: 500Mi #
VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_RESOURCE_REQUEST_CPU: 100m #
VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_RESOURCE_REQUEST_EPHEMERAL_STORAGE: unlimited #
VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_IMAGE: victoriametrics/vmstorage #
VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_VERSION: ${VM_METRICS_VERSION}-cluster #
VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_VMINSERTPORT: 8400 #
VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_VMSELECTPORT: 8401 #
VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_PORT: 8482 #
VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_RESOURCE_LIMIT_MEM: 1500Mi #
VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_RESOURCE_LIMIT_CPU: 1000m #
VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_RESOURCE_LIMIT_EPHEMERAL_STORAGE: unlimited #
VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_RESOURCE_REQUEST_MEM: 500Mi #
VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_RESOURCE_REQUEST_CPU: 250m #
VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_RESOURCE_REQUEST_EPHEMERAL_STORAGE: unlimited #
VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_IMAGE: victoriametrics/vminsert #
VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_VERSION: ${VM_METRICS_VERSION}-cluster #
VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_PORT: 8480 #
VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_RESOURCE_LIMIT_MEM: 500Mi #
VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_RESOURCE_LIMIT_CPU: 500m #
VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_RESOURCE_LIMIT_EPHEMERAL_STORAGE: unlimited #
VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_RESOURCE_REQUEST_MEM: 200Mi #
VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_RESOURCE_REQUEST_CPU: 150m #
VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_RESOURCE_REQUEST_EPHEMERAL_STORAGE: unlimited #
VM_VMALERTMANAGER_ALERTMANAGERDEFAULTBASEIMAGE: prom/alertmanager #
VM_VMALERTMANAGER_ALERTMANAGERVERSION: v0.31.0 #
VM_VMALERTMANAGER_USEDEFAULTRESOURCES: true #
VM_VMALERTMANAGER_RESOURCE_LIMIT_MEM: 256Mi #
VM_VMALERTMANAGER_RESOURCE_LIMIT_CPU: 100m #
VM_VMALERTMANAGER_RESOURCE_LIMIT_EPHEMERAL_STORAGE: unlimited #
VM_VMALERTMANAGER_RESOURCE_REQUEST_MEM: 56Mi #
VM_VMALERTMANAGER_RESOURCE_REQUEST_CPU: 30m #
VM_VMALERTMANAGER_RESOURCE_REQUEST_EPHEMERAL_STORAGE: unlimited #
VM_DISABLESELFSERVICESCRAPECREATION: false #
VM_VMBACKUP_IMAGE: victoriametrics/vmbackupmanager #
VM_VMBACKUP_VERSION: ${VM_METRICS_VERSION}-enterprise #
VM_VMBACKUP_PORT: 8300 #
VM_VMBACKUP_USEDEFAULTRESOURCES: true #
VM_VMBACKUP_RESOURCE_LIMIT_MEM: 500Mi #
VM_VMBACKUP_RESOURCE_LIMIT_CPU: 500m #
VM_VMBACKUP_RESOURCE_LIMIT_EPHEMERAL_STORAGE: unlimited #
VM_VMBACKUP_RESOURCE_REQUEST_MEM: 200Mi #
VM_VMBACKUP_RESOURCE_REQUEST_CPU: 150m #
VM_VMBACKUP_RESOURCE_REQUEST_EPHEMERAL_STORAGE: unlimited #
VM_VMAUTHDEFAULT_IMAGE: victoriametrics/vmauth #
VM_VMAUTHDEFAULT_VERSION: ${VM_METRICS_VERSION} #
VM_VMAUTHDEFAULT_PORT: 8427 #
VM_VMAUTHDEFAULT_USEDEFAULTRESOURCES: true #
VM_VMAUTHDEFAULT_RESOURCE_LIMIT_MEM: 300Mi #
VM_VMAUTHDEFAULT_RESOURCE_LIMIT_CPU: 200m #
VM_VMAUTHDEFAULT_RESOURCE_LIMIT_EPHEMERAL_STORAGE: unlimited #
VM_VMAUTHDEFAULT_RESOURCE_REQUEST_MEM: 100Mi #
VM_VMAUTHDEFAULT_RESOURCE_REQUEST_CPU: 50m #
VM_VMAUTHDEFAULT_RESOURCE_REQUEST_EPHEMERAL_STORAGE: unlimited #
VM_VLCLUSTERDEFAULT_USEDEFAULTRESOURCES: true #
VM_VLCLUSTERDEFAULT_VLSELECTDEFAULT_IMAGE: victoriametrics/victoria-logs #
VM_VLCLUSTERDEFAULT_VLSELECTDEFAULT_VERSION: ${VM_LOGS_VERSION} #
VM_VLCLUSTERDEFAULT_VLSELECTDEFAULT_PORT: 9471 #
VM_VLCLUSTERDEFAULT_VLSELECTDEFAULT_RESOURCE_LIMIT_MEM: 1024Mi #
VM_VLCLUSTERDEFAULT_VLSELECTDEFAULT_RESOURCE_LIMIT_CPU: 1000m #
VM_VLCLUSTERDEFAULT_VLSELECTDEFAULT_RESOURCE_LIMIT_EPHEMERAL_STORAGE: unlimited #
VM_VLCLUSTERDEFAULT_VLSELECTDEFAULT_RESOURCE_REQUEST_MEM: 256Mi #
VM_VLCLUSTERDEFAULT_VLSELECTDEFAULT_RESOURCE_REQUEST_CPU: 100m #
VM_VLCLUSTERDEFAULT_VLSELECTDEFAULT_RESOURCE_REQUEST_EPHEMERAL_STORAGE: unlimited #
VM_VLCLUSTERDEFAULT_VLSTORAGEDEFAULT_IMAGE: victoriametrics/victoria-logs #
VM_VLCLUSTERDEFAULT_VLSTORAGEDEFAULT_VERSION: ${VM_LOGS_VERSION} #
VM_VLCLUSTERDEFAULT_VLSTORAGEDEFAULT_PORT: 9491 #
VM_VLCLUSTERDEFAULT_VLSTORAGEDEFAULT_RESOURCE_LIMIT_MEM: 2048Mi #
VM_VLCLUSTERDEFAULT_VLSTORAGEDEFAULT_RESOURCE_LIMIT_CPU: 1000m #
VM_VLCLUSTERDEFAULT_VLSTORAGEDEFAULT_RESOURCE_LIMIT_EPHEMERAL_STORAGE: unlimited #
VM_VLCLUSTERDEFAULT_VLSTORAGEDEFAULT_RESOURCE_REQUEST_MEM: 512Mi #
VM_VLCLUSTERDEFAULT_VLSTORAGEDEFAULT_RESOURCE_REQUEST_CPU: 200m #
VM_VLCLUSTERDEFAULT_VLSTORAGEDEFAULT_RESOURCE_REQUEST_EPHEMERAL_STORAGE: unlimited #
VM_VLCLUSTERDEFAULT_VLINSERTDEFAULT_IMAGE: victoriametrics/victoria-logs #
VM_VLCLUSTERDEFAULT_VLINSERTDEFAULT_VERSION: ${VM_LOGS_VERSION} #
VM_VLCLUSTERDEFAULT_VLINSERTDEFAULT_PORT: 9481 #
VM_VLCLUSTERDEFAULT_VLINSERTDEFAULT_RESOURCE_LIMIT_MEM: 1024Mi #
VM_VLCLUSTERDEFAULT_VLINSERTDEFAULT_RESOURCE_LIMIT_CPU: 1000m #
VM_VLCLUSTERDEFAULT_VLINSERTDEFAULT_RESOURCE_LIMIT_EPHEMERAL_STORAGE: unlimited #
VM_VLCLUSTERDEFAULT_VLINSERTDEFAULT_RESOURCE_REQUEST_MEM: 256Mi #
VM_VLCLUSTERDEFAULT_VLINSERTDEFAULT_RESOURCE_REQUEST_CPU: 100m #
VM_VLCLUSTERDEFAULT_VLINSERTDEFAULT_RESOURCE_REQUEST_EPHEMERAL_STORAGE: unlimited #
VM_VTCLUSTERDEFAULT_USEDEFAULTRESOURCES: true #
VM_VTCLUSTERDEFAULT_SELECT_IMAGE: victoriametrics/victoria-traces #
VM_VTCLUSTERDEFAULT_SELECT_VERSION: ${VM_TRACES_VERSION} #
VM_VTCLUSTERDEFAULT_SELECT_PORT: 10471 #
VM_VTCLUSTERDEFAULT_SELECT_RESOURCE_LIMIT_MEM: 1024Mi #
VM_VTCLUSTERDEFAULT_SELECT_RESOURCE_LIMIT_CPU: 1000m #
VM_VTCLUSTERDEFAULT_SELECT_RESOURCE_LIMIT_EPHEMERAL_STORAGE: unlimited #
VM_VTCLUSTERDEFAULT_SELECT_RESOURCE_REQUEST_MEM: 256Mi #
VM_VTCLUSTERDEFAULT_SELECT_RESOURCE_REQUEST_CPU: 100m #
VM_VTCLUSTERDEFAULT_SELECT_RESOURCE_REQUEST_EPHEMERAL_STORAGE: unlimited #
VM_VTCLUSTERDEFAULT_STORAGE_IMAGE: victoriametrics/victoria-traces #
VM_VTCLUSTERDEFAULT_STORAGE_VERSION: ${VM_TRACES_VERSION} #
VM_VTCLUSTERDEFAULT_STORAGE_PORT: 10491 #
VM_VTCLUSTERDEFAULT_STORAGE_RESOURCE_LIMIT_MEM: 2048Mi #
VM_VTCLUSTERDEFAULT_STORAGE_RESOURCE_LIMIT_CPU: 1000m #
VM_VTCLUSTERDEFAULT_STORAGE_RESOURCE_LIMIT_EPHEMERAL_STORAGE: unlimited #
VM_VTCLUSTERDEFAULT_STORAGE_RESOURCE_REQUEST_MEM: 512Mi #
VM_VTCLUSTERDEFAULT_STORAGE_RESOURCE_REQUEST_CPU: 200m #
VM_VTCLUSTERDEFAULT_STORAGE_RESOURCE_REQUEST_EPHEMERAL_STORAGE: unlimited #
VM_VTCLUSTERDEFAULT_INSERT_IMAGE: victoriametrics/victoria-traces #
VM_VTCLUSTERDEFAULT_INSERT_VERSION: ${VM_TRACES_VERSION} #
VM_VTCLUSTERDEFAULT_INSERT_PORT: 10481 #
VM_VTCLUSTERDEFAULT_INSERT_RESOURCE_LIMIT_MEM: 1024Mi #
VM_VTCLUSTERDEFAULT_INSERT_RESOURCE_LIMIT_CPU: 1000m #
VM_VTCLUSTERDEFAULT_INSERT_RESOURCE_LIMIT_EPHEMERAL_STORAGE: unlimited #
VM_VTCLUSTERDEFAULT_INSERT_RESOURCE_REQUEST_MEM: 256Mi #
VM_VTCLUSTERDEFAULT_INSERT_RESOURCE_REQUEST_CPU: 100m #
VM_VTCLUSTERDEFAULT_INSERT_RESOURCE_REQUEST_EPHEMERAL_STORAGE: unlimited #
VM_ENABLEDPROMETHEUSCONVERTER_PODMONITOR: true #
VM_ENABLEDPROMETHEUSCONVERTER_SERVICESCRAPE: true #
VM_ENABLEDPROMETHEUSCONVERTER_PROMETHEUSRULE: true #
VM_ENABLEDPROMETHEUSCONVERTER_PROBE: true #
VM_ENABLEDPROMETHEUSCONVERTER_ALERTMANAGERCONFIG: true #
VM_ENABLEDPROMETHEUSCONVERTER_SCRAPECONFIG: true #
VM_PROMETHEUSCONVERTERADDARGOCDIGNOREANNOTATIONS: false #
adds compare-options and sync-options for prometheus objects converted by operator. It helps to properly use converter with ArgoCD
VM_ENABLEDPROMETHEUSCONVERTEROWNERREFERENCES: false #
VM_FILTERPROMETHEUSCONVERTERLABELPREFIXES: - #
allows filtering for converted labels, labels with matched prefix will be ignored
VM_FILTERPROMETHEUSCONVERTERANNOTATIONPREFIXES: - #
allows filtering for converted annotations, annotations with matched prefix will be ignored
VM_CLUSTERDOMAINNAME: - #
Defines domain name suffix for in-cluster addresses most known ClusterDomainName is .cluster.local
VM_APPREADYTIMEOUT: 80s #
Defines deadline for deployment/statefulset to transit into ready state to wait for transition to ready state
VM_PODWAITREADYTIMEOUT: 80s #
Defines single pod deadline to wait for transition to ready state
VM_PODWAITREADYINTERVALCHECK: 5s #
Defines poll interval for pods ready check at statefulset rollout update
VM_FORCERESYNCINTERVAL: 60s #
configures force resync interval for VMAgent, VMAlert, VMAlertmanager and VMAuth.
VM_ENABLESTRICTSECURITY: false #
EnableStrictSecurity will add default securityContext to pods and containers created by operator Default PodSecurityContext include: 1. RunAsNonRoot: true 2. RunAsUser/RunAsGroup/FSGroup: 65534 ‘65534’ refers to ’nobody’ in all the used default images like alpine, busybox. If you’re using customize image, please make sure ‘65534’ is a valid uid in there or specify SecurityContext. 3. FSGroupChangePolicy: &onRootMismatch If KubeVersion>=1.20, use FSGroupChangePolicy="onRootMismatch" to skip the recursive permission change when the root of the volume already has the correct permissions 4. SeccompProfile: type: RuntimeDefault Use RuntimeDefault seccomp profile by default, which is defined by the container runtime, instead of using the Unconfined (seccomp disabled) mode. Default container SecurityContext include: 1. AllowPrivilegeEscalation: false 2. ReadOnlyRootFilesystem: true 3. Capabilities: drop: - all turn off EnableStrictSecurity by default, see https://github.com/VictoriaMetrics/operator/issues/749 for details

Modify environment variables #

To add environment variables to the operator, use the following Kustomize -based approach. This method assumes the operator was installed using the Quick Start guide . Alternatively, you can edit the manifest file directly. If you used Helm, apply the changes using Helm’s values configuration.

The example below customize CPU\Memory default limits for VMSingle resource. The commands create a patch add-operator-envs/patch.yaml that adds environment variables to the operator deployment, a add-operator-envs/kustomization.yaml configuration to apply the patch, and then call kustomize build to rewrite the operator-and-crds.yaml file with the applied changes:

      mkdir -p add-operator-envs;

cat <<'EOF' > add-operator-envs/patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vm-operator
  namespace: vm
spec:
  template:
    spec:
      containers:
      - name: manager
        env:
        - name: VM_VMSINGLEDEFAULT_RESOURCE_LIMIT_MEM
          value: "3000Mi"
        - name: VM_VMSINGLEDEFAULT_RESOURCE_LIMIT_CPU
          value: "2400m"
EOF

cat <<'EOF' > add-operator-envs/kustomization.yaml
resources:
  - ../operator-and-crds.yaml

patches:
  - path: patch.yaml
    target:
      kind: Deployment
      name: vm-operator
EOF

kustomize build add-operator-envs -o operator-and-crds.yaml --load-restrictor=LoadRestrictionsNone;
cat operator-and-crds.yaml | grep -E -A 1 "VM_VMSINGLEDEFAULT_RESOURCE_LIMIT_MEM|VM_VMSINGLEDEFAULT_RESOURCE_LIMIT_CPU";

# Output:
#        - name: VM_VMSINGLEDEFAULT_RESOURCE_LIMIT_MEM
#          value: 3000Mi
#        - name: VM_VMSINGLEDEFAULT_RESOURCE_LIMIT_CPU
#          value: 2400m
    

Apply the changes to the operator deployment:

      kubectl apply -f operator-and-crds.yaml;
kubectl -n vm rollout status deployment vm-operator --watch=true;

# Output:
# Waiting for deployment "vm-operator" rollout to finish: 1 old replicas are pending termination...
# Waiting for deployment "vm-operator" rollout to finish: 1 old replicas are pending termination...
# deployment "vm-operator" successfully rolled out
    

Run this command to print modified environment variables:

      kubectl get deployment -n vm vm-operator \
    -o jsonpath='{range .spec.template.spec.containers[?(@.name=="manager")].env[*]}{.name}{"\n"}{end}';

# Output:
# WATCH_NAMESPACE
# VM_VMSINGLEDEFAULT_RESOURCE_LIMIT_MEM
# VM_VMSINGLEDEFAULT_RESOURCE_LIMIT_CPU
    

Labels #

Each managed by operator CRs resource has a set of labels, which is a result of spec.managedMetadata.labels and predefined immutable labels merge. Immutable labels are needed to simplify dependent resources discovery and guarantee predictability in resources interconnection. List of immutable labels:

  • app.kubernetes.io/name
  • app.kubernetes.io/instance
  • app.kubernetes.io/component
  • app.kubernetes.io/part-of
  • managed-by

In case if spec.managedMetadata.labels and immutable labels collision, least ones have higher priority.

Flags #

Run this command to see all flags your operator supports:

      OPERATOR_POD_NAME=$(kubectl get pod -l "app.kubernetes.io/name=victoria-metrics-operator"  -n vm -o jsonpath="{.items[0].metadata.name}");
kubectl exec -n vm "$OPERATOR_POD_NAME" -- /app --help 2>&1;

# Output:
# Usage of /app:
#   -client.burst int
#       defines K8s client burst (default 100)
# ...
    

These are the latest operator flags:

      Usage of bin/operator:
  -client.burst int
    	defines K8s client burst (default 100)
  -client.qps int
    	defines K8s client QPS. The value should be increased for the cluster with large number of objects > 10_000. (default 50)
  -controller.cacheSyncTimeout duration
    	controls timeout for caches to be synced. (default 3m0s)
  -controller.disableCRDOwnership
    	disables CRD ownership add to cluster wide objects, must be disabled for clusters, lower than v1.16.0
  -controller.disableCacheFor string
    	disables client for cache for API resources. Supported objects - namespace,pod,service,secret,configmap,deployment,statefulset (default "configmap,secret")
  -controller.disableReconcileFor string
    	disables reconcile controllers for given list of comma separated CRD names. For example - VMCluster,VMSingle,VMAuth.Note, child controllers still require parent object CRDs.
  -controller.maxConcurrentReconciles int
    	Configures number of concurrent reconciles. It should improve performance for clusters with many objects. (default 15)
  -controller.prometheusCRD.resyncPeriod duration
    	Configures resync period for prometheus CRD converter. Disabled by default
  -controller.statusLastUpdateTimeTTL duration
    	Configures TTL for LastUpdateTime status.conditions fields. It's used to detect stale parent objects on child objects. Like VMAlert->VMRule .status.Conditions.Type (default 1h0m0s)
  -default.kubernetesVersion.major uint
    	Major version of kubernetes server, if operator cannot parse actual kubernetes response (default 1)
  -default.kubernetesVersion.minor uint
    	Minor version of kubernetes server, if operator cannot parse actual kubernetes response (default 21)
  -disableSecretKeySpaceTrim
    	disables trim of space at Secret/Configmap value content. It's a common mistake to put new line to the base64 encoded secret value.
  -health-probe-bind-address string
    	The address the probes (health, ready) binds to. (default ":8081")
  -leader-elect
    	Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.
  -leader-elect-id string
    	Defines the name of the resource that leader election will use for holding the leader lock. (default "57410f0d.victoriametrics.com")
  -leader-elect-lease-duration duration
    	Defines the duration that non-leader candidates will wait to force acquire leadership. This is measured against time of last observed ack. (default 15s)
  -leader-elect-namespace string
    	Defines optional namespace name in which the leader election resource will be created. By default, uses in-cluster namespace name.
  -leader-elect-renew-deadline duration
    	Defines the duration that the acting controlplane will retry refreshing leadership lock before giving up. (default 10s)
  -loggerJSONFields string
    	Allows renaming fields in JSON formatted logsExample: "ts:timestamp,msg:message" renames "ts" to "timestamp" and "msg" to "message".Supported fields: ts, level, caller, msg
  -metrics-bind-address string
    	The address the metric endpoint binds to. (default ":8080")
  -mtls.CAName string
    	Optional name of TLS Root CA for verifying client certificates at the corresponding -metrics-bind-address when -mtls.enable is enabled. By default the host system TLS Root CA is used for client certificate verification.  (default "clietCA.crt")
  -mtls.enable
    	Whether to require valid client certificate for https requests to the corresponding -metrics-bind-address. This flag works only if -tls.enable flag is set.
  -pprof-addr string
    	The address for pprof/debug API. Empty value disables server (default ":8435")
  -printDefaults
    	print all variables with their default values and exit
  -printFormat string
    	output format for --printDefaults. Can be table, json, yaml or list (default "table")
  -tls.certDir string
    	root directory for metrics webserver cert, key and mTLS CA. (default "/tmp/k8s-metrics-server/serving-certs")
  -tls.certName string
    	name of metric server Tls certificate inside tls.certDir. Default -  (default "tls.crt")
  -tls.enable
    	enables secure tls (https) for metrics webserver.
  -tls.keyName string
    	name of metric server Tls key inside tls.certDir. Default - tls.key (default "tls.key")
  -version
    	Show operator version
  -webhook.certDir string
    	root directory for webhook cert and key (default "/tmp/k8s-webhook-server/serving-certs/")
  -webhook.certName string
    	name of webhook server Tls certificate inside tls.certDir (default "tls.crt")
  -webhook.enable
    	adds webhook server, you must mount cert and key or use cert-manager
  -webhook.keyName string
    	name of webhook server Tls key inside tls.certDir (default "tls.key")
  -webhook.port int
    	port to start webhook server on (default 9443)
  -zap-devel
    	Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error)
  -zap-encoder value
    	Zap log encoding (one of 'json' or 'console')
  -zap-log-level value
    	Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', 'panic'or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
    	Note: warn is missing by design due to warn level not being supported by controller-runtime
    	See: https://dave.cheney.net/2015/11/05/lets-talk-about-logging and https://github.com/kubernetes-sigs/controller-runtime/issues/2002 for more information.
  -zap-stacktrace-level value
    	Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
  -zap-time-encoding value
    	Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'.
    

Config reloader flags #

These are the latest config reloader flags:

      Usage of bin/config-reloader:
  -config-envsubst-file string
    	target file, where content of configFile or configSecret would be written
  -config-file string
    	config file watched by reloader
  -config-secret-key string
    	key of config-secret-name for retrieving configuration from (default "config.yaml.gz")
  -config-secret-name string
    	name of kubernetes secret in form of namespace/name
  -delay-interval duration
    	delays config reload time. (default 3s)
  -enableTCP6
    	Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used
  -envflag.enable
    	Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#environment-variables for more details
  -envflag.prefix string
    	Prefix for environment variables if -envflag.enable is set
  -filestream.disableFadvise
    	Whether to disable fadvise() syscall when reading large data files. The fadvise() syscall prevents from eviction of recently accessed data from OS page cache during background merges and backups. In some rare cases it is better to disable the syscall if it uses too much CPU
  -flagsAuthKey value
    	Auth key for /flags endpoint. It must be passed via authKey query arg. It overrides -httpAuth.*
    	Flag value can be read from the given file when using -flagsAuthKey=file:///abs/path/to/file or -flagsAuthKey=file://./relative/path/to/file.
    	Flag value can be read from the given http/https url when using -flagsAuthKey=http://host/path or -flagsAuthKey=https://host/path
  -fs.maxConcurrency int
    	The maximum number of concurrent goroutines to work with files; smaller values may help reducing Go scheduling latency on systems with small number of CPU cores; higher values may help reducing data ingestion latency on systems with high-latency storage such as NFS or Ceph (default 64)
  -http.connTimeout duration
    	Incoming connections to -httpListenAddr are closed after the configured timeout. This may help evenly spreading load among a cluster of services behind TCP-level load balancer. Zero value disables closing of incoming connections (default 2m0s)
  -http.disableCORS
    	Disable CORS for all origins (*)
  -http.disableKeepAlive
    	Whether to disable HTTP keep-alive for incoming connections at -httpListenAddr
  -http.disableResponseCompression
    	Disable compression of HTTP responses to save CPU resources. By default, compression is enabled to save network bandwidth
  -http.header.csp string
    	Value for 'Content-Security-Policy' header, recommended: "default-src 'self'"
  -http.header.frameOptions string
    	Value for 'X-Frame-Options' header
  -http.header.hsts string
    	Value for 'Strict-Transport-Security' header, recommended: 'max-age=31536000; includeSubDomains'
  -http.idleConnTimeout duration
    	Timeout for incoming idle http connections (default 1m0s)
  -http.listenAddr string
    	http server listen addr (default ":8435")
  -http.maxGracefulShutdownDuration duration
    	The maximum duration for a graceful shutdown of the HTTP server. A highly loaded server may require increased value for a graceful shutdown (default 7s)
  -http.pathPrefix string
    	An optional prefix to add to all the paths handled by http server. For example, if '-http.pathPrefix=/foo/bar' is set, then all the http requests will be handled on '/foo/bar/*' paths. This may be useful for proxied requests. See https://www.robustperception.io/using-external-urls-and-proxies-with-prometheus
  -http.shutdownDelay duration
    	Optional delay before http server shutdown. During this delay, the server returns non-OK responses from /health page, so load balancers can route new requests to other servers
  -httpAuth.password value
    	Password for HTTP server's Basic Auth. The authentication is disabled if -httpAuth.username is empty
    	Flag value can be read from the given file when using -httpAuth.password=file:///abs/path/to/file or -httpAuth.password=file://./relative/path/to/file.
    	Flag value can be read from the given http/https url when using -httpAuth.password=http://host/path or -httpAuth.password=https://host/path
  -httpAuth.username string
    	Username for HTTP server's Basic Auth. The authentication is disabled if empty. See also -httpAuth.password
  -internStringCacheExpireDuration duration
    	The expiry duration for caches for interned strings. See https://en.wikipedia.org/wiki/String_interning . See also -internStringMaxLen and -internStringDisableCache (default 6m0s)
  -internStringDisableCache
    	Whether to disable caches for interned strings. This may reduce memory usage at the cost of higher CPU usage. See https://en.wikipedia.org/wiki/String_interning . See also -internStringCacheExpireDuration and -internStringMaxLen
  -internStringMaxLen int
    	The maximum length for strings to intern. A lower limit may save memory at the cost of higher CPU usage. See https://en.wikipedia.org/wiki/String_interning . See also -internStringDisableCache and -internStringCacheExpireDuration (default 500)
  -loggerDisableTimestamps
    	Whether to disable writing timestamps in logs
  -loggerErrorsPerSecondLimit int
    	Per-second limit on the number of ERROR messages. If more than the given number of errors are emitted per second, the remaining errors are suppressed. Zero values disable the rate limit
  -loggerFormat string
    	Format for logs. Possible values: default, json (default "default")
  -loggerJSONFields string
    	Allows renaming fields in JSON formatted logs. Example: "ts:timestamp,msg:message" renames "ts" to "timestamp" and "msg" to "message". Supported fields: ts, level, caller, msg
  -loggerLevel string
    	Minimum level of errors to log. Possible values: INFO, WARN, ERROR, FATAL, PANIC (default "INFO")
  -loggerMaxArgLen int
    	The maximum length of a single logged argument. Longer arguments are replaced with 'arg_start..arg_end', where 'arg_start' and 'arg_end' is prefix and suffix of the arg with the length not exceeding -loggerMaxArgLen / 2 (default 5000)
  -loggerOutput string
    	Output for the logs. Supported values: stderr, stdout (default "stderr")
  -loggerTimezone string
    	Timezone to use for timestamps in logs. Timezone must be a valid IANA Time Zone. For example: America/New_York, Europe/Berlin, Etc/GMT+3 or Local (default "UTC")
  -loggerWarnsPerSecondLimit int
    	Per-second limit on the number of WARN messages. If more than the given number of warns are emitted per second, then the remaining warns are suppressed. Zero values disable the rate limit
  -memory.allowedBytes size
    	Allowed size of system memory VictoriaMetrics caches may occupy. This option overrides -memory.allowedPercent if set to a non-zero value. Too low a value may increase the cache miss rate usually resulting in higher CPU and disk IO usage. Too high a value may evict too much data from the OS page cache resulting in higher disk IO usage
    	Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0)
  -memory.allowedPercent float
    	Allowed percent of system memory VictoriaMetrics caches may occupy. See also -memory.allowedBytes. Too low a value may increase cache miss rate usually resulting in higher CPU and disk IO usage. Too high a value may evict too much data from the OS page cache which will result in higher disk IO usage (default 60)
  -metrics.exposeMetadata
    	Whether to expose TYPE and HELP metadata at the /metrics page, which is exposed at -httpListenAddr . The metadata may be needed when the /metrics page is consumed by systems, which require this information. For example, Managed Prometheus in Google Cloud - https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting#missing-metric-type
  -metricsAuthKey value
    	Auth key for /metrics endpoint. It must be passed via authKey query arg. It overrides -httpAuth.*
    	Flag value can be read from the given file when using -metricsAuthKey=file:///abs/path/to/file or -metricsAuthKey=file://./relative/path/to/file.
    	Flag value can be read from the given http/https url when using -metricsAuthKey=http://host/path or -metricsAuthKey=https://host/path
  -only-init-config
    	enables will read config and write to config-envsubst-file once before exit
  -pprofAuthKey value
    	Auth key for /debug/pprof/* endpoints. It must be passed via authKey query arg. It overrides -httpAuth.*
    	Flag value can be read from the given file when using -pprofAuthKey=file:///abs/path/to/file or -pprofAuthKey=file://./relative/path/to/file.
    	Flag value can be read from the given http/https url when using -pprofAuthKey=http://host/path or -pprofAuthKey=https://host/path
  -reload-url string
    	reload URL to trigger config reload (default "http://127.0.0.1:8429/-/reload")
  -reload-url-auth-key value
    	authKey for config reload API requests
    	Flag value can be read from the given file when using -reload-url-auth-key=file:///abs/path/to/file or -reload-url-auth-key=file://./relative/path/to/file.
    	Flag value can be read from the given http/https url when using -reload-url-auth-key=http://host/path or -reload-url-auth-key=https://host/path
  -reload-use-proxy-protocol
    	enables proxy-protocol for reload connections.
  -reload.tlsCAFile string
    	Optional path to client-side TLS CA file to use when connecting to -reload-url
  -reload.tlsCertFile string
    	Optional path to client-side TLS certificate file to use when connecting to -reload-url
  -reload.tlsInsecureSkipVerify
    	Whether to skip tls verification when connecting to -reload-url (default true)
  -reload.tlsKeyFile string
    	Optional path to client-side TLS key file to use when connecting to -reload-url
  -reload.tlsServerName string
    	Optional TLS server name to use for connections to -reload-url.
  -resync-interval duration
    	interval for force resync of the last configuration
  -rules-dir array
    	the same as watched-dir, legacy
    	Supports an array of values separated by comma or specified via multiple flags.
    	Each array item can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
  -secret.flags array
    	Comma-separated list of flag names with secret values. Values for these flags are hidden in logs and on /metrics page
    	Supports an array of values separated by comma or specified via multiple flags.
    	Each array item can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
  -tls array
    	Whether to enable TLS for incoming HTTP requests at the given -httpListenAddr (aka https). -tlsCertFile and -tlsKeyFile must be set if -tls is set. See also -mtls
    	Supports array of values separated by comma or specified via multiple flags.
    	Empty values are set to false.
  -tlsCertFile array
    	Path to file with TLS certificate for the corresponding -httpListenAddr if -tls is set. Prefer ECDSA certs instead of RSA certs as RSA certs are slower. The provided certificate file is automatically re-read every second, so it can be dynamically updated. See also -tlsAutocertHosts
    	Supports an array of values separated by comma or specified via multiple flags.
    	Each array item can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
  -tlsCipherSuites array
    	Optional list of TLS cipher suites for incoming requests over HTTPS if -tls is set. See the list of supported cipher suites at https://pkg.go.dev/crypto/tls#pkg-constants
    	Supports an array of values separated by comma or specified via multiple flags.
    	Each array item can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
  -tlsKeyFile array
    	Path to file with TLS key for the corresponding -httpListenAddr if -tls is set. The provided key file is automatically re-read every second, so it can be dynamically updated. See also -tlsAutocertHosts
    	Supports an array of values separated by comma or specified via multiple flags.
    	Each array item can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
  -tlsMinVersion array
    	Optional minimum TLS version to use for the corresponding -httpListenAddr if -tls is set. Supported values: TLS10, TLS11, TLS12, TLS13
    	Supports an array of values separated by comma or specified via multiple flags.
    	Each array item can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
  -version
    	Show VictoriaMetrics version
  -watch-interval duration
    	no-op for prometheus config-reloader compatibility (default 3m0s)
  -watched-dir array
    	directory to watch non-recursively
    	Supports an array of values separated by comma or specified via multiple flags.
    	Each array item can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
  -webhook-method string
    	the HTTP method url to use to send the webhook (default "GET")
    

Modify flags #

To add flags to the operator, use the following Kustomize -based approach. This method assumes the operator was installed using the Quick Start guide . Alternatively, you can edit the manifest file directly. If you used Helm, apply the changes using Helm’s values configuration.

The example below shows how to change log level. The commands create a patch add-operator-flag/patch.yaml that adds command line argument to the operator deployment, a add-operator-flag/kustomization.yaml configuration to apply the patch, and then call kustomize build to rewrite the operator-and-crds.yaml file with the applied changes:

      mkdir -p add-operator-flag;

cat <<'EOF' > add-operator-flag/patch.yaml
- op: add
  path: /spec/template/spec/containers/0/args/-
  value: '-zap-log-level=debug'
EOF

cat <<'EOF' > add-operator-flag/kustomization.yaml
resources:
  - ../operator-and-crds.yaml

patches:
  - path: patch.yaml
    target:
      kind: Deployment
      name: vm-operator
EOF

kustomize build add-operator-flag -o operator-and-crds.yaml --load-restrictor=LoadRestrictionsNone;
cat operator-and-crds.yaml | grep "zap-log-level";

# Output:
#        - -zap-log-level=debug
    

Apply the changes to the operator deployment:

      kubectl apply -f operator-and-crds.yaml;
kubectl -n vm rollout status deployment vm-operator --watch=true;

# Output:
# Waiting for deployment "vm-operator" rollout to finish: 1 old replicas are pending termination...
# Waiting for deployment "vm-operator" rollout to finish: 1 old replicas are pending termination...
# deployment "vm-operator" successfully rolled out
    

Run this command to print modified flags variables:

      kubectl get deployment -n vm vm-operator \
  -o jsonpath='{range .spec.template.spec.containers[?(@.name=="manager")]}{.args[*]}{end}{"\n"}'

# Output:
# --leader-elect --health-probe-bind-address=:8081 --metrics-bind-address=:8080 -zap-log-level=debug
    

Scrape operator metrics #

To collect the operator metrics, you can create a VMServiceScrape resource. Configure it to collects metrics from the pods that match the operator labels. Apply scrape config in vm namespace, same where the operator is running.

The example below works if you installed the operator using the Quick Start - Operator section. You may need to update the match labels and\or namespace to fit your own operator setup.

      cat <<'EOF' > operator-scrape.yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
  name: operator-service-scrape
  namespace: vm
spec:
  selector:
    matchLabels:
      # You might need to change the labels below
      app.kubernetes.io/instance: default
      app.kubernetes.io/name: victoria-metrics-operator
  endpoints:
    - port: http
  # Uncomment the lines below if the VMServiceScrape is applied in  a namespace
  # different from the one where the operator is running.
  # namespaceSelector:
  #  matchNames:
  #    - default
EOF

kubectl apply -f operator-scrape.yaml;
kubectl wait -n vm --for=jsonpath='{.status.updateStatus}'=operational vmservicescrape/operator-service-scrape;

# Output:
# vmservicescrape.operator.victoriametrics.com/operator-service-scrape created
# vmservicescrape.operator.victoriametrics.com/operator-service-scrape condition met
    

You can check if the operator metrics are collected correctly by using the vmagent UI. Note, It may take a minute or two for vmagent to load the new scrape config and begin collecting the metrics.

You can find instructions for accessing the vmagent UI in the Quick Start - Scraping section.

Conversion of prometheus-operator objects #

You can read detailed instructions about configuring prometheus-objects conversion in this document .

Helm-charts #

In Helm charts some important configuration parameters are implemented as separate flags in values.yaml:

victoria-metrics-k8s-stack #

For possible values refer to parameters .

Also, checkout here possible ENV variables to configure operator behaviour. ENV variables can be set in the victoria-metrics-operator.env section.

      # values.yaml

victoria-metrics-operator:
  image:
    # -- Image repository
    repository: victoriametrics/operator
    # -- Image tag
    tag: v0.35.0
    # -- Image pull policy
    pullPolicy: IfNotPresent

  # -- Tells helm to remove CRD after chart remove
  cleanupCRD: true
  cleanupImage:
    repository: gcr.io/google_containers/hyperkube
    tag: v1.18.0
    pullPolicy: IfNotPresent

  operator:
    # -- By default, operator converts prometheus-operator objects.
    disable_prometheus_converter: false
    # -- Compare-options and sync-options for prometheus objects converted by operator for properly use with ArgoCD
    prometheus_converter_add_argocd_ignore_annotations: false
    # -- Enables ownership reference for converted prometheus-operator objects,
    # it will remove corresponding victoria-metrics objects in case of deletion prometheus one.
    enable_converter_ownership: false
    # -- By default, operator creates psp for its objects.
    psp_auto_creation_enabled: true

  # -- extra settings for the operator deployment. full list Ref: https://docs.victoriametrics.com/operator/vars
  env:
    # -- default version for vmsingle
    - name: VM_VMSINGLEDEFAULT_VERSION
      value: v1.45.0
    # -- container registry name prefix, e.g. docker.io
    - name: VM_CONTAINERREGISTRY
      value: ""
    # -- image for custom reloader
    - name: VM_CONFIG_RELOADER_IMAGE
      value: victoriametrics/operator:config-reloader-v0.32.0

  # By default, the operator will watch all the namespaces
  # If you want to override this behavior, specify the namespace it needs to watch separated by a comma.
  # Ex: my_namespace1,my_namespace2
  watchNamespace: ""

  # Count of operator instances (can be increased for HA mode)
  replicaCount: 1

  # -- VM operator log level
  # -- possible values: info and error.
  logLevel: "info"

  # -- Resource object
  resources:
    {}
    # limits:
    #   cpu: 120m
    #   memory: 320Mi
    # requests:
    #   cpu: 80m
    #   memory: 120Mi
    

victoria-metrics-operator #

For possible values refer to parameters .

Also, checkout here possible ENV variables to configure operator behaviour. ENV variables can be set in the env section.

      # values.yaml

image:
  # -- Image repository
  repository: victoriametrics/operator
  # -- Image tag
  tag: v0.35.0
  # -- Image pull policy
  pullPolicy: IfNotPresent

operator:
  # -- By default, operator converts prometheus-operator objects.
  disable_prometheus_converter: false
  # -- Compare-options and sync-options for prometheus objects converted by operator for properly use with ArgoCD
  prometheus_converter_add_argocd_ignore_annotations: false
  # -- Enables ownership reference for converted prometheus-operator objects,
  # it will remove corresponding victoria-metrics objects in case of deletion prometheus one.
  enable_converter_ownership: false
  # -- By default, operator creates psp for its objects.
  psp_auto_creation_enabled: true

# -- extra settings for the operator deployment. full list Ref: https://docs.victoriametrics.com/operator/vars
env:
  # -- default version for vmsingle
  - name: VM_VMSINGLEDEFAULT_VERSION
    value: v1.45.0
  # -- container registry name prefix, e.g. docker.io
  - name: VM_CONTAINERREGISTRY
    value: ""
  # -- image for custom reloader
  - name: VM_CONFIG_RELOADER_IMAGE
    value: victoriametrics/operator:config-reloader-v0.32.0

# By default, the operator will watch all the namespaces
# If you want to override this behavior, specify the namespace it needs to watch separated by a comma.
# Ex: my_namespace1,my_namespace2
watchNamespace: ""

# Count of operator instances (can be increased for HA mode)
replicaCount: 1

# -- VM operator log level
# -- possible values: info and error.
logLevel: "info"

# -- Resource object
resources:
  {}
  # limits:
  #   cpu: 120m
  #   memory: 320Mi
  # requests:
  #   cpu: 80m
  #   memory: 120Mi
    

Namespaced mode #

By default, the operator will watch all namespaces, but it can be configured to watch only specific namespace or multiple namespaces.

If you want to override this behavior, specify the namespace:

  • in the WATCH_NAMESPACE environment variable.
  • in the watchNamespace field in the values.yaml file of helm-charts.

The operator supports comma separated namespace names for this setting.

If namespaced mode is enabled, operator uses a limited set of features:

  • it cannot make any cluster wide API calls.
  • it cannot assign rbac permissions for managed resource. It must be done manually via serviceAccount.
  • it ignores namespaceSelector fields at CRD objects and uses WATCH_NAMESPACE value for object matching.

At each namespace operator must have a set of required permissions, an example can be found at this file .

IPv6 mode #

By default, VM services are accepting only IPv4 TCP and UDP traffic..

To allow accepting IPv6 traffic set VM_ENABLETCP6 environment variable to true. With this option set all VM components and config reloaders become IPv6-compatible.

Monitoring of cluster components #

By default, operator creates VMServiceScrape object for each component that it manages.

You can disable this behaviour with VM_DISABLESELFSERVICESCRAPECREATION environment variable:

      VM_DISABLESELFSERVICESCRAPECREATION=false
    

Also, you can override default configuration for self-scraping with ServiceScrapeSpec field in each deployable resource (vmcluster/select, vmcluster/insert, vmcluster/storage, vmagent, vmalert, vmalertmanager, vmauth, vmsingle):

CRD Validation #

Operator supports validation admission webhook docs

It checks resources configuration and returns errors to caller before resource will be created at kubernetes api. This should reduce errors and simplify debugging.

Validation hooks at operator side must be enabled with flags:

      ./operator
    --webhook.enable
    # optional configuration for certDir and tls names.
    --webhook.certDir=/tmp/k8s-webhook-server/serving-certs/
    --webhook.keyName=tls.key
    --webhook.certName=tls.crt
    

You have to mount correct certificates at give directory. It can be simplified with cert-manager and kustomize command:

      kustomize build config/deployments/webhook/
    

Requirements #

  • Valid certificate with key must be provided to operator
  • Valid CABundle must be added to the ValidatingWebhookConfiguration

The following legacy links are retained for historical reference.

List of command-line flags #

Moved to operator/configuration/#flags