Victoria Metrics Operator
Prerequisites #
- Install the follow packages:
git,kubectl,helm,helm-docs. See this tutorial. - PV support on underlying infrastructure.
ArgoCD issues #
When running operator using ArgoCD without Cert Manager (.Values.admissionWebhooks.certManager.enabled: false) it will rerender webhook certificates
on each sync since Helm lookup function is not respected by ArgoCD. To prevent this please update you operator Application spec.syncPolicy and spec.ignoreDifferences with a following:
apiVersion: argoproj.io/v1alpha1
kind: Application
...
spec:
...
syncPolicy:
syncOptions:
# https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#respect-ignore-difference-configs
# argocd must also ignore difference during apply stage
# otherwise it ll silently override changes and cause a problem
- RespectIgnoreDifferences=true
ignoreDifferences:
- group: ""
kind: Secret
name: <fullname>-validation
namespace: kube-system
jsonPointers:
- /data
- group: admissionregistration.k8s.io
kind: ValidatingWebhookConfiguration
name: <fullname>-admission
jqPathExpressions:
- '.webhooks[]?.clientConfig.caBundle'
where <fullname> is output of {{ include "vm-operator.fullname" }} for your setup
Upgrade guide #
During release an issue with helm CRD was discovered. So for upgrade from version less then 0.1.3 you have to two options:
- use helm management for CRD, enabled by default.
- use own management system, need to add variable: –set createCRD=false.
If you choose helm management, following steps must be done before upgrade:
- define namespace and helm release name variables
export NAMESPACE=default
export RELEASE_NAME=operator
execute kubectl commands:
kubectl get crd | grep victoriametrics.com | awk '{print $1 }' | xargs -i kubectl label crd {} app.kubernetes.io/managed-by=Helm --overwrite
kubectl get crd | grep victoriametrics.com | awk '{print $1 }' | xargs -i kubectl annotate crd {} meta.helm.sh/release-namespace="$NAMESPACE" meta.helm.sh/release-name="$RELEASE_NAME" --overwrite
run helm upgrade command.
Chart Details #
This chart will do the following:
- Rollout victoria metrics operator
How to install #
Access a Kubernetes cluster.
Setup chart repository (can be omitted for OCI repositories) #
Add a chart helm repository with follow commands:
helm repo add vm https://victoriametrics.github.io/helm-charts/
helm repo update
List versions of vm/victoria-metrics-operator chart available to installation:
helm search repo vm/victoria-metrics-operator -l
Install victoria-metrics-operator chart
#
Export default values of victoria-metrics-operator chart to file values.yaml:
For HTTPS repository
helm show values vm/victoria-metrics-operator > values.yamlFor OCI repository
helm show values oci://ghcr.io/victoriametrics/helm-charts/victoria-metrics-operator > values.yaml
Change the values according to the need of the environment in values.yaml file.
Test the installation with command:
For HTTPS repository
helm install vmo vm/victoria-metrics-operator -f values.yaml -n NAMESPACE --debug --dry-runFor OCI repository
helm install vmo oci://ghcr.io/victoriametrics/helm-charts/victoria-metrics-operator -f values.yaml -n NAMESPACE --debug --dry-run
Install chart with command:
For HTTPS repository
helm install vmo vm/victoria-metrics-operator -f values.yaml -n NAMESPACEFor OCI repository
helm install vmo oci://ghcr.io/victoriametrics/helm-charts/victoria-metrics-operator -f values.yaml -n NAMESPACE
Get the pods lists by running this commands:
kubectl get pods -A | grep 'vmo'
Get the application by running this command:
helm list -f vmo -n NAMESPACE
See the history of versions of vmo application with command.
helm history vmo -n NAMESPACE
Validation webhook #
Its possible to use validation of created resources with operator. For now, you need cert-manager to easily certificate management https://cert-manager.io/docs/
admissionWebhooks:
enabled: true
# what to do in case, when operator not available to validate request.
certManager:
# enables cert creation and injection by cert-manager
enabled: true
How to uninstall #
Remove application with command.
helm uninstall vmo -n NAMESPACE
Documentation of Helm Chart #
Install helm-docs following the instructions on this tutorial.
Generate docs with helm-docs command.
cd charts/victoria-metrics-operator
helm-docs
The markdown generation is entirely go template driven. The tool parses metadata from charts and generates a number of sub-templates that can be referenced in a template file (by default README.md.gotmpl). If no template file is provided, the tool has a default internal template that will generate a reasonably formatted README.
Parameters #
The following tables lists the configurable parameters of the chart and their default values.
Change the values according to the need of the environment in victoria-metrics-operator/values.yaml file.
| Key | Type | Default | Description |
|---|---|---|---|
| admissionWebhooks | object |
certManager:
enabled: false
issuer: {}
enabled: true
enabledCRDValidation:
vlogs: true
vmagent: true
vmalert: true
vmalertmanager: true
vmalertmanagerconfig: true
vmauth: true
vmcluster: true
vmrule: true
vmsingle: true
vmuser: true
keepTLSSecret: true
policy: Fail
tls:
caCert: null
cert: null
key: null
| Configures resource validation |
| admissionWebhooks.certManager | object |
enabled: false
issuer: {}
| with keys: tls.key, tls.crt, ca.crt |
| admissionWebhooks.certManager.enabled | bool | false | Enables cert creation and injection by cert-manager. |
| admissionWebhooks.certManager.issuer | object |
{}
| If needed, provide own issuer. Operator will create self-signed if empty. |
| admissionWebhooks.enabled | bool | true | Enables validation webhook. |
| admissionWebhooks.policy | string | Fail | What to do in case, when operator not available to validate request. |
| affinity | object |
{}
| Pod affinity |
| annotations | object |
{}
| Annotations to be added to the all resources |
| crd.cleanup.enabled | bool | false | Tells helm to clean up all the vm resources under this release’s namespace when uninstalling |
| crd.cleanup.image | object |
pullPolicy: IfNotPresent
repository: bitnami/kubectl
tag: ""
| Image configuration for CRD cleanup Job |
| crd.create | bool | true | with this option, if you remove this chart, all crd resources will be deleted with it. |
| env | list |
[]
| extra settings for the operator deployment. Full list here |
| envFrom | list |
[]
| |
| extraArgs | object |
{}
| operator container additional commandline arguments |
| extraContainers | list |
[]
| |
| extraHostPathMounts | list |
[]
| Additional hostPath mounts |
| extraLabels | object |
{}
| Labels to be added to the all resources |
| extraObjects | list |
[]
| Add extra specs dynamically to this chart |
| extraVolumeMounts | list |
[]
| Extra Volume Mounts for the container |
| extraVolumes | list |
[]
| Extra Volumes for the pod |
| fullnameOverride | string | "" | Overrides the full name of server component |
| global.cluster.dnsDomain | string | cluster.local | |
| global.image.registry | string | "" | |
| global.imagePullSecrets | list |
[]
| |
| image | object |
pullPolicy: IfNotPresent
registry: ""
repository: victoriametrics/operator
tag: ""
variant: ""
| operator image configuration |
| image.pullPolicy | string | IfNotPresent | Image pull policy |
| image.registry | string | "" | Image registry |
| image.repository | string | victoriametrics/operator | Image repository |
| image.tag | string | "" | Image tag override Chart.AppVersion |
| imagePullSecrets | list |
[]
| Secret to pull images |
| logLevel | string | info | possible values: info and error. |
| nameOverride | string | "" | VM operatror deployment name override |
| nodeSelector | object |
{}
| Pod’s node selector. Details are here |
| operator.disable_prometheus_converter | bool | false | By default, operator converts prometheus-operator objects. |
| operator.enable_converter_ownership | bool | false | Enables ownership reference for converted prometheus-operator objects, it will remove corresponding victoria-metrics objects in case of deletion prometheus one. |
| operator.prometheus_converter_add_argocd_ignore_annotations | bool | false | Compare-options and sync-options for prometheus objects converted by operator for properly use with ArgoCD |
| operator.useCustomConfigReloader | bool | false | Enables custom config-reloader, bundled with operator. It should reduce vmagent and vmauth config sync-time and make it predictable. |
| podDisruptionBudget | object |
enabled: false
labels: {}
| See |
| podLabels | object |
{}
| |
| podSecurityContext | object |
{}
| |
| probe.liveness | object |
failureThreshold: 3
initialDelaySeconds: 5
periodSeconds: 15
tcpSocket:
port: probe
timeoutSeconds: 5
| Liveness probe |
| probe.readiness | object |
failureThreshold: 3
httpGet:
port: probe
initialDelaySeconds: 5
periodSeconds: 15
timeoutSeconds: 5
| Readiness probe |
| probe.startup | object |
{}
| Startup probe |
| rbac.aggregatedClusterRoles | object |
enabled: true
labels:
admin:
rbac.authorization.k8s.io/aggregate-to-admin: "true"
view:
rbac.authorization.k8s.io/aggregate-to-view: "true"
| create aggregated clusterRoles for CRD readonly and admin permissions |
| rbac.aggregatedClusterRoles.labels | object |
admin:
rbac.authorization.k8s.io/aggregate-to-admin: "true"
view:
rbac.authorization.k8s.io/aggregate-to-view: "true"
| labels attached to according clusterRole |
| rbac.create | bool | true | Specifies whether the RBAC resources should be created |
| replicaCount | int | 1 | |
| resources | object |
{}
| Resource object |
| securityContext | object |
{}
| |
| service.annotations | object |
{}
| |
| service.clusterIP | string | "" | |
| service.externalIPs | string | "" | |
| service.externalTrafficPolicy | string | "" | |
| service.healthCheckNodePort | string | "" | |
| service.ipFamilies | list |
[]
| |
| service.ipFamilyPolicy | string | "" | |
| service.labels | object |
{}
| |
| service.loadBalancerIP | string | "" | |
| service.loadBalancerSourceRanges | list |
[]
| |
| service.servicePort | int | 8080 | |
| service.type | string | ClusterIP | |
| service.webhookPort | int | 9443 | |
| serviceAccount.create | bool | true | Specifies whether a service account should be created |
| serviceAccount.name | string | "" | The name of the service account to use. If not set and create is true, a name is generated using the fullname template |
| serviceMonitor | object |
annotations: {}
basicAuth: {}
enabled: false
extraLabels: {}
interval: ""
relabelings: []
scheme: ""
scrapeTimeout: ""
tlsConfig: {}
| configures monitoring with serviceScrape. VMServiceScrape must be pre-installed |
| tolerations | list |
[]
| Array of tolerations object. Spec is here |
| topologySpreadConstraints | list |
[]
| Pod Topology Spread Constraints. Spec is here |
| watchNamespaces | list |
[]
| By default, the operator will watch all the namespaces If you want to override this behavior, specify the namespace. Operator supports multiple namespaces for watching. |