Prometheus

Design and implementation inspired by prometheus-operator . It’s great a tool for managing monitoring configuration of your applications. VictoriaMetrics operator has api capability with it.

So you can use familiar CRD objects: ServiceMonitor, PodMonitor, PrometheusRule, Probe and AlertmanagerConfig.

Or you can use VictoriaMetrics CRDs:

  • VMServiceScrape (instead of ServiceMonitor) - defines scraping metrics configuration from pods backed by services. See details .
  • VMPodScrape (instead of PodMonitor) - defines scraping metrics configuration from pods. See details .
  • VMRule (instead of PrometheusRule) - defines alerting or recording rules. See details .
  • VMProbe (instead of Probe) - defines a probing configuration for targets with blackbox exporter. See details .
  • VMAlertmanagerConfig (instead of AlertmanagerConfig) - defines a configuration for AlertManager. See details .
  • VMScrapeConfig (instead of ScrapeConfig) - define a scrape config using any of the service discovery options supported in victoriametrics.

Note that Prometheus CRDs are not supplied with the VictoriaMetrics operator, so you need to install them separately . VictoriaMetrics operator supports conversion from Prometheus CRD of version monitoring.coreos.com/v1 for kinds ServiceMonitor, PodMonitor, PrometheusRule, Probe and version monitoring.coreos.com/v1alpha1 for kind AlertmanagerConfig.

The default behavior of the operator is as follows:

  • It converts all existing Prometheus ServiceMonitor, PodMonitor, PrometheusRule, Probe and ScrapeConfig objects into corresponding VictoriaMetrics Operator objects.
  • It syncs updates (including labels) from Prometheus ServiceMonitor, PodMonitor, PrometheusRule, Probe and ScrapeConfig objects to corresponding VictoriaMetrics Operator objects.
  • It DOES NOT delete converted objects after original ones are deleted.

With this configuration removing prometheus-operator API objects wouldn’t delete any converted objects. So you can safely migrate or run two operators at the same time.

You can change default behavior with operator configuration - see details below .

Objects conversion #

By default, the vmoperator converts all existing prometheus-operator API objects into corresponding VictoriaMetrics Operator objects, i.e. creates resources of VictoriaMetrics similar to Prometheus resources in the same namespace.

You can control this behaviour by setting -controller.disableReconcileFor operator command-line flag with values:

  • PodMonitor - disable PodMonitor conversion
  • ServiceMonitor - disable ServiceMonitor conversion
  • PrometheusRule - disable PrometheusRule conversion
  • Probe - disable Probe conversion
  • ScrapeConfig - disable ScrapeConfig conversion

For victoria-metrics-operator helm-chart you can use following way:

      # values.yaml

# ...
operator:
  # -- By default, operator converts prometheus-operator objects.
  disable_prometheus_converter: true
# ...

Otherwise, VictoriaMetrics Operator would try to discover prometheus-operator API and convert it.

migration from prometheus

For more information about the operator’s workflow, see this doc .

Deletion synchronization #

By default, the operator doesn’t make converted objects disappear after original ones are deleted. To change this behaviour configure adding OwnerReferences to converted objects with following operator parameter :

      VM_ENABLEDPROMETHEUSCONVERTEROWNERREFERENCES=true

For victoria-metrics-operator helm-chart you can use following way:

      # values.yaml

# ...
operator:
  # -- Enables ownership reference for converted prometheus-operator objects,
  # it will remove corresponding victoria-metrics objects in case of deletion prometheus one.
  enable_converter_ownership: true
# ...

Converted objects will be linked to the original ones and will be deleted by kubernetes after the original ones are deleted.

Update synchronization #

Conversion of api objects can be controlled by annotations, added to VMObjects.

Annotation operator.victoriametrics.com/ignore-prometheus-updates controls updates from Prometheus api objects.

By default, it set to disabled. You define it to enabled state and all updates from Prometheus api objects will be ignored.

Example:

      apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
  annotations:
    meta.helm.sh/release-name: prometheus
    operator.victoriametrics.com/ignore-prometheus-updates: enabled
  labels:
    release: prometheus
  name: prometheus-monitor
spec:
  endpoints: []

Annotation operator.victoriametrics.com/ignore-prometheus-updates can be set on one of the resources:

And annotation doesn’t make sense for VMStaticScrape and VMNodeScrape because these objects are not created as a result of conversion.

Labels and annotations synchronization #

Conversion of api objects can be controlled by annotations, added to VMObjects.

Annotation operator.victoriametrics.com/merge-meta-strategy controls syncing of metadata labels and annotations between VMObjects and Prometheus api objects during updates to Prometheus objects.

By default, it has prefer-prometheus. And annotations and labels will be used from Prometheus objects, manually set values will be dropped.

You can set it to prefer-victoriametrics. In this case all labels and annotations applied to Prometheus object will be ignored and VMObject will use own values.

Two additional strategies annotations -merge-victoriametrics-priority and merge-prometheus-priority merges labelSets into one combined labelSet, with priority.

Example:

      apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
  annotations:
    meta.helm.sh/release-name: prometheus
    operator.victoriametrics.com/merge-meta-strategy: prefer-victoriametrics
  labels:
    release: prometheus
  name: prometheus-monitor
spec:
  endpoints: []

Annotation operator.victoriametrics.com/merge-meta-strategy can be set on one of the resources:

And annotation doesn’t make sense for VMStaticScrape and VMNodeScrape because these objects are not created as a result of conversion.

You can filter labels for syncing with operator parameter VM_FILTERPROMETHEUSCONVERTERLABELPREFIXES:

      # it excludes all labels that start with "helm.sh" or "argoproj.io" from synchronization
VM_FILTERPROMETHEUSCONVERTERLABELPREFIXES=helm.sh,argoproj.io

In the same way, annotations with specified prefixes can be excluded from synchronization with operator parameter VM_FILTERPROMETHEUSCONVERTERANNOTATIONPREFIXES:

      # it excludes all annotations that start with "helm.sh" or "argoproj.io" from synchronization
VM_FILTERPROMETHEUSCONVERTERANNOTATIONPREFIXES=helm.sh,argoproj.io

Using converter with ArgoCD #

If you use ArgoCD, you can allow ignoring objects at ArgoCD converted from Prometheus CRD with operator parameter VM_PROMETHEUSCONVERTERADDARGOCDIGNOREANNOTATIONS.

It helps to properly use converter with ArgoCD and should help prevent out-of-sync issues with argo-cd based deployments:

      # adds compare-options and sync-options for prometheus objects converted by operator 
VM_PROMETHEUSCONVERTERADDARGOCDIGNOREANNOTATIONS=true

Data migration #

You can use vmctl for migrating your data from Prometheus to VictoriaMetrics.

See this doc for more details.

Auto-discovery for prometheus.io annotations #

There is a scenario where auto-discovery using prometheus.io-annotations (such as prometheus.io/port, prometheus.io/scrape, prometheus.io/path, etc.) is required when migrating from Prometheus instead of manually managing scrape objects.

You can enable this feature using special scrape object like that:

Pod annotations based discovery:

      apiVersion: operator.victoriametrics.com/v1beta1
kind: VMPodScrape
metadata:
  name: annotations-discovery
spec:
  # use well-known kubernetes label for job
  # otherwise any discovered target will have annotations-discovery as a job
  jobLabel: "app.kubernetes.io/name"
  podMetricsEndpoints:
    - relabelConfigs:
        # Skip scrape for init containers
        - action: drop
          source_labels: [__meta_kubernetes_pod_container_init]
          regex: "true"
        # Match container port with port from annotation
        - action: keep_if_equal
          source_labels:
            [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number]
        # Check if scrape is enabled
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: "true"
        # Set scrape path
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
        - source_labels: [__meta_kubernetes_pod_node_name]
          action: replace
          target_label: node
  namespaceSelector: {} # You need to specify namespaceSelector here
  selector: {} # You need to specify selector here

You can find yaml-file with this example here .

Service annotations based discovery:

      apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
  name: annotations-discovery
spec:
  # use well-known kubernetes label for job
  # otherwise any discovered target will have annotations-discovery as a job
  jobLabel: "app.kubernetes.io/name"
  endpoints:
    - relabelConfigs:
        # Skip scrape for init containers
        - action: drop
          source_labels: [__meta_kubernetes_pod_container_init]
          regex: "true"
        # Match container port with port from annotation 
        - action: keep_if_equal
          source_labels: [__meta_kubernetes_service_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number]
        # Check if scrape is enabled
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
          action: keep
          regex: "true"
        # Set scrape path
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
        - source_labels: [__meta_kubernetes_pod_node_name]
          action: replace
          target_label: node
  namespaceSelector: {} # You need to specify namespaceSelector here
  selector: {} # You need to specify selector here

You can find yaml-file with this example here .

Check out more information about:

Previous Grafana Next Istio