VMProbe

The VMProbe CRD provides probing target ability with some external prober. The most common prober is blackbox exporter. By specifying configuration at CRD, operator generates config for VMAgent and syncs it. It’s possible to use static targets or use standard k8s discovery mechanism with Ingress.

VMProbe object generates part of VMAgent configuration; It has various options for scraping configuration of target (with basic auth, tls access, by specific port name etc.).

You have to configure blackbox exporter before you can use this feature. The second requirement is VMAgent selectors, it must match your VMProbe by label or namespace selector. VMAgent probeSelector must match VMProbe labels.

See more details about selectors here.

Specification #

You can see the full actual specification of the VMProbe resource in the API docs -> VMProbe.

Also, you can check out the examples section.

Migration from Prometheus #

The VMProbe CRD from VictoriaMetrics Operator is a drop-in replacement for the Prometheus Probe from prometheus-operator.

More details about migration from prometheus-operator you can read in this doc.

Examples #

Static targets #

It will probe VMAgent with url - vmagent-example-vmagent.default.svc:9115/health with blackbox url: prometheus-blackbox-exporter.default.svc:9115 and module http_2xx (it was specified at blackbox configmap).

              1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
                          apiVersion: operator.victoriametrics.com/v1beta1
kind: VMProbe
metadata:
  name: vmprobe-static-example
spec:
  jobName: static-probe
  vmProberSpec:
     # by default scheme http, and path is /probe
     url: prometheus-blackbox-exporter.default.svc:9115
  module: http_2xx
  targets:
   staticConfig: 
      targets:
      -  vmagent-example-vmagent.default.svc:8429/health
  interval: 2s 
            

After adding target to VMAgent configuration it starts probing itself throw blackbox exporter.

Ingress targets #

              1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
                          apiVersion: operator.victoriametrics.com/v1beta1
kind: VMProbe
metadata:
  name: vmprobe-ingress-example
spec:
  vmProberSpec:
     # by default scheme http, and path is /probe
     url: prometheus-blackbox-exporter.default.svc:9115
  module: http_2xx
  targets:
   ingress: 
      selector:
       matchLabels:
        app: victoria-metrics-single
  interval: 10s
            

This configuration will add 2 additional targets for probing: vmsingle2.example.com and vmsingle.example.com.

But probes will be unsuccessful, because there is no such hosts.

Following resources will be used for the examples below:

Blackbox exporter #

              1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
                          apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-blackbox-exporter
  labels:
    app: prometheus-blackbox-exporter
data:
  blackbox.yaml: |
    modules:
      http_2xx:
        http:
          preferred_ip_protocol: ip4
          valid_http_versions:
          - HTTP/1.1
          - HTTP/2.0
          valid_status_codes: []
        prober: http
        timeout: 5s

---

kind: Service
apiVersion: v1
metadata:
  name: prometheus-blackbox-exporter
  labels:
    app: prometheus-blackbox-exporter
spec:
  type: ClusterIP
  ports:
    - name: http
      port: 9115
      protocol: TCP
  selector:
    app: prometheus-blackbox-exporter

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-blackbox-exporter
  labels:
    app: prometheus-blackbox-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-blackbox-exporter
  template:
    metadata:
      labels:
        app: prometheus-blackbox-exporter
    spec:
      containers:
        - name: blackbox-exporter
          image: "prom/blackbox-exporter:v0.17.0"
          args:
            - "--config.file=/config/blackbox.yaml"
          resources:
            {}
          ports:
            - containerPort: 9115
              name: http
          livenessProbe:
            httpGet:
              path: /health
              port: http
          readinessProbe:
            httpGet:
              path: /health
              port: http
          volumeMounts:
            - mountPath: /config
              name: config
      volumes:
        - name: config
          configMap:
            name: prometheus-blackbox-exporter
            

VMSingle #

              1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
                          apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
  name: example-vmsingle-persisted
spec:
  retentionPeriod: "1"
  removePvcAfterDelete: true
  storage:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 1Gi 

---

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  labels:
    app: victoria-metrics-single
  name: victoria-metrics-single
spec:
  rules:
    - host: vmsingle.example.com
      http:
        paths:
          - backend:
              serviceName: vmsingle-example-vmsingle-persisted
              servicePort: 8428
            path: /
    - host: vmsingle2.example.com
      http:
        paths:
          - backend:
              serviceName: vmsingle-example-vmsingle-persisted
              servicePort: 8428
            path: /
            

VMAgent #

              1
2
3
4
5
6
7
8
9
                          apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
   name: example-vmagent
spec:
   selectAllByDefault: true
   replicaCount: 1
   remoteWrite:
     - url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
            

VMProbe #