VLCluster #

VLCluster represents database for storing logs Available from v0.59.0 . The VLCluster CRD declaratively defines a VictoriaLogs cluster installation to run in a Kubernetes cluster.

For each VLCluster resource, the Operator deploys a properly configured components VLInsert’s Deployment, VLSelect’s Deployment and VLStorage’s StatefulSet in the same namespace. The VLStorage Pods are configured to mount an empty dir or StorageSpec for storing data.

For each VLCluster resource components, the Operator adds Service and VMServiceScrape in the same namespace prefixed with name from VLCluster.metadata.name.

Specification #

You can see the full actual specification of the VLCluster resource in the API docs -> VLCluster .

If you can’t find necessary field in the specification of the custom resource, see Extra arguments section .

Also, you can check out the examples section.

Services and URLs #

For a VLCluster named <name> in namespace <namespace>, the Operator creates the following Kubernetes services:

Common internal cluster URLs (replace <name> and <namespace> with your values):

vlinsert also accepts logs from all data ingestion protocols supported by VictoriaLogs (Elasticsearch, Loki, OpenTelemetry, Syslog, etc.) directly at port 9481.

Configuring VLAgent to write to VLCluster #

      apiVersion: operator.victoriametrics.com/v1
kind: VLAgent
metadata:
  name: example
  namespace: default
spec:
  remoteWrite:
    - url: "http://vlinsert-<name>.default.svc:9481/internal/insert"
    

Customizing service type or port #

By default each component service is created with the type and port shown in the table above. Use serviceSpec on each component to change these settings. Setting useAsDefault: true applies the changes to the main service rather than creating an additional one:

      apiVersion: operator.victoriametrics.com/v1
kind: VLCluster
metadata:
  name: example
spec:
  vlinsert:
    replicaCount: 2
    serviceSpec:
      useAsDefault: true
      spec:
        type: LoadBalancer
  vlselect:
    replicaCount: 2
    serviceSpec:
      useAsDefault: true
      spec:
        type: LoadBalancer
    

To expose VLCluster components outside the cluster via an ingress with authentication, see Authorization and exposing components — VLCluster .

Version management #

To set VLCluster version add spec.clusterVersion or spec.COMPONENT.image.tag name from releases

      apiVersion: operator.victoriametrics.com/v1
kind: VLCluster
metadata:
  name: example
spec:
  ...
  clusterVersion: v1.26.0
  vlstorage:
    image:
      repository: victoriametrics/victoria-logs
      tag: v1.25.0
      pullPolicy: Always
  # ...
    

Also, you can specify imagePullSecrets if you are pulling images from private repo:

      apiVersion: operator.victoriametrics.com/v1
kind: VLCluster
metadata:
  name: example
spec:
  imagePullSecrets:
    - name: my-repo-secret
# ...
    

Update strategy #

The Operator provides fine-grained control over how VLCluster nodes are restarted when their spec changes (image tag, flags, etc.).

vlstorage #

vlstorage nodes are restarted one by one by default. The rollingUpdateStrategy field controls this behavior:

Use rollingUpdateStrategyBehavior.maxUnavailable to control how many nodes are restarted simultaneously (default is 1):

      apiVersion: operator.victoriametrics.com/v1
kind: VLCluster
metadata:
  name: example
spec:
  vlstorage:
    replicaCount: 6
    rollingUpdateStrategy: OnDelete
    rollingUpdateStrategyBehavior:
      maxUnavailable: 2        # restart 2 nodes at a time
    

Setting maxUnavailable: "100%" restarts all vlstorage nodes in parallel, minimizing total upgrade time at the cost of temporary unavailability.

The timeout for each node to become ready is controlled by the Operator environment variable VM_PODWAITREADYTIMEOUT (default 80s). The poll interval is VM_PODWAITREADYINTERVALCHECK (default 5s).

vlinsert and vlselect #

vlinsert and vlselect nodes support standard Deployment update parameters:

      apiVersion: operator.victoriametrics.com/v1
kind: VLCluster
metadata:
  name: example
spec:
  vlinsert:
    replicaCount: 2
    updateStrategy: RollingUpdate   # default; set to Recreate to stop all nodes before starting new ones
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  vlselect:
    replicaCount: 2
    updateStrategy: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
    

Storage node maintenance mode #

vlstorage.maintenanceInsertNodeIDs and vlstorage.maintenanceSelectNodeIDs exclude the listed vlstorage pods from vlinsert and vlselect routing. Each ID is a pod ordinal, the numeric suffix of the pod name, e.g. 2 for the pod ending with -2. Use them when a vlstorage pod is expected to be unavailable for a while, for example when its Kubernetes node is under maintenance and the pod can’t move, since its volume is bound to that node.

Excluding a pod from routing affects the cluster:

• with the pod in maintenanceSelectNodeIDs, logs stored on this pod are missing from query responses, and vlselect doesn’t mark them as partial, since it doesn’t know about the pod;
• with the pod in maintenanceInsertNodeIDs, vlinsert spreads incoming logs across the remaining pods, so they take over the excluded pod’s share of writes.

Pods excluded from all routing #

A pod listed in both vlstorage.maintenanceInsertNodeIDs and vlstorage.maintenanceSelectNodeIDs fields receives no reads and no writes. Starting from v0.75.1 , with the default rollingUpdateStrategy: OnDelete, where the Operator updates vlstorage pods itself, the Operator leaves such a pod alone during reconciliation:

• it doesn’t wait for the pod to become ready, so a pod stuck Pending or Terminating no longer blocks reconciliation of vlinsert and vlselect;
• it doesn’t apply vlstorage spec changes to the pod or its volumes, e.g. a new image or a bigger volume size.

      apiVersion: operator.victoriametrics.com/v1
kind: VLCluster
metadata:
  name: example
spec:
  vlstorage:
    replicaCount: 3
    maintenanceInsertNodeIDs: [2]
    maintenanceSelectNodeIDs: [2]   # the pod with ordinal 2 gets no traffic and is left alone by the Operator
    

When the pod is ready to get back, remove the ID from both fields before making other changes. The Operator then applies the vlstorage changes deferred during maintenance to the pod, and waits for it to become ready before adding it back to vlinsert and vlselect, so an unhealthy pod never receives traffic.

Limitations:

• The pod is left alone only with the default rollingUpdateStrategy: OnDelete and without minReadySeconds, where the StatefulSet uses Parallel pod management. Setting rollingUpdateStrategy: RollingUpdate or minReadySeconds switches the StatefulSet to OrderedReady pod management, where Kubernetes handles pods one by one in ordinal order: while the pod under maintenance isn’t ready, it doesn’t roll out updates, scale the StatefulSet or recreate pods with a higher ordinal. So reconciliation that changes vlstorage pods can block until the pod is ready.
• With vlstorage.podDisruptionBudget.maxUnavailable: 1, a pod under maintenance that isn’t ready uses up the disruption budget, so the Operator can’t evict other vlstorage pods, and changes that require restarting them fail until the pod is ready again.

Resource management #

You can specify resources for each VLCluster resource components in the spec section of the VLCluster CRD.

      apiVersion: operator.victoriametrics.com/v1
kind: VLCluster
metadata:
  name: resources
spec:
    # ...
    vlstorage:
      resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"
    # ...
    

If these parameters are not specified, then, by default all VLCluster pods have resource requests and limits from the default values of the following operator parameters :

• VM_VLCLUSTERDEFAULT_VLSTORAGEDEFAULT_RESOURCE_LIMIT_MEM - default memory limit for VLCluster.vlstorage pods,
• VM_VLCLUSTERDEFAULT_VLSTORAGEDEFAULT_RESOURCE_LIMIT_CPU - default cpu limit for VLCluster.vlstorage pods,
• VM_VLCLUSTERDEFAULT_VLSTORAGEDEFAULT_RESOURCE_REQUEST_MEM - default memory limit for VLCluster.vlstorage pods,
• VM_VLCLUSTERDEFAULT_VLSTORAGEDEFAULT_RESOURCE_REQUEST_CPU - default cpu limit for VLCluster.vlstorage pods.

These default parameters will be used if:

• VM_VLCLUSTERDEFAULT_USEDEFAULTRESOURCES is set to true (default value),
• VLCluster CR doesn’t have resources field in spec section for component.

Field resources in VLSingle spec have higher priority than operator parameters.

If you set VM_VLCLUSTEREDEFAULT_USEDEFAULTRESOURCES to false and don’t specify resources in VLSingle CRD, then VLSingle pods will be created without resource requests and limits.

Also, you can specify requests without limits - in this case default values for limits will not be used.

Requests Load-Balancing #

Operator provides enhanced load-balancing mechanism for vlinsert and vlselect clients. By default, operator uses built-in Kubernetes service with clusterIP type for clients connection. It’s good solution for short lived connections. But it acts poorly with long-lived TCP sessions and leads to the uneven resources utilization for vlselect and vlinsert components.

Operator allows to tweak Kubernetes TCP-based load-balancing with enabled requestsLoadBalancer :

      apiVersion: operator.victoriametrics.com/v1
kind: VLCluster
metadata:
  name: with-balancer
spec:
  vlinsert:
    replicaCount: 2
  vlselect:
    replicaCount: 2
  vlstorage:
    retentionPeriod: "1y"
    replicaCount: 2
  requestsLoadBalancer:
    enabled: true
    spec:
     replicaCount: 2
    

Operator will deploy VMAuth deployment with 2 replicas. And update vlinsert and vlselect services to point to vmauth. In addition, operator will create 3 additional services with the following pattern:

• vlinsertint-CLUSTER_NAME - needed for vlselect pod discovery
• vlselectint-CLUSTER_NAME - needed for vlinsert pod discovery
• vlclusterlb-CLUSTER_NAME - needed for metrics collection and exposing vlselect and vlinsert components via VMAuth balancer.

The requestsLoadBalancer feature works transparently and is managed entirely by the VLCluster operator, with no direct access to the underlying VMAuth configuration. If you need more control over load balancing behavior, or want to combine request routing with authentication or (m)TLS, consider deploying a standalone VMAuth resource instead of enabling requestsLoadBalancer.

Syslog ingestion #

vlinsert can accept logs over syslog via TCP and/or UDP listeners, set via spec.vlinsert.syslogSpec.tcpListeners and spec.vlinsert.syslogSpec.udpListeners:

      apiVersion: operator.victoriametrics.com/v1
kind: VLCluster
metadata:
  name: example
spec:
  vlinsert:
    syslogSpec:
      tcpListeners:
        - listenPort: 8094
          tenantID: "0:0"
      udpListeners:
        - listenPort: 8095
          tenantID: "0:0"
    

TLS is supported for TCP listeners only. Set tlsConfig with a certificate and key, either referencing a Secret or a file already mounted into the pod; minVersion and cipherSuites further restrict which TLS versions and cipher suites clients may use:

      apiVersion: operator.victoriametrics.com/v1
kind: VLCluster
metadata:
  name: example
spec:
  vlinsert:
    syslogSpec:
      tcpListeners:
        - listenPort: 8094
          tenantID: "0:0"
          tlsConfig:
            certSecret:
              name: vlinsert-syslog-tls
              key: tls.crt
            keySecret:
              name: vlinsert-syslog-tls
              key: tls.key
            minVersion: TLS12
            cipherSuites:
              - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    

Examples #

      
apiVersion: operator.victoriametrics.com/v1
kind: VLCluster
metadata:
  name: sample
spec:
  vlinsert:
    replicaCount: 1
  vlselect:
    replicaCount: 1
  vlstorage:
    retentionPeriod: "1y"
    replicaCount: 2