VictoriaLogs Collector #

VictoriaLogs Collector - collects logs from Kubernetes containers and stores them to VictoriaLogs

Prerequisites #

Before installing this chart, ensure your environment meets the following requirements:

Kubernetes cluster - A running Kubernetes cluster with sufficient resources
Helm - Helm package manager installed and configured

Additional requirements depend on your configuration:

Persistent storage - Required if you enable persistent volumes for data retention (enabled by default)
kubectl - Needed for cluster management and troubleshooting

For installation instructions, refer to the official documentation:

Quick start #

This Helm chart deploys a log collection agent as a DaemonSet. It collects logs from all containers in a Kubernetes cluster and forwards them to the configured VictoriaLogs destinations. If more than one destination is specified, then the collected logs are replicated among the configured destinations.

This chart will expand its functionality as the corresponding features are added to vlagent .

To quickly install single-node version of VictoriaLogs and victoria-logs-collector, see these docs .
To start with a VictoriaLogs cluster and victoria-logs-collector, see these docs .

Chart configuration #

The simplest working configuration includes specifying the remoteWrite array and setting CPU and memory resources for the chart.

Example of a minimal working configuration:

      remoteWrite:
  - url: http://victoria-logs:9428

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi
    

If multiple remoteWrite entries are defined, logs are replicated to all the specified destinations.

See victoria-logs-single and victoria-logs-cluster documentation for details how to determine remote write URL for each installation type.

On-disk buffer #

If the remote VictoriaLogs instance is temporarily unavailable, vlagent buffers collected logs on disk and sends them as soon as the connection is restored. By default, the buffer is unlimited in size and uses the entire disk space.

To cap how much disk space the buffer can use per destination, set maxDiskUsagePerURL:

      remoteWrite:
  - url: http://victoria-logs:9428
    maxDiskUsagePerURL: 10GB
    

When the limit is reached, the oldest buffered logs are dropped to make room for new ones. See replication and high availability docs for more details.

Kubernetes metadata #

By default, vlagent attaches Pod labels to every log entry. You can control which metadata fields are included using the following settings:

      collector:
  includePodLabels: true              # default: true
  includePodAnnotations: false        # default: false
  includeNodeLabels: false            # default: false
  includeNodeAnnotations: false       # default: false
  includeNamespaceLabels: false       # default: false
  includeNamespaceAnnotations: false  # default: false
    

Note that even when a metadata field is excluded from log entries, it is still available for filtering via collector.excludeFilter. For example, you can filter by Pod annotations without storing them in every log entry:

      collector:
  includePodAnnotations: false
  excludeFilter: "kubernetes.pod_annotations.victoriametrics.com/vlagent/exclude:=true"
    

Note that vlagent does not update Node, Pod, and Namespace metadata during runtime. If Node, Pod, or Namespace labels change, you must restart vlagent to apply those changes.

Ignore fields #

VictoriaLogs efficiently compresses repeated values, such as Pod and Node labels. However, if you prefer not to store certain fields, you have two options.

Use collector.ignoreFields to drop fields on the vlagent side before sending to any destination:

      collector:
  ignoreFields:
    - request.payload*
    - password
    - token
    

Use the VL-Ignore-Fields header to drop fields for a specific destination only. In this case, the fields are still sent over the network but ignored by VictoriaLogs on ingestion:

      remoteWrite:
  - url: http://victoria-logs:9428
    headers:
      VL-Ignore-Fields:
        - kubernetes.namespace_labels*
        - kubernetes.pod_labels*
    

This allows you to exclude unnecessary or sensitive fields from being ingested. If sensitive data has already been ingested, see how to exclude logs from search result .

Extra fields #

You have two options to attach additional fields to log entries.

Use collector.extraFields to add fields on the vlagent side for all destinations:

      collector:
  extraFields: '{"env":"production","cluster":"us-east1"}'
    

Use the VL-Extra-Fields header to add fields for a specific destination only:

      remoteWrite:
  - url: http://victoria-logs:9428
    headers:
      VL-Extra-Fields:
        zone: us-east1-c
        source: victoria-logs-collector
    

This feature lets you attach metadata to every log entry, making it easier to filter, group, or analyze logs based on these additional attributes.

Filtering Kubernetes logs #

To skip logs from specific containers before reading them, use collector.excludeFilter. The filter is applied to container metadata fields before reading the log files, which significantly reduces CPU and I/O usage compared to filtering after ingestion.

The filter supports any LogsQL filter expression . See the full list of supported metadata fields in vlagent filtering docs .

Exclude vlagent’s own logs:

      collector:
  excludeFilter: "kubernetes.pod_name:=%{HOSTNAME}"
    

Exclude all logs from specific namespaces:

      collector:
  excludeFilter: "kubernetes.pod_namespace:in(kube-system, monitoring)"
    

Exclude logs from specific containers:

      collector:
  excludeFilter: "kubernetes.container_name:in(config-reloader, prometheus-config-reloader) and kubernetes.pod_namespace:=monitoring"
    

Exclude Pods by annotation - add victoriametrics.com/vlagent/exclude: "true" to a Pod and it will be skipped:

      collector:
  excludeFilter: "kubernetes.pod_annotations.victoriametrics.com/vlagent/exclude:=true"
    

Sending logs to external systems #

By default, vlagent uses the native VictoriaLogs protocol. To send logs to external systems, set format: jsonline on the corresponding remoteWrite entry.

You can mix destinations with different formats in the same configuration:

      remoteWrite:
  - url: http://victoria-logs:9428
    format: native
  - url: http://vector:8080
    format: jsonline
    

See vlagent remote write format docs for more details.

Fluent Bit #

Start Fluent Bit with an HTTP input on port 8080, then configure the collector to forward logs to it. Fluent Bit requires the Content-Type: application/json header:

      remoteWrite:
  - url: http://fluent-bit:8080
    format: jsonline
    headers:
      Content-Type: application/json
    

Vector #

Start Vector with an http_server source listening on port 8080 with codec: json, then point the collector at it:

      remoteWrite:
  - url: http://vector:8080
    format: jsonline
    

ClickHouse #

To send logs to ClickHouse, include the INSERT statement as the query HTTP parameter in the URL. Choose the FORMAT clause based on your table schema:

JSONAsObject - table with a single JSON column.
JSONAsString - table with a single String column.
JSONEachRow - table with multiple columns, each JSON field maps to a column.

      remoteWrite:
  - url: "http://clickhouse:8123?query=INSERT%20INTO%20default.logs%20FORMAT%20JSONAsObject"
    format: jsonline
    basicAuth:
      username: clickhouse-user
      passwordFile: /etc/vlagent/ch-password.txt
    

Multitenancy #

To define tenant , set ProjectID and AccountID headers as shown below:

      remoteWrite:
  - url: http://localhost:9428
    headers:
      ProjectID: 12
      AccountID: 42
    

Basic auth #

To authenticate against VictoriaLogs using basic auth, store the password in a Kubernetes Secret and reference it via passwordFile to avoid exposing credentials in values:

      remoteWrite:
  - url: http://victoria-logs:9428
    basicAuth:
      username: vlagent
      passwordFile: /etc/vlagent/password.txt

extraVolumes:
  - name: auth-secret
    secret:
      secretName: vlagent-auth

extraVolumeMounts:
  - name: auth-secret
    mountPath: /etc/vlagent
    readOnly: true
    

Alternatively, pass credentials via environment variables:

      remoteWrite:
  - url: http://victoria-logs:9428

env:
  - name: VL_remoteWrite_basicAuth_password
    valueFrom:
      secretKeyRef:
        name: auth-secret
        key: VL_PASSWORD
  - name: VL_remoteWrite_basicAuth_username
    valueFrom:
      secretKeyRef:
        name: auth-secret
        key: VL_USERNAME
    

Bearer token auth #

To authenticate using a bearer token, store the token in a Kubernetes Secret and reference it via bearerTokenFile. The token file is reloaded automatically when it changes:

      remoteWrite:
  - url: http://victoria-logs:9428
    bearerTokenFile: /etc/vlagent/token.txt

extraVolumes:
  - name: auth-token
    secret:
      secretName: vlagent-token

extraVolumeMounts:
  - name: auth-token
    mountPath: /etc/vlagent
    readOnly: true
    

TLS #

To enable TLS verification for the remoteWrite target, you can specify the tls-prefixed params inside each remoteWrite entry.

At a minimum, you should provide the tlsCAFile path so that the collector can verify the server’s TLS certificate. This is useful when the target endpoint uses a certificate signed by a custom or self-signed Certificate Authority (CA).

      remoteWrite:
  - url: https://victoria-logs:9428
    tlsCAFile: /etc/tls/ca.crt

extraVolumes:
  - name: tls-certs
    secret:
      secretName: tls-secret

extraVolumeMounts:
  - name: tls-certs
    mountPath: /etc/tls
    readOnly: true
    

For mutual TLS (mTLS), additionally specify the client certificate and key:

      remoteWrite:
  - url: https://victoria-logs:9428
    tlsCAFile: /etc/tls/ca.crt
    tlsCertFile: /etc/tls/client.crt
    tlsKeyFile: /etc/tls/client.key
    

If you want to disable TLS certificate verification (not recommended in production), you can set tlsInsecureSkipVerify to true:

      remoteWrite:
  - url: https://victoria-logs:9428
    tlsInsecureSkipVerify: true
    

How to install #

Access a Kubernetes cluster.

Setup chart repository (can be omitted for OCI repositories) #

Add a chart helm repository with follow commands:

      helm repo add vm https://victoriametrics.github.io/helm-charts/

helm repo update

List versions of vm/victoria-logs-collector chart available to installation:

      helm search repo vm/victoria-logs-collector -l

Install `victoria-logs-collector` chart #

Export default values of victoria-logs-collector chart to file values.yaml:

For HTTPS repository

      helm show values vm/victoria-logs-collector > values.yaml

For OCI repository

      helm show values oci://ghcr.io/victoriametrics/helm-charts/victoria-logs-collector > values.yaml

Change the values according to the need of the environment in values.yaml file.

Note

Consider setting .Values.nameOverride to a small value like vlc to avoid hitting resource name limits of 63 characters

Test the installation with command:

For HTTPS repository

      helm install vlc vm/victoria-logs-collector -f values.yaml -n NAMESPACE --debug

For OCI repository

      helm install vlc oci://ghcr.io/victoriametrics/helm-charts/victoria-logs-collector -f values.yaml -n NAMESPACE --debug

Install chart with command:

For HTTPS repository

      helm install vlc vm/victoria-logs-collector -f values.yaml -n NAMESPACE

For OCI repository

      helm install vlc oci://ghcr.io/victoriametrics/helm-charts/victoria-logs-collector -f values.yaml -n NAMESPACE

Get the pods lists by running this commands:

      kubectl get pods -A | grep 'vlc'

Get the application by running this command:

      helm list -f vlc -n NAMESPACE

See the history of versions of vlc application with command.

      helm history vlc -n NAMESPACE

How to uninstall #

Remove application with command.

      helm uninstall vlc -n NAMESPACE

Parameters #

The following tables lists the configurable parameters of the chart and their default values.

Change the values according to the need of the environment in victoria-logs-collector/values.yaml file.

Key	Description
`affinity: {}`	`(object)` Pod affinity.
`annotations: {}`	`(object)` Annotations to be added to the deployment.
`collector.excludeFilter: ""`	`(string)` LogsQL filter for excluding container logs. The filter is applied to container metadata fields (e.g., kubernetes.namespace_name, kubernetes.container_name) before reading the log files. This significantly reduces CPU and I/O usage by skipping logs from unwanted containers. See https://docs.victoriametrics.com/victorialogs/vlagent/#filtering-kubernetes-logs
`collector.extraFields: ""`	`(string)` Add additional fields to the log entries. If you want to add a field per remoteWrite destination, set the VL-Extra-Fields header instead.
`collector.ignoreFields: []`	`(list)` List of fields to be ignored. A field can be ended with ‘*’ to ignore all fields starting with the specified prefix. If you want to exclude Kubernetes metadata fields, use includePodLabels/includePodAnnotations/includeNodeLabels/includeNodeAnnotations instead.
`collector.includeNodeAnnotations: false`	`(bool)` Include Node annotations as additional fields in the log entries. Even this setting is disabled, Node annotations are available for filtering via .Values.excludeFilter parameter.
`collector.includeNodeLabels: false`	`(bool)` Include Node labels as additional fields in the log entries. Even this setting is disabled, Node labels are available for filtering via .Values.excludeFilter parameter.
`collector.includePodAnnotations: false`	`(bool)` Include Pod annotations as additional fields in the log entries. Even this setting is disabled, Pod annotations are available for filtering via .Values.excludeFilter parameter.
`collector.includePodLabels: true`	`(bool)` Include Pod labels as additional fields in the log entries. Even this setting is disabled, Pod labels are available for filtering via .Values.excludeFilter parameter.
`collector.msgField: - message - msg - log`	`(list)` List of fields to be used as `_msg` field. The first found field will be used. See https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field for more details. Add more fields if you see the ‘missing _msg field’ message in VictoriaLogs. If you set an empty list, then the default message fields will be used.
`collector.streamFields: - kubernetes.container_name - kubernetes.pod_name - kubernetes.pod_namespace`	`(list)` List of fields to be used as `_stream` field. See https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields for more details. It is recommended to leave this with default values. If you set an empty list, then the default stream fields will be used.
`collector.timeField: - time - ts - timestamp`	`(list)` List of fields to be used as `_time` field. The first found field will be used. See https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field for more details. If you set an empty list, then the default time fields will be used.
`defaultVolumeMounts: - mountPath: /var/log name: varlog readOnly: true - mountPath: /var/lib name: varlib readOnly: true`	`(list)` Mounts for defaultVolumes.
`defaultVolumes: - hostPath: path: /var/log name: varlog - hostPath: path: /var/lib name: varlib`	`(list)` Default volumes that are mounted into Pods. In most cases, these should not be changed. Use `extraVolumes`/`extraVolumeMounts` for additional custom volumes.
`env: []`	`(list)` Environment variables (ex.: secret tokens).
`extraArgs: envflag.enable: true envflag.prefix: VL_ loggerFormat: json tmpDataPath: /var/lib/vl-collector`	`(object)` vlagent extra command line arguments.
`extraVolumeMounts: []`	`(list)` Extra Volume Mounts for the container.
`extraVolumes: []`	`(list)` Extra Volumes for the Pod.
`fullnameOverride: ""`	`(string)` Override resources fullname.
`global.cluster.dnsDomain: cluster.local.`	`(string)` K8s cluster domain suffix, used for building storage Pods’ FQDN. Details are here.
`global.compatibility: openshift: adaptSecurityContext: auto`	`(object)` Openshift security context compatibility configuration
`global.image.registry: ""`	`(string)` Image registry, that can be shared across multiple helm charts
`global.imagePullSecrets: []`	`(list)` Image pull secrets, that can be shared across multiple helm charts
`image.pullPolicy: IfNotPresent`	`(string)` Image pull policy.
`image.registry: ""`	`(string)` Image registry.
`image.repository: victoriametrics/vlagent`	`(string)` Image repository.
`image.tag: ""`	`(string)` Image tag.
`image.variant: ""`	`(string)` Image tag suffix, which is appended to `Chart.AppVersion` if no `server.image.tag` is defined.
`license: key: "" secret: key: "" name: ""`	`(object)` Enterprise license key configuration for VictoriaLogs enterprise. Required only for VictoriaLogs enterprise. Check docs here, Request a trial license here.
`license.key: ""`	`(string)` License key.
`license.secret: key: "" name: ""`	`(object)` Use existing secret with license key.
`license.secret.key: ""`	`(string)` Key in secret with license key.
`license.secret.name: ""`	`(string)` Existing secret name.
`nameOverride: ""`	`(string)` Override chart name.
`nodeSelector: {}`	`(object)` Pod’s node selector. Details are here.
`persistence.volume: {}`	`(object)`
`podAnnotations: {}`	`(object)` Annotations to be added to Pod.
`podLabels: {}`	`(object)` Extra labels for Pods only.
`podMonitor.annotations: {}`	`(object)` PodMonitor annotations
`podMonitor.enabled: false`	`(bool)` Enable PodMonitor
`podMonitor.extraLabels: {}`	`(object)` PodMonitor labels
`podMonitor.vm: false`	`(bool)` Whether to use VMPodScrape from VM operator instead of PodMonitor.
`podSecurityContext: enabled: true`	`(object)` Security context to be added to Pod.
`priorityClassName: ""`	`(string)` Priority class to be assigned to the Pod(s).
`remoteWrite: []`	`(list)` List of log destinations. Logs will be replicated to all listed destinations. If the url path is not specified, the logs will be sent to the /insert/native endpoint.
`resources: null`	`(string)`
`securityContext: enabled: true`	`(object)` Security context to be added to Pod’s containers.
`serviceAccount: annotations: {} automount: true name: ""`	`(object)` Service account is needed to enrich logs with Pod metadata using Kubernetes API.
`tolerations: []`	`(list)` Node tolerations for server scheduling to nodes with taints. Details are here.
`topologySpreadConstraints: null`	`(string)` Pod topologySpreadConstraints.
`updateStrategy: {}`	`(object)` DaemonSet update strategy.

VictoriaLogs Collector #

Prerequisites #

Quick start #

Chart configuration #

On-disk buffer #

Kubernetes metadata #

Ignore fields #

Extra fields #

Filtering Kubernetes logs #

Sending logs to external systems #

Fluent Bit #

Vector #

ClickHouse #

Multitenancy #

Basic auth #

Bearer token auth #

TLS #

How to install #

Setup chart repository (can be omitted for OCI repositories) #

Install victoria-logs-collector chart #

How to uninstall #

Parameters #

Install `victoria-logs-collector` chart #