Security and Load Balancing

This document describes how to configure and use vmauth and VictoriaLogs components in the context of load balancing, access protection and log visibility management.

To configure secure communication between components in VictoriaLogs cluster mode, see Security on Untrusted Networks .

vmauth is an HTTP proxy that provides the following features:

Load balancing across configured HTTP backends.
Authentication via Basic Auth, Bearer tokens, or mTLS.
Access control to specific endpoints or data paths.
Easy to configure.

vmauth is not specifically aware of VictoriaLogs and does not offer any hidden features for tighter integration with other VictoriaMetrics components. Therefore, you can use any other HTTP proxy such as Nginx, Traefik, Envoy or HAProxy.

However, using vmauth makes it easy to configure authorization and receive community support or enterprise support from the VictoriaMetrics team if any issues arise.

For more detailed information and advanced vmauth configuration, see vmauth documentation .

All configuration examples in this documentation apply to VictoriaLogs single-node , vlselect , vlinsert and vlagent since they have the same search/write API.

Search Authorization #

For log search, both VictoriaLogs single-node and vlselect expose the same search API endpoints, which start with the /select/ prefix . When configuring request authorization or load balancing, it is important to allow access to this path prefix.

Below is an example of a vmauth configuration that:

Uses Basic auth for request authentication.
Authorizes access to paths starting with /select/.
Distributes requests between two VictoriaLogs instances: victoria-logs-1 and victoria-logs-2.

      users:
- username: foo
  password: bar
  url_map:
  - src_paths: ["/select/.*"]
    url_prefix:
    - http://victoria-logs-1:9428
    - http://victoria-logs-2:9428
    

victoria-logs-1 and victoria-logs-2 can be either two VictoriaLogs single-node instances with replicated data according to these docs , or vlselect instances in the VictoriaLogs cluster . Enumerate all the vlselect instances in the cluster under the url_prefix config above in order to spread load among all the available vlselect instances.

The diagram below illustrates this architecture in the clustered version of VictoriaLogs:

Update the connection settings in all clients (like Grafana) after configuring the vmauth in order to match the selected authentication method and the vmauth endpoint.

Important: Requests sent directly to VictoriaLogs bypass vmauth and are not authorized. To ensure security, it is strongly recommended to restrict network access to VictoriaLogs and prevent direct access from unauthorized clients.

It is recommended to pass the -insert.disable command-line flag to dedicated vlselect nodes to disable the write API. This disables both /insert/* and /internal/insert endpoints and helps protect against accidental data ingestion via vlselect in case of improperly configured log shippers.

For configuration examples using Bearer tokens, Basic auth, and mTLS, see vmauth/Authorization .

Cluster routing #

vmauth allows selecting different clusters depending on the request path. For example:

      unauthorized_user:
  url_map:
  - src_paths: ["/cold/select/.*"]
    url_prefix: http://victoria-logs-cold:9428
    # drop /cold/ path prefix from the original request before proxying it to url_prefix.
    drop_src_path_prefix_parts: 1

  - src_paths: ["/hot/select/.*"]
    url_prefix: http://victoria-logs-hot:9428
    # drop /hot/ path prefix from the original request before proxying it to url_prefix.
    drop_src_path_prefix_parts: 1
    

The configuration above enables proxying all requests that start with the path prefix /cold/select/ to the backend at http://victoria-logs-cold:9428, and requests with the path prefix /hot/select/ to the backend located at http://victoria-logs-hot:9428.

This approach is useful when applying different retention policies for various types of logs. For example, you might store warn-level and higher severity logs in the cold instance/cluster with longer retention, while keeping debug-level and higher severity logs only in the hot instance/cluster with shorter retention.

The drop_src_path_prefix_parts parameter is used to remove the prefix from the path when proxying the request to VictoriaLogs. For example, if vmauth receives a request to /cold/select/logsql/query, VictoriaLogs will receive the path without the /cold/ prefix, allowing it to properly handle the search query.

Tenant-based request proxying #

The following vmauth config proxies /select/* requests with the AccountID: 0 HTTP header to the long-term VictoriaLogs instance or cluster, while requests with the AccountID: 1 HTTP header are proxied to the short-term VictoriaLogs instance or cluster.

      unauthorized_user:
  url_map:

  # Proxy requests for the given tenant (AccountID=0) to long-term VictoriaLogs
  # and override the ProjectID with 0.
  - src_paths: ["/select/.*"]
    src_headers:
    - "AccountID: 0"
    url_prefix: "http://victoria-logs-longterm:9428"
    headers:
    - "ProjectID: 0"

  # Proxy requests for the given tenant (AccountID=1) to short-term VictoriaLogs
  # and override the AccountID with 0.
  - src_paths: ["/select/.*"]
    src_headers:
    - "AccountID: 1"
    url_prefix: "http://victoria-logs-shortterm:9428"
    headers:
    - "AccountID: 0"
    

This allows building a VictoriaLogs storage system with distinct per-tenant retention configs similar to this one .

Proxying requests to the given tenants #

To properly separate and access data across tenants, two headers must be set when writing logs: AccountID and ProjectID according to these docs . When querying logs, you must provide the same headers to retrieve the corresponding data.

You can use vmauth to enforce tenant-level access control by automatically setting the required headers after successful authentication. For example, the vmauth configuration below overrides tenant headers before proxying requests to VictoriaLogs for user foo upon successful authentication:

      users:
- username: foo
  password: bar
  url_map:
  - src_paths: ["/select/.*"]
    url_prefix: "http://victoria-logs:9428"
    headers:
    - "AccountID: 2"
    - "ProjectID: 4"
    

If the user sets the AccountID or ProjectID headers themselves, for example through vmui or Grafana data source settings, they will be overridden.

A more practical example: if you have many tenants and want to separate them by name, vmauth configuration might look like this:

      users:
- username: foo
  password: bar
  url_map:
  - src_paths: ["/my-account/kubernetes-logs/select/.*"]
    url_prefix: "http://victoria-logs:9428"
    headers:
    - "AccountID: 1"
    - "ProjectID: 5"
    # drop /my-account/kubernetes-logs/ path prefix from the original request before proxying it to url_prefix.
    drop_src_path_prefix_parts: 2

  - src_paths: ["/my-account/mobile-logs/select/.*"]
    url_prefix: "http://victoria-logs:9428"
    headers:
    - "AccountID: 1"
    - "ProjectID: 6"
    # drop /my-account/mobile-logs/ path prefix from the original request before proxying it to url_prefix.
    drop_src_path_prefix_parts: 2

  - src_paths: ["/my-account/frontend-logs/select/.*"]
    url_prefix: "http://victoria-logs:9428"
    headers:
    - "AccountID: 1"
    - "ProjectID: 7"
    # drop /my-account/frontend-logs/ path prefix from the original request before proxying it to url_prefix.
    drop_src_path_prefix_parts: 2

- username: admin
  password: secret
  url_map:
  - src_paths: ["/select/.*"]
    url_prefix: "http://victoria-logs:9428"
    

This configuration allows user foo to access 3 different tenants, and user admin to access all tenants. However, user admin needs to set the required AccountID or ProjectID headers themselves, because vmauth will not set them.

In Grafana, you need to create a separate data source for each tenant and user, an example of such an address is: http://vmauth:8427/my-account/mobile-logs. Using the configuration above, you do not need to set the tenant in the data source settings because vmauth will set it. Each tenant will have vmui at the address /select/vmui/, for example: http://vmauth:8427/my-account/mobile-logs/select/vmui/.

If you want to restrict users by only one of the fields AccountID or ProjectID, it is enough to not specify the corresponding field in the headers section. For example, the following configuration allows user my-account-admin to have access to all ProjectIDs, but only for one AccountID:

      users:
- username: my-account-admin
  password: foobar
  url_map:
  - src_paths: ["/my-account/select/.*"]
    url_prefix: "http://victoria-logs:9428"
    headers:
    - "AccountID: 1"
    # drop /my-account/ path prefix from the original request before proxying it to url_prefix.
    drop_src_path_prefix_parts: 1
    

To allow unauthenticated access to a specific tenant, define the unauthorized_user as shown below:

      unauthorized_user:
  url_map:
  - src_paths: ["/my-account/frontend-logs/select/.*"]
    url_prefix: "http://victoria-logs:9428"
    headers:
    - "AccountID: 1"
    - "ProjectID: 7"
    # drop /my-account/frontend-logs/ path prefix from the original request before proxying it to url_prefix.
    drop_src_path_prefix_parts: 2
    

This will grant all users access to logs for the specified tenant without additional authentication.

Note that if you don’t specify the AccountID or ProjectID header, VictoriaLogs will assume that the corresponding header has a value of 0.

Access control inside a single tenant #

VictoriaLogs can apply extra filters for each request to the select APIs according to these docs . This is useful when you need to give access to a subset of data within a single tenant. If you want to hide a subset of data within a tenant, use the HTTP query parameter extra_filters.

extra_filters are enforced globally - they are propagated into all the subqueries inside the provided query. This makes it impossible to bypass the restrictions via join, union, in(<query>) and other subqueries.

Consider the example below:

      users:
- username: foo
  password: bar
  url_map:
  - src_paths: ["/select/.*"]
    url_prefix:
    - http://victoria-logs-1:9428?extra_filters=password:''
    - http://victoria-logs-2:9428?extra_filters=password:''
    

With this configuration, vmauth will add the empty filter password:'' to each request, which means that the password field must be empty or missing in the log. This is useful in cases when sensitive information has leaked and needs to be hidden.

To restrict log visibility within a specific log stream , use the extra_stream_filters query parameter. The configuration below adds an additional stream filter to each request based on the request path, and routes /audit-logs to a separate VictoriaLogs instance:

      users:
- username: frontend-logs-viewer
  password: secret
  url_map:
  - src_paths: ["/frontend-logs/select/.*"]
    url_prefix: http://victoria-logs:9428?extra_stream_filters=_stream%3A%7Bservice%3Dfrontend-logs%7D
    # drop /frontend-logs/ path prefix from the original request before proxying it to url_prefix.
    drop_src_path_prefix_parts: 1

- username: mobile-logs-viewer
  password: secret
  url_map:
  - src_paths: ["/mobile-logs/select/.*"]
    url_prefix: http://victoria-logs:9428?extra_stream_filters=_stream%3A%7Bservice%3Dmobile-logs%7D
    # drop /mobile-logs/ path prefix from the original request before proxying it to url_prefix.
    drop_src_path_prefix_parts: 1

- username: audit-logs-viewer
  password: secret
  url_map:
  - src_paths: ["/audit-logs/select/.*"]
    url_prefix: http://victoria-logs-audit:9428
    # drop /audit-logs/ path prefix from the original request before proxying it to url_prefix.
    drop_src_path_prefix_parts: 1
    

extra_filters and extra_stream_filters should be percent-encoded when they include characters that are not URL-safe. For example, the query _stream:{service=frontend-logs} should be written as _stream%3A%7Bservice%3Dfrontend-logs%7D.

Prefer using extra_stream_filters over extra_filters whenever possible. This can improve search query performance because VictoriaLogs processes searches using stream filters faster than regular filters. See LogsQL performance optimization tips .

Write Authorization #

For log writing, VictoriaLogs single-node and vlinsert expose the same write API endpoints which start with the /insert/ prefix , so when configuring write requests, it is important to set this path for request authorization.

Example vmauth configuration that allows insert requests with Basic auth and distributes load between vlinsert nodes in the cluster:

      users:
- username: foo
  password: bar
  url_map:
  - src_paths: ["/insert/.*"]
    url_prefix:
    - "http://vlinsert-1:9428"
    - "http://vlinsert-2:9428"
    - "http://vlinsert-3:9428"
    

Note that vmauth cannot replicate data to multiple destinations - it spreads incoming requests among the configured backends. Use vlagent for replicating the data to multiple VictoriaLogs instances or multiple VictoriaLogs clusters.

If you trust the writing side, for example when collecting logs within your own cluster, you can set the query parameters and tenant headers directly on the log shipper side without an extra proxy. In such cases, it’s usually sufficient to restrict network access so only trusted agents can write to VictoriaLogs.

On the other hand, if you do not trust the writing side, for example, if the logs come from frontend or mobile apps, it is very important to secure the write API:

Set up a secure HTTPS connection for vmauth (see these docs ) and VictoriaLogs (see Security on Untrusted Networks ).
Protect vmauth with anti-DDoS services if needed.
Consider the max_concurrent_requests parameter to control the number of concurrent write requests.
Add monitoring and alerting for vmauth to control the load.
Write logs from untrusted applications to dedicated VictoriaLogs instances or clusters so that the unpredictable write load does not affect other instances.

It is recommended to pass the -select.disable command-line flag to dedicated vlinsert nodes in order to disable the search API. This disables both /select/* and /internal/select/* endpoints and secures access to the stored logs in case an attacker has direct network access to vlinsert.

For configuration examples using Bearer tokens, Basic auth, and mTLS, see these docs .

Tenant assignment #

vmauth allows redirecting requests to different tenants based on the request path. Example vmauth configuration that allows insert requests with Basic auth and distributes load between the configured vlinsert nodes for three different tenants :

      users:
- username: kubernetes
  password: secret
  url_map:
  - src_paths: ["/my-account/kubernetes-logs/insert/.*"]
    url_prefix:
    - "http://vlinsert-1:9428"
    - "http://vlinsert-2:9428"
    headers:
    - "AccountID: 1"
    - "ProjectID: 5"
    # drop /my-account/kubernetes-logs/ path prefix from the original request before proxying it to url_prefix.
    drop_src_path_prefix_parts: 2
  max_concurrent_requests: 10
- username: mobile
  password: secret
  url_map:
  - src_paths: ["/my-account/mobile-logs/insert/.*"]
    url_prefix:
    - "http://vlinsert-1:9428"
    - "http://vlinsert-2:9428"
    headers:
    - "AccountID: 1"
    - "ProjectID: 6"
    # drop /my-account/mobile-logs/ path prefix from the original request before proxying it to url_prefix.
    drop_src_path_prefix_parts: 2
  max_concurrent_requests: 10
- username: frontend
  password: secret
  url_map:
  - src_paths: ["/my-account/frontend-logs/insert/.*"]
    url_prefix:
    - "http://vlinsert-1:9428"
    - "http://vlinsert-2:9428"
    headers:
    - "AccountID: 1"
    - "ProjectID: 7"
    # drop /my-account/frontend-logs/ path prefix from the original request before proxying it to url_prefix.
    drop_src_path_prefix_parts: 2
  max_concurrent_requests: 10
    

Below is a diagram of this architecture for the clustered version of VictoriaLogs:

Tenant-based proxying of data ingestion requests #

The following vmauth config proxies data ingestion requests with the AccountID: 0 HTTP header to the long-term VictoriaLogs instance or cluster, while data ingestion requests with the AccountID: 1 HTTP header are proxied to the short-term VictoriaLogs instance or cluster:

      unauthorized_user:
  url_map:

  # Proxy data ingestion requests for the given tenant (AccountID=0) to long-term VictoriaLogs
  # and override the ProjectID with 0.
  - src_paths: ["/insert/.*"]
    src_headers:
    - "AccountID: 0"
    url_prefix: "http://victoria-logs-longterm:9428"
    headers:
    - "ProjectID: 0"

  # Proxy data ingestion requests for the given tenant (AccountID=1) to short-term VictoriaLogs
  # and override the AccountID with 0.
  - src_paths: ["/insert/.*"]
    src_headers:
    - "AccountID: 1"
    url_prefix: "http://victoria-logs-shortterm:9428"
    headers:
    - "AccountID: 0"
    

This allows building a VictoriaLogs storage system with distinct per-tenant retention configs similar to this one .

Adding extra fields #

You can use the extra_fields parameter in vmauth to automatically add fields to incoming log entries according to these docs . This is helpful when the writing side cannot include certain metadata, such as the source service name.

The example below adds a service field with the value frontend-logs to all the logs received at the /frontend-logs/insert/* path. It also includes the _stream_fields parameter as an example of how to configure stream for such logs.

      users:
- bearer_token: foobar
  url_map:
  - src_paths: ["/frontend-logs/insert/.*"]
    url_prefix:
    - "http://vlinsert-1:9428?extra_fields=service=frontend-logs&_stream_fields=service"
    - "http://vlinsert-2:9428?extra_fields=service=frontend-logs&_stream_fields=service"
    - "http://vlinsert-3:9428?extra_fields=service=frontend-logs&_stream_fields=service"
    # drop /frontend-logs/ path prefix from the original request before proxying it to url_prefix.
    drop_src_path_prefix_parts: 1
  max_concurrent_requests: 10
    

Any field sent by the application will be overridden by the value set in the extra_fields, if defined. This prevents the log shipper from unexpectedly overriding the provided extra_fields.

Security on Untrusted Networks #

It supports Basic Auth and mTLS for authentication and access control. Traffic between clients and servers can be encrypted using TLS/SSL .

Recommended practices:

Deploy all VictoriaLogs components inside a private network, avoid direct Internet exposure.
Use vmauth as an authorization proxy and load balancer in front of vlinsert and vlselect for authentication and authorization of data ingestion and querying requests. See Security and Load Balancing for configuration details. For TLS termination proxy setup, see TLS termination proxy docs .
Protect components exposed to external clients with Basic Auth + TLS or mTLS.
Disable read API on vlinsert with the -select.disable command-line flag.
Disable write API on vlselect with the -insert.disable command-line flag.

Basic Auth #

It is recommended to enable Basic Auth when VictoriaLogs is deployed in untrusted or unsecured networks. For a full overview of security best practices, see the recommendations listed above.

Basic Auth validates the Authorization: Basic <base64(user:password)> HTTP header. If the username and password match the values configured in the auth layer (for example, VictoriaLogs or an upstream authorization proxy), the request is allowed.

Basic Auth encodes credentials using Base64 encoding, but does not encrypt the transmitted data. Always use Basic Auth exclusively over secure TLS/SSL connections .

Production Security Recommendations:

TLS encryption: never use Basic Auth over unencrypted HTTP connections.
Consider mTLS: implement mutual TLS for certificate-based authentication instead of password-based auth.
Regular credential rotation: change passwords periodically and use strong, unique passwords.

Quick start #

To quickly start VictoriaLogs with Basic Auth, run the following command:

      ./victoria-logs -httpAuth.username vlstorage -httpAuth.password secret

This will start the VictoriaLogs server with Basic authentication on the default port (9428).

To test the configuration, you can use curl:

      curl -u vlstorage:secret http://localhost:9428/metrics

This HTTP example is intended for local testing on localhost.

For cluster mode, you may configure both client authentication (users connecting to your service) and internal authentication (communication between VictoriaLogs components), depending on your network trust boundary and security requirements:

      ./victoria-logs -httpListenAddr :9429 \
    -httpAuth.username vlinsert -httpAuth.password top-secret \
    -storageNode localhost:9428 \
    -storageNode.username vlstorage -storageNode.password secret
    

This configuration:

Starts a VictoriaLogs server on port 9429 that functions as both vlinsert and vlselect.
Accepts requests to this node only from user vlinsert with password top-secret.
Connects to vlstorage at localhost:9428 using credentials vlstorage:secret.

Security Warning: Passing passwords directly in command-line arguments is not recommended for production, as they may appear in process listings (e.g., ps aux) or debug endpoints (e.g., http://victoria-logs:9428/debug/pprof/cmdline?debug=1).

Use one of the safer methods below:

After configuring Basic Auth, update your clients accordingly, such as vmauth (see these docs ) and VictoriaMetrics components (see these docs ).

Providing password from a file #

To read passwords from a file instead of command-line arguments, use the file:// prefix:

      ./victoria-logs -httpAuth.username vlstorage -httpAuth.password file:///absolute/path/to/file
# or relative path
./victoria-logs -httpAuth.username vlstorage -httpAuth.password file://./path/to/file
    

The value from the file is periodically reloaded, which allows changing the password without restarting the application.

Note: the -httpAuth.username command-line flag does not support reading values from a file.

For cluster deployments, you can also use dedicated file-based flags for storage node authentication:

      ./victoria-logs -httpListenAddr :9429 \
    -httpAuth.username vlinsert -httpAuth.password file://./path/to/file \
    -storageNode localhost:9428 \
    -storageNode.usernameFile /path/to/username/file -storageNode.passwordFile /path/to/password/file
    

Example with vlagent connecting to vlinsert:

      ./vlagent -httpListenAddr :9430 \
    -httpAuth.username vlagent -httpAuth.password file://./path/to/file \
    -remoteWrite.url http://localhost:9429/insert/native \
    -remoteWrite.basicAuth.username vlinsert -remoteWrite.basicAuth.passwordFile ./path/to/file
    

To test that vlagent is properly configured with Basic Auth, you can send a test log entry (use the same password value as configured in ./path/to/file for -httpAuth.password):

      curl -u "vlagent:$(cat ./path/to/file)" http://localhost:9430/insert/jsonline -H 'Content-Type: application/json' \
    -d '{"_msg":"Hello, VictoriaLogs!"}'
    

Providing password via environment variables #

To use environment variables for configuration, enable the feature with -envflag.enable. Optionally set a custom prefix with -envflag.prefix:

      export VL_httpAuth_username=vlstorage
export VL_httpAuth_password=secret

./victoria-logs -envflag.enable -envflag.prefix VL_
    

Flag names are converted to environment variables by replacing dots with underscores. For example, -httpAuth.password becomes VL_httpAuth_password.

Environment variables are read only at startup. You must restart VictoriaLogs to pick up changes.

Cluster configuration example:

      export VL_httpAuth_username=vlinsert
export VL_httpAuth_password=top-secret
export VL_storageNode_username=vlstorage
export VL_storageNode_password=secret

./victoria-logs -envflag.enable -envflag.prefix VL_ \
    -httpListenAddr :9429 -storageNode localhost:9428
    

vlagent configuration example:

      export VL_httpAuth_username=vlagent
export VL_httpAuth_password=top-top-secret
export VL_remoteWrite_basicAuth_username=vlinsert
export VL_remoteWrite_basicAuth_password=top-secret

./vlagent -envflag.enable -envflag.prefix VL_ \
    -httpListenAddr :9430 \
    -remoteWrite.url http://localhost:9429/insert/native
    

To test that vlagent is properly configured with Basic Auth, you can send a test log entry:

      curl -u vlagent:top-top-secret http://localhost:9430/insert/jsonline -H 'Content-Type: application/json' \
    -d '{"_msg":"Hello, VictoriaLogs!"}'
    

See also all supported command-line flags for VictoriaLogs (see these docs ) and vlagent (see these docs ).

Providing password via HTTP/HTTPS #

VictoriaLogs can retrieve passwords from HTTP/HTTPS endpoints for dynamic password management:

      ./victoria-logs -httpAuth.username foo \
    -httpAuth.password 'https://example.com/victoria-logs/password?example-arg=example-value'
    

The value from the URL is periodically reloaded, which allows changing the password without restarting the application.

Note: the -httpAuth.username command-line flag does not support reading values via HTTP/HTTPS.

If the HTTP request fails, VictoriaLogs continues using the last successfully retrieved password. Errors are logged for troubleshooting. If no password has ever been successfully retrieved and -httpAuth.username is set, requests protected by Basic Auth will fail with authentication errors.

Note: Ensure the URL endpoint is fast and reliable, because password retrieval from a slow or unstable endpoint can increase request latency during authentication checks.

System endpoints #

When Basic Auth or mTLS is enabled, the following system endpoints inherit the same authentication requirements by default. These defaults can be overridden per endpoint with dedicated *AuthKey flags.

/metrics - monitoring endpoint for VictoriaMetrics, vmagent, and Prometheus. Override authentication using the -metricsAuthKey command-line flag .
/flags - debugging endpoint that shows all active command-line flags. Override authentication using the -flagsAuthKey command-line flag . Note: Passwords are hidden in the output for security reasons.
/debug/pprof/* - profiling endpoints for performance analysis (advanced users only). Override authentication using the -pprofAuthKey command-line flag .
/internal/log_new_streams - enables logging new log streams during data ingestion. Override authentication using the -logNewStreamsAuthKey command-line flag .
/internal/force_flush - forces immediate flushing of in-memory data to disk. Override authentication using the -forceFlushAuthKey command-line flag .
/internal/force_merge - triggers manual data compaction and merge operations. Override authentication using the -forceMergeAuthKey command-line flag .
/internal/partition/* - manages partition lifecycle operations including creation and deletion. Override authentication using the -partitionManageAuthKey command-line flag .

Example of accessing the /metrics endpoint using an authentication key (works when -metricsAuthKey is configured):

      curl 'http://victoria-logs:9428/metrics?authKey=monitoring-secret-key'

TLS/SSL #

All VictoriaLogs components support TLS encryption for secure communication. TLS is disabled by default but can be enabled on both server and client sides.

Enabling TLS on the server #

To enable TLS encryption, provide the server certificate and private key:

      ./victoria-logs -tls -tlsCertFile /path/to/victoria-logs.pem -tlsKeyFile /path/to/victoria-logs-key.pem

This starts VictoriaLogs with HTTPS encryption on the default port (9428).

Connecting vlselect and vlinsert to vlstorage with TLS #

To connect VictoriaLogs components to a TLS-enabled vlstorage server:

      ./victoria-logs -httpListenAddr :9429 \
    -storageNode localhost:9428 -storageNode.tls
    

By default, VictoriaLogs verifies server certificates using the system’s trusted certificate store. If using self-signed certificates, you have two options:

Install your CA certificate in the system’s trusted certificate store.
Specify the CA certificate path manually using -storageNode.tlsCAFile.

For testing purposes only, you can disable certificate verification with -storageNode.tlsInsecureSkipVerify.

Security Warning: Disabling certificate verification eliminates protection against man-in-the-middle attacks. Never use this in production.

Connecting vlagent to VictoriaLogs with TLS #

To send data over TLS, simply change the URL scheme from http to https:

      ./vlagent -httpListenAddr :9430 \
    -remoteWrite.url https://localhost:9428/insert/native
    

By default, vlagent verifies server certificates using the system’s trusted certificate store. If using self-signed certificates, you have two options:

Install your CA certificate in the system’s trusted certificate store.
Specify the CA certificate path manually using -remoteWrite.tlsCAFile.

For testing purposes only, you can disable certificate verification with -remoteWrite.tlsInsecureSkipVerify.

Security Warning: Disabling certificate verification eliminates protection against man-in-the-middle attacks. Never use this in production.

Automatic issuing of TLS certificates #

All the VictoriaLogs Enterprise components support automatic issuing of TLS certificates for public HTTPS server running at -httpListenAddr via Let’s Encrypt service . The following command-line flags must be set in order to enable automatic issuing of TLS certificates:

-httpListenAddr must be set for listening TCP port 443. For example, -httpListenAddr=:443. This port must be accessible by the Let’s Encrypt service .
-tls must be set in order to accept HTTPS requests at -httpListenAddr. Note that -tlsCertFile and -tlsKeyFile aren’t needed when automatic TLS certificate issuing is enabled.
-tlsAutocertHosts must be set to comma-separated list of hosts, which can be reached via -httpListenAddr. TLS certificates are automatically issued for these hosts.
-tlsAutocertEmail must be set to contact email for the issued TLS certificates.
-tlsAutocertCacheDir may be set to the directory path for persisting the issued TLS certificates between VictoriaLogs restarts. If this flag isn’t set, then TLS certificates are re-issued on every restart.

Example of starting VictoriaLogs with automatic TLS certificate issuing:

      ./victoria-logs -httpListenAddr=:443 \
    -tls \
    -tlsAutocertHosts=victorialogs.example.com,logs.example.com \
    -tlsAutocertEmail=admin@example.com \
    -tlsAutocertCacheDir=/path/to/tls-cache \
    -licenseFile /path/to/license
    

The `no viable challenge type found` error #

Let’s Encrypt validates domain ownership by performing TLS-ALPN-01 challenges to verify you control the specified domains. The Let’s Encrypt service must be able to reach your VictoriaLogs instance over the public internet on port 443 to complete this validation.

Common troubleshooting steps:

Ensure your server is reachable from the internet on port 443.
Confirm that your domain names resolve to your server’s public IP address.
Use external tools to verify HTTPS connectivity to port 443.

Example connectivity verification:

      # Test domain resolution
nslookup victorialogs.example.com

# Test HTTPS connectivity from an external network (port 443)
curl -I https://victorialogs.example.com/
    

This functionality can be evaluated for free according to these docs .

Individual Configuration #

When connecting to multiple clients (storage nodes or remote write endpoints), each connection can have its own configuration.

Configuration Rules:

Single flag value: applied to all clients.
Multiple flag values: each client gets its own value in declaration order.
If multiple values are provided but fewer than the number of clients, remaining clients use the flag default/empty value unless explicitly set.

Example: Different TLS settings for storage nodes #

Enable TLS for one storage node while keeping another unencrypted:

      ./victoria-logs -storageNode vlstorage-1:9428 -storageNode.tls \
    -storageNode vlstorage-2:9428 -storageNode.tls=false
    

Example: Shared username with different passwords #

Use the same username for all nodes but different passwords:

      ./victoria-logs -storageNode.username vlselect \
    -storageNode vlstorage-1:9428 -storageNode.passwordFile /path/to/password-vlstorage-1.txt \
    -storageNode vlstorage-2:9428 -storageNode.passwordFile /path/to/password-vlstorage-2.txt
    

Example: Mixed authentication methods #

Some clients use Bearer tokens while others use Basic Auth:

      ./victoria-logs \
    -storageNode vlstorage-1:9428 -storageNode.bearerTokenFile /path/to/bearer.txt -storageNode.passwordFile="" -storageNode.username="" \
    -storageNode vlstorage-2:9428 -storageNode.username vlstorage -storageNode.passwordFile /path/to/password.txt -storageNode.bearerTokenFile=""
    

Note: When mixing authentication methods, explicitly set unused parameters to empty strings ("") to avoid configuration conflicts.

The same configuration logic applies to all -remoteWrite.* and -storageNode.* flags.

For the complete list of available flags, refer to these docs .

mTLS #

This feature requires Enterprise binaries for components that use mTLS (for example, VictoriaLogs and/or vlagent).

Mutual TLS (mTLS) provides the highest level of security by requiring both client and server to present valid certificates for authentication. Unlike standard TLS where only the server authenticates itself, mTLS enables bidirectional authentication.

Production Security Recommendations:

Implement proper certificate rotation and expiration monitoring
Store private keys securely with appropriate file permissions (e.g., 600)
Implement certificate revocation procedures for compromised certificates
Keep certificates updated before expiration to avoid service disruptions

Enabling mTLS on the server #

mTLS requires both standard TLS configuration and additional mutual authentication settings:

      ./victoria-logs -tls \
    -tlsCertFile ./victoria-logs.pem \
    -tlsKeyFile ./victoria-logs-key.pem \
    -mtls \
    -licenseFile /path/to/license
    

By default, VictoriaLogs verifies client certificates using the system’s trusted certificate store. If using certificates signed by a private CA not present in the system trust store, you have two options:

Install your CA certificate in the system’s trusted certificate store.
Specify the CA certificate path manually using -mtlsCAFile.

Connecting vlselect and vlinsert to vlstorage with mTLS #

Client components need both server verification and client authentication certificates:

      ./victoria-logs -storageNode vlstorage:9428 \
    -storageNode.tls \
    -storageNode.tlsCAFile /path/to/server-ca.pem \
    -storageNode.tlsCertFile /path/to/client-cert.pem \
    -storageNode.tlsKeyFile /path/to/client-key.pem
    

-storageNode.tlsCAFile - CA certificate to verify the server’s identity.
-storageNode.tlsCertFile - client certificate for client authentication.
-storageNode.tlsKeyFile - private key corresponding to the client certificate.

Connecting vlagent to VictoriaLogs with mTLS #

vlagent mTLS configuration follows the same pattern:

      ./vlagent -remoteWrite.url https://vlinsert:9428/insert/native \
    -remoteWrite.tlsCAFile /path/to/server-ca.pem \
    -remoteWrite.tlsCertFile /path/to/client-cert.pem \
    -remoteWrite.tlsKeyFile /path/to/client-key.pem
    

mTLS with different client certificates #

For multiple storage nodes, each client can use its own unique certificate:

      ./victoria-logs -storageNode.tls \
    -storageNode.tlsCAFile /path/to/server-ca.pem \
    -storageNode vlstorage-1:9428 -storageNode.tlsCertFile /path/to/client1-cert.pem -storageNode.tlsKeyFile /path/to/client1-key.pem \
    -storageNode vlstorage-2:9428 -storageNode.tlsCertFile /path/to/client2-cert.pem -storageNode.tlsKeyFile /path/to/client2-key.pem
    

This approach allows for granular access control and easier certificate management in complex deployments.

Certificate reloading #

VictoriaLogs automatically re-reads TLS certificate files (server certificates, client certificates, and CA certificates) without requiring server or client restarts. Certificate and TLS config values are cached for up to 1 second and then refreshed for new handshakes/requests, which enables seamless certificate rotation in production environments.

The certificate reloading feature works for:

Server certificates (-tlsCertFile and -tlsKeyFile).
Client certificates (-storageNode.tlsCertFile, -storageNode.tlsKeyFile, -remoteWrite.tlsCertFile, -remoteWrite.tlsKeyFile).
CA certificates (-mtlsCAFile, -storageNode.tlsCAFile, -remoteWrite.tlsCAFile).

To update certificates:

Replace the certificate files on disk with new versions.
VictoriaLogs will automatically detect and load the updated certificates.
New connections will use the updated certificates immediately.
Existing connections will continue using the old certificates until they reconnect.

Note: Ensure new certificates are valid and properly signed before replacing the old ones, as invalid certificates will cause authentication failures for new connections.

Certificate generation example #

Create a Certificate Authority (CA):

      # Generate CA private key
openssl genrsa -out ca-key.pem 4096

# Generate CA certificate (valid for 1 year)
openssl req -new -x509 -days 365 -key ca-key.pem -out ca-cert.pem -subj "/CN=VictoriaLogs-CA"
    

Generate server certificate:

      # Generate server private key
openssl genrsa -out server-key.pem 4096

# Create certificate signing request
openssl req -new -key server-key.pem -out server.csr -subj "/CN=localhost"

# Sign server certificate with CA
openssl x509 -req -days 365 -in server.csr -CA ca-cert.pem -CAkey ca-key.pem -out server-cert.pem -CAcreateserial
    

This minimal example is intended for localhost testing. For production (or IP/real hostname access), generate the server certificate with subjectAltName (SAN).

Generate client certificate:

      # Generate client private key
openssl genrsa -out client-key.pem 4096

# Create client certificate
openssl req -new -key client-key.pem -out client.csr -subj "/CN=victoria-logs-client"

# Sign client certificate with CA
openssl x509 -req -days 365 -in client.csr -CA ca-cert.pem -CAkey ca-key.pem -out client-cert.pem -CAcreateserial
    

Certificate Files Summary:

ca-cert.pem - Used for both server and client certificate verification
server-cert.pem + server-key.pem - Server certificate and key
client-cert.pem + client-key.pem - Client certificate and key

Start VictoriaLogs with these certificates:

      ./victoria-logs -tls \
    -tlsCertFile ./server-cert.pem \
    -tlsKeyFile ./server-key.pem \
    -mtls \
    -mtlsCAFile ./ca-cert.pem \
    -licenseFile /path/to/license
    

Test the connection using curl:

      curl --cacert ./ca-cert.pem --key ./client-key.pem --cert ./client-cert.pem https://localhost:9428/metrics

Previous Alerting with Logs Next FAQ

Security and Load Balancing #