VictoriaMetrics is a fast, cost-effective and scalable time series database. It can be used as a long-term remote storage for Prometheus.
It is recommended to use the single-node version instead of the cluster version
for ingestion rates lower than a million data points per second.
The single-node version scales perfectly
with the number of CPU cores, RAM and available storage space.
The single-node version is easier to configure and operate compared to the cluster version, so think twice before choosing the cluster version.
See this question for more details.
There is also user-friendly database for logs - VictoriaLogs.
VictoriaMetrics is developed at a fast pace, so it is recommended periodically checking the CHANGELOG and performing regular upgrades.
VictoriaMetrics has achieved security certifications for Database Software Development and Software-Based Monitoring Services. We apply strict security measures in everything we do. See our Security page for more details.
VictoriaMetrics cluster consists of the following services:
vmstorage - stores the raw data and returns the queried data on the given time range for the given label filters
vminsert - accepts the ingested data and spreads it among vmstorage nodes according to consistent hashing over metric name and all its labels
vmselect - performs incoming queries by fetching the needed data from all the configured vmstorage nodes
Each service may scale independently and may run on the most suitable hardware.
vmstorage nodes don’t know about each other, don’t communicate with each other and don’t share any data.
This is a shared nothing architecture.
It increases cluster availability, and simplifies cluster maintenance as well as cluster scaling.
Note
Note that vmselect despite being stateless still requires some disk space (a few GBs) for temporary caches. Refer to the -cacheDataPath command-line flag for more details.
VictoriaMetrics cluster version provides UI for query troubleshooting and exploration. The UI is available at
http://<vmselect>:8481/select/<accountID>/vmui/ in each vmeselect service.
The UI allows exploring query results via graphs and tables. See more details about vmui.
VictoriaMetrics cluster supports multiple isolated tenants (aka namespaces).
Tenants are identified by accountID or accountID:projectID, which are put inside request URLs for writes and reads.
See these docs for details.
Some facts about tenants in VictoriaMetrics:
Each accountID and projectID is identified by an arbitrary 32-bit integer in the range [0 .. 2^32).
If projectID is missing, then it is automatically assigned to 0. It is expected that other information about tenants
such as auth tokens, tenant names, limits, accounting, etc. is stored in a separate relational database. This database must be managed
by a separate service sitting in front of VictoriaMetrics cluster such as vmauth
or vmgateway. Contact us if you need assistance with such service.
Tenants are automatically created when the first data point is written into the given tenant.
Data for all the tenants is evenly spread among available vmstorage nodes. This guarantees even load among vmstorage nodes
when different tenants have different amounts of data and different query load.
The database performance and resource usage doesn’t depend on the number of tenants. It depends mostly on the total number of active time series in all the tenants. A time series is considered active if it received at least a single sample during the last hour or it has been touched by queries during the last hour.
The list of registered tenants can be obtained via http://<vmselect>:8481/admin/tenants url. See these docs.
VictoriaMetrics exposes various per-tenant statistics via metrics - see these docs.
vminsert can accept data from multiple tenants via a special multitenant endpoints http://vminsert:8480/insert/multitenant/<suffix>,
where <suffix> can be replaced with any supported suffix for data ingestion from this list.
In this case the account ID and project ID are obtained from optional vm_account_id and vm_project_id labels of the incoming samples.
If vm_account_id or vm_project_id labels are missing or invalid, then the corresponding account ID and project ID are set to 0.
These labels are automatically removed from samples before forwarding them to vmstorage.
For example, if the following samples are written into http://vminsert:8480/insert/multitenant/prometheus/api/v1/write:
Then the http_requests_total{path="/foo"} 12 would be stored in the tenant accountID=42, projectID=0,
while the http_requests_total{path="/bar"} 34 would be stored in the tenant accountID=7, projectID=9.
The vm_account_id and vm_project_id labels are extracted after applying the relabeling
set via -relabelConfig command-line flag, so these labels can be set at this stage.
vmselect can execute Available from v1.104.0queries over multiple tenants via special multitenant endpoints http://vmselect:8481/select/multitenant/<suffix>.
Currently supported endpoints for <suffix> are:
/prometheus/api/v1/query
/prometheus/api/v1/query_range
/prometheus/api/v1/series
/prometheus/api/v1/labels
/prometheus/api/v1/label/<label_name>/values
/prometheus/api/v1/status/active_queries
/prometheus/api/v1/status/top_queries
/prometheus/api/v1/status/tsdb
/prometheus/api/v1/export
/prometheus/api/v1/export/csv
/vmui
It is allowed to explicitly specify tenant IDs via vm_account_id and vm_project_id labels in the query.
For example, the following query fetches metric up for the tenants accountID=42 and accountID=7, projectID=9:
1
up{vm_account_id="7", vm_project_id="9" or vm_account_id="42"}
vm_account_id and vm_project_id labels support all operators for label matching. For example:
1
2
up{vm_account_id!="42"} # selects all the time series except those belonging to accountID=42
up{vm_account_id=~"4.*"} # selects all the time series belonging to accountIDs starting with 4
The precedence for applying filters for tenants follows this order:
Filter tenants by extra_label and extra_filters filters.
These filters have the highest priority and are applied first when provided through the query arguments.
Filter tenants from labels selectors defined at metricsQL query expression.
Security considerations
It is recommended restricting access to multitenant endpoints only to trusted sources,
since untrusted source may break per-tenant data by writing unwanted samples or get access to data of arbitrary tenants.
Run make package. It will build the following docker images locally:
victoriametrics/vminsert:<PKG_TAG>
victoriametrics/vmselect:<PKG_TAG>
victoriametrics/vmstorage:<PKG_TAG>
<PKG_TAG> is auto-generated image tag, which depends on source code in the repository.
The <PKG_TAG> may be manually set via PKG_TAG=foobar make package.
By default, images are built on top of alpine image in order to improve debuggability.
It is possible to build an image on top of any other base image by setting it via <ROOT_IMAGE> environment variable.
For example, the following command builds images on top of scratch image:
The database is considered highly available if it continues accepting new data and processing incoming queries when some of its components are temporarily unavailable.
VictoriaMetrics cluster is highly available according to this definition - see cluster availability docs.
It is recommended to run all the components for a single cluster in the same subnetwork with high bandwidth, low latency and low error rates.
This improves cluster performance and availability. It isn’t recommended spreading components for a single cluster
across multiple availability zones, since cross-AZ network usually has lower bandwidth, higher latency and higher
error rates comparing the network inside a single AZ.
If you need multi-AZ setup, then it is recommended running independent clusters in each AZ and setting up
vmagent in front of these clusters, so it could replicate incoming data
into all the cluster - see these docs for details.
Then an additional vmselect nodes can be configured for reading the data from multiple clusters according to these docs.
It is recommended to run at least two nodes for each service for high availability purposes. In this case the cluster continues working when a single node is temporarily unavailable and the remaining nodes can handle the increased workload. The node may be temporarily unavailable when the underlying hardware breaks, during software upgrades, migration or other maintenance tasks.
It is preferred to run many small vmstorage nodes over a few big vmstorage nodes, since this reduces the workload increase on the remaining vmstorage nodes when some of vmstorage nodes become temporarily unavailable.
An http load balancer such as vmauth or nginx must be put in front of vminsert and vmselect nodes.
It must contain the following routing configs according to the url format:
requests starting with /insert must be routed to port 8480 on vminsert nodes.
requests starting with /select must be routed to port 8481 on vmselect nodes.
Ports may be altered by setting -httpListenAddr on the corresponding nodes.
It is recommended setting up monitoring for the cluster.
It is possible manually setting up a toy cluster on a single host. In this case every cluster component - vminsert, vmselect and vmstorage - must have distinct values for -httpListenAddr command-line flag. This flag specifies http address for accepting http requests for monitoring and profiling. vmstorage node must have distinct values for the following additional command-line flags in order to prevent resource usage clash:
-storageDataPath - every vmstorage node must have a dedicated data storage.
-vminsertAddr - every vmstorage node must listen for a distinct tcp address for accepting data from vminsert nodes.
-vmselectAddr - every vmstorage node must listen for a distinct tcp address for accepting requests from vmselect nodes.
VictoriaMetrics cluster architecture prioritizes availability over data consistency.
This means that the cluster remains available for data ingestion and data querying
if some of its components are temporarily unavailable.
VictoriaMetrics cluster remains available if the following conditions are met:
HTTP load balancer must stop routing requests to unavailable vminsert and vmselect nodes
(vmauth stops routing requests to unavailable nodes).
At least a single vminsert node must remain available in the cluster for processing data ingestion workload.
The remaining active vminsert nodes must have enough compute capacity (CPU, RAM, network bandwidth)
for handling the current data ingestion workload.
If the remaining active vminsert nodes have no enough resources for processing the data ingestion workload,
then arbitrary delays may occur during data ingestion.
See capacity planning and cluster resizing docs for more details.
At least a single vmselect node must remain available in the cluster for processing query workload.
The remaining active vmselect nodes must have enough compute capacity (CPU, RAM, network bandwidth, disk IO)
for handling the current query workload.
If the remaining active vmselect nodes have no enough resources for processing query workload,
then arbitrary failures and delays may occur during query processing.
See capacity planning and cluster resizing docs for more details.
At least a single vmstorage node must remain available in the cluster for accepting newly ingested data
and for processing incoming queries. The remaining active vmstorage nodes must have enough compute capacity
(CPU, RAM, network bandwidth, disk IO, free disk space) for handling the current workload.
If the remaining active vmstorage nodes have no enough resources for processing query workload,
then arbitrary failures and delay may occur during data ingestion and query processing.
See capacity planning and cluster resizing docs for more details.
The cluster works in the following way when some of vmstorage nodes are unavailable:
vminsert re-routes newly ingested data from unavailable vmstorage nodes to remaining healthy vmstorage nodes.
This guarantees that the newly ingested data is properly saved if the healthy vmstorage nodes have enough CPU, RAM, disk IO and network bandwidth
for processing the increased data ingestion workload.
vminsert spreads evenly the additional data among the healthy vmstorage nodes in order to spread evenly
the increased load on these nodes. During re-routing, healthy vmstorage nodes will experience higher resource usage
and increase in number of active time series.
vmselect continues serving queries if at least a single vmstorage nodes is available.
It marks responses as partial for queries served from the remaining healthy vmstorage nodes,
since such responses may miss historical data stored on the temporarily unavailable vmstorage nodes.
Every partial JSON response contains "isPartial": true option.
If you prefer consistency over availability, then run vmselect nodes with -search.denyPartialResponse command-line flag.
In this case vmselect returns an error if at least a single vmstorage node is unavailable.
Another option is to pass deny_partial_response=1 query arg to requests to vmselect nodes.
vmselect also accepts -replicationFactor=N command-line flag. This flag instructs vmselect to return full response
if less than -replicationFactor vmstorage nodes are unavailable during querying, since it assumes that the remaining
vmstorage nodes contain the full data. See these docs for details.
It is also possible to configure independent replication factor per distinct vmstorage groups - see these docs.
vmselect doesn’t serve partial responses for API handlers returning raw datapoints,
since users usually expect this data is always complete. The following handlers return raw samples:
/api/v1/query when the query contains series selector
ending with some duration in square brackets. For example, /api/v1/query?query=up[1h]&time=2024-01-02T03:00:00Z.
This query returns raw samples for time series
with the up name on the time range (2024-01-02T02:00:00 .. 2024-01-02T03:00:00]. See this article
for details.
Data replication can be used for increasing storage durability. See these docs for details.
vmselect nodes can be queried by other vmselect nodes if they run with -clusternativeListenAddr command-line flag.
For example, if vmselect is started with -clusternativeListenAddr=:8401, then it can accept queries from another vmselect nodes at TCP port 8401
in the same way as vmstorage nodes do. This allows chaining vmselect nodes and building multi-level cluster topologies.
For example, the top-level vmselect node can query second-level vmselect nodes in different availability zones (AZ),
while the second-level vmselect nodes can query vmstorage nodes in local AZ. See also vmstorage groups at vmselect.
vminsert nodes can accept data from another vminsert nodes if they run with -clusternativeListenAddr command-line flag.
For example, if vminsert is started with -clusternativeListenAddr=:8400, then it can accept data from another vminsert nodes at TCP port 8400
in the same way as vmstorage nodes do. This allows chaining vminsert nodes and building multi-level cluster topologies.
For example, the top-level vminsert node can replicate data among the second level of vminsert nodes located in distinct availability zones (AZ),
while the second-level vminsert nodes can spread the data among vmstorage nodes in local AZ.
The multi-level cluster setup for vminsert nodes has the following shortcomings because of synchronous replication and data sharding:
Data ingestion speed is limited by the slowest link to AZ.
vminsert nodes at top level re-route incoming data to the remaining AZs when some AZs are temporarily unavailable. This results in data gaps at AZs which were temporarily unavailable.
These issues are addressed by vmagent when it runs in multitenancy mode.
vmagent buffers data, which must be sent to a particular AZ, when this AZ is temporarily unavailable. The buffer is stored on disk. The buffered data is sent to AZ as soon as it becomes available.
vmselect can be configured to query multiple distinct groups of vmstorage nodes with individual -replicationFactor per each group.
The following format for -storageNode command-line flag value should be used for assigning a particular addr of vmstorage to a particular groupName -
-storageNode=groupName/addr. The groupName can contain arbitrary value. The only rule is that every vmstorage group must have an unique name.
For example, the following command runs vmselect, which continues returning full responses if up to one node per each group is temporarily unavailable
because the given -replicationFactor=2 is applied individually per each group:
It is possible specifying distinct -replicationFactor per each group via the following format - -replicationFactor=groupName:rf.
For example, the following command runs vmselect, which uses -replicationFactor=3 for the group g1, -replicationFactor=2 for the group g2
and -replicationFactor=1 for the group g3:
If every ingested sample is replicated across multiple vmstorage groups, then pass -globalReplicationFactor=N command-line flag to vmselect,
so it could continue returning full responses if up to N-1vmstorage groups are temporarily unavailable.
For example, the following command runs vmselect, which continues returning full responses if any number of vmstorage nodes
in a single vmstorage group are temporarily unavailable:
It is OK to mix -replicationFactor and -globalReplicationFactor. For example, the following command runs vmselect, which continues returning full responses
if any number of vmstorage nodes in a single vmstorage group are temporarily unavailable and the remaining groups contain up to two unavailable vmstorage node:
vminsert and vmselect components in enterprise version of VictoriaMetrics support
the following approaches for automatic discovery of vmstorage nodes:
file-based discovery - put the list of vmstorage nodes into a file - one node address per each line - and then pass -storageNode=file:/path/to/file-with-vmstorage-list
to vminsert and vmselect. It is possible to read the list of vmstorage nodes from http or https urls.
For example, -storageNode=file:http://some-host/vmstorage-list would read the list of storage nodes
from http://some-host/vmstorage-list.
The list of discovered vmstorage nodes is automatically updated when the file contents changes.
The update frequency can be controlled with -storageNode.discoveryInterval command-line flag.
DNS SRV - pass srv+some-name value to -storageNode command-line flag.
In this case the provided some-name is resolved into tcp addresses of vmstorage nodes.
The list of discovered vmstorage nodes is automatically updated at vminsert and vmselect
when it changes behind the some-name DNS SRV record.
The update frequency can be controlled with -storageNode.discoveryInterval command-line flag.
It is possible passing multiple file and DNS SRV names to -storageNode command-line flag. In this case all these names
are resolved to tcp addresses of vmstorage nodes to connect to.
For example, -storageNode=file:/path/to/local-vmstorage-list -storageNode='srv+vmstorage-hot' -storageNode='srv+vmstorage-cold'.
It is OK to pass regular static vmstorage addresses together with file and DNS SRV addresses at -storageNode. For example,
-storageNode=vmstorage1,vmstorage2 -storageNode='srv+vmstorage-autodiscovery'.
The discovered addresses can be filtered with optional -storageNode.filter command-line flag, which can contain arbitrary regular expression filter.
For example, -storageNode.filter='^[^:]+:8400$' would leave discovered addresses ending with 8400 port only, e.g. the default port used
for sending data from vminsert to vmstorage node according to -vminsertAddr command-line flag.
The currently discovered vmstorage nodes can be monitored with vm_rpc_vmstorage_is_reachable and vm_rpc_vmstorage_is_read_only metrics.
All the VictoriaMetrics components allow referring environment variables in command-line flags via %{ENV_VAR} syntax.
For example, -metricsAuthKey=%{METRICS_AUTH_KEY} is automatically expanded to -metricsAuthKey=top-secret
if METRICS_AUTH_KEY=top-secret environment variable exists at VictoriaMetrics startup.
This expansion is performed by VictoriaMetrics itself.
VictoriaMetrics recursively expands %{ENV_VAR} references in environment variables on startup.
For example, FOO=%{BAR} environment variable is expanded to FOO=abc if BAR=a%{BAZ} and BAZ=bc.
Additionally, all the VictoriaMetrics components allow setting flag values via environment variables according to these rules:
The -envflag.enable flag must be set
Each . in flag names must be substituted by _ (for example -insert.maxQueueDuration <duration> will translate to insert_maxQueueDuration=<duration>)
For repeating flags, an alternative syntax can be used by joining the different values into one using , as separator (for example -storageNode <nodeA> -storageNode <nodeB> will translate to storageNode=<nodeA>,<nodeB>)
It is possible setting prefix for environment vars with -envflag.prefix. For instance, if -envflag.prefix=VM_, then env vars must be prepended with VM_
All the VictoriaMetrics cluster components must run in protected private network without direct access from untrusted networks such as Internet.
External clients must access vminsert and vmselect via auth proxy such as vmauth
or vmgateway.
The auth proxy must accept auth tokens from untrusted networks only via https in order to protect the auth tokens from MitM attacks.
It is recommended using distinct auth tokens for distinct tenants in order to reduce potential damage in case of compromised auth token for some tenants.
Prefer using lists of allowed API endpoints, while disallowing access to other endpoints when configuring auth proxy in front of vminsert and vmselect.
This minimizes attack surface.
By default vminsert and vmselect nodes accept http requests at 8480 and 8481 ports accordingly (these ports can be changed via -httpListenAddr command-line flags),
since it is expected that vmauth is used for authorization and TLS termination
in front of vminsert and vmselect.
Enterprise version of VictoriaMetrics supports the ability to accept mTLS
requests at 8480 and 8481 ports for vminsert and vmselect nodes, by specifying -tls and -mtls command-line flags.
For example, the following command runs vmselect, which accepts only mTLS requests at port 8481:
1
./vmselect -tls -mtls
By default, system-wide TLS Root CA is used for verifying client certificates if -mtls command-line flag is specified.
It is possible to specify custom TLS Root CA via -mtlsCAFile command-line flag.
By default vminsert and vmselect nodes use unencrypted connections to vmstorage nodes, since it is assumed that all the cluster components run in a protected environment. Enterprise version of VictoriaMetrics provides optional support for mTLS connections between cluster components. Pass -cluster.tls=true command-line flag to vminsert, vmselect and vmstorage nodes in order to enable mTLS protection. Additionally, vminsert, vmselect and vmstorage must be configured with mTLS certificates via -cluster.tlsCertFile, -cluster.tlsKeyFile command-line options. These certificates are mutually verified when vminsert and vmselect dial vmstorage.
The following optional command-line flags related to mTLS are supported:
-cluster.tlsInsecureSkipVerify can be set at vminsert, vmselect and vmstorage in order to disable peer certificate verification. Note that this breaks security.
-cluster.tlsCAFile can be set at vminsert, vmselect and vmstorage for verifying peer certificates issued with custom certificate authority. By default, system-wide certificate authority is used for peer certificate verification.
When vmselect runs with -clusternativeListenAddr command-line option, then it can be configured with -clusternative.tls* options similar to -cluster.tls* for accepting mTLS connections from top-level vmselect nodes in multi-level cluster setup.
See these docs on how to set up mTLS in VictoriaMetrics cluster.
All the cluster components expose various metrics in Prometheus-compatible format at /metrics page on the TCP port set in -httpListenAddr command-line flag.
By default, the following TCP ports are used:
If you use Google Cloud Managed Prometheus for scraping metrics from VictoriaMetrics components, then pass -metrics.exposeMetadata
command-line to them, so they add TYPE and HELP comments per each exposed metric at /metrics page.
See these docs for details.
vmstorage nodes can be configured with limits on the number of unique time series across all the tenants with the following command-line flags:
-storage.maxHourlySeries is the limit on the number of active time series during the last hour.
-storage.maxDailySeries is the limit on the number of unique time series during the day. This limit can be used for limiting daily time series churn rate.
Note that these limits are set and applied individually per each vmstorage node in the cluster. So, if the cluster has Nvmstorage nodes, then the cluster-level limits will be N times bigger than the per-vmstorage limits.
See more details about cardinality limiter in these docs.
vmstorage nodes automatically switch to readonly mode when the directory pointed by -storageDataPath
contains less than -storage.minFreeDiskSpaceBytes of free space. vminsert nodes stop sending data to such nodes
and start re-routing the data to the remaining vmstorage nodes.
vmstorage sets vm_storage_is_read_only metric at http://vmstorage:8482/metrics to 1 when it enters read-only mode.
The metric is set to 0 when the vmstorage isn’t in read-only mode.
The main differences between URL formats of cluster and Single server
versions are that cluster has separate components for read and ingestion path, and because of multi-tenancy support.
Also in the cluster version the /prometheus/api/v1 endpoint ingests jsonl, csv, native and prometheus data formats not only prometheus data.
Check practical examples of VictoriaMetrics API here.
URLs for data ingestion: http://<vminsert>:8480/insert/<accountID>/<suffix>, where:
<accountID> is an arbitrary 32-bit integer identifying namespace for data ingestion (aka tenant). It is possible to set it as accountID:projectID,
where projectID is also arbitrary 32-bit integer. If projectID isn’t set, then it equals to 0. See multitenancy docs for more details.
The <accountID> can be set to multitenant string, e.g. http://<vminsert>:8480/insert/multitenant/<suffix>. Such urls accept data from multiple tenants
specified via vm_account_id and vm_project_id labels. See multitenancy via labels for more details.
influx/write and influx/api/v2/write - for ingesting data with InfluxDB line protocol. TCP and UDP receiver is disabled by default. It is exposed on a distinct TCP address set via -influxListenAddr command-line flag. See these docs for details.
opentsdb/api/put - for accepting OpenTSDB HTTP /api/put requests. This handler is disabled by default. It is exposed on a distinct TCP address set via -opentsdbHTTPListenAddr command-line flag. See these docs for details.
URLs for Prometheus querying API: http://<vmselect>:8481/select/<accountID>/prometheus/<suffix>, where:
<accountID> is an arbitrary number identifying data namespace for the query (aka tenant)
api/v1/export - exports raw data in JSON line format. See this article for details.
api/v1/export/native - exports raw data in native binary format. It may be imported into another VictoriaMetrics via api/v1/import/native (see above).
api/v1/export/csv - exports data in CSV. It may be imported into another VictoriaMetrics via api/v1/import/csv (see above).
api/v1/series/count - returns the total number of series.
api/v1/status/tsdb - for time series stats. See these docs for details.
api/v1/status/active_queries - for currently executed active queries. Note that every vmselect maintains an independent list of active queries,
which is returned in the response.
api/v1/status/top_queries - for listing the most frequently executed queries and queries taking the most duration.
tags/<tag_name> - returns tag values for the given <tag_name>. See these docs.
tags/findSeries - returns series matching the given expr. See these docs.
tags/autoComplete/tags - returns tags matching the given tagPrefix and/or expr. See these docs.
tags/autoComplete/values - returns tag values matching the given valuePrefix and/or expr. See these docs.
tags/delSeries - deletes series matching the given path. See these docs.
URL with basic Web UI: http://<vmselect>:8481/select/<accountID>/vmui/.
URL for query stats across all tenants: http://<vmselect>:8481/api/v1/status/top_queries. It lists with the most frequently executed queries and queries taking the most duration.
URL for time series deletion: http://<vmselect>:8481/delete/<accountID>/prometheus/api/v1/admin/tsdb/delete_series?match[]=<timeseries_selector_for_delete>.
Note that the delete_series handler should be used only in exceptional cases such as deletion of accidentally ingested incorrect time series. It shouldn’t
be used on a regular basis, since it carries non-zero overhead.
URL for listing tenants with the ingested data on the given time range: http://<vmselect>:8481/admin/tenants?start=...&end=... .
The start and end query args are optional. If they are missing, then all the tenants with at least one sample stored in VictoriaMetrics are returned.
URL for accessing vmalerts UI: http://<vmselect>:8481/select/<accountID>/prometheus/vmalert/.
This URL works only when -vmalert.proxyURL flag is set. See more about vmalert here.
vmstorage nodes provide the following HTTP endpoints on 8482 port:
/internal/force_merge - initiate forced compactions on the given vmstorage node.
/snapshot/create - create instant snapshot,
which can be used for backups in background. Snapshots are created in <storageDataPath>/snapshots folder, where <storageDataPath> is the corresponding
command-line flag value.
/snapshot/list - list available snapshots.
/snapshot/delete?snapshot=<id> - delete the given snapshot.
/snapshot/delete_all - delete all the snapshots.
Snapshots may be created independently on each vmstorage node. There is no need in synchronizing snapshots’ creation
across vmstorage nodes.
Cluster performance and capacity can be scaled up in two ways:
By adding more resources (CPU, RAM, disk IO, disk space, network bandwidth) to existing nodes in the cluster (aka vertical scalability).
By adding more nodes to the cluster (aka horizontal scalability).
General recommendations for cluster scalability:
Adding more CPU and RAM to existing vmselect nodes improves the performance for heavy queries, which process big number of time series with big number of raw samples.
See this article on how to detect and optimize heavy queries.
Adding more vmstorage nodes (aka horizontal scaling) increases the number of active time series
the cluster can handle. This also increases query performance over time series with high churn rate,
since every vmstorage node contains lower number of time series when the number of vmstorage nodes increases.
The cluster stability is also improved with the number of vmstorage nodes, since active vmstorage nodes need to handle lower additional workload
when some of vmstorage nodes become unavailable. For example, if one node out of 3 nodes is unavailable, then 1/3=33% of the load is re-distributed across 2 remaining nodes,
so per-node workload increase is (1/3/2)/(1/3) = 1/2 = 50%.
If one node out of 10 nodes is unavailable, then 1/10=10% of the load is re-distributed across 9 remaining nodes, so per-node workload increase is (1/10/9)/(1/10) = 1/9 =~ 11%.
Adding more CPU and RAM to existing vmstorage nodes (aka vertical scaling) increases the number
of active time series the cluster can handle.
It is preferred to add more vmstorage nodes over adding more CPU and RAM to existing vmstorage nodes, since higher number of vmstorage nodes
increases cluster stability and improves query performance over time series with high churn rate.
Adding more vminsert nodes increases the maximum possible data ingestion speed, since the ingested data may be split among bigger number of vminsert nodes.
Adding more vmselect nodes increases the maximum possible queries rate, since the incoming concurrent requests may be split among bigger number of vmselect nodes.
Steps to add vmstorage node:
Start new vmstorage node with the same -retentionPeriod as existing nodes in the cluster.
Gradually restart all the vmselect nodes with new -storageNode arg containing <new_vmstorage_host>.
Gradually restart all the vminsert nodes with new -storageNode arg containing <new_vmstorage_host>.
In order to handle uneven disk space usage distribution after adding new vmstorage node it is possible to update vminsert configuration to route newly ingested metrics only to new storage nodes. Once disk usage will be similar configuration can be updated to include all nodes again. Note that vmselect nodes need to reference all storage nodes for querying.
All the node types - vminsert, vmselect and vmstorage - may be updated via graceful shutdown.
Send SIGINT signal to the corresponding process, wait until it finishes and then start new version
with new configs.
There are the following cluster update / upgrade approaches exist:
Gracefully restart every node in the cluster one-by-one with the updated config / upgraded binary.
It is recommended restarting the nodes in the following order:
Restart vmstorage nodes.
Restart vminsert nodes.
Restart vmselect nodes.
This strategy allows upgrading the cluster without downtime if the following conditions are met:
The cluster has at least a pair of nodes of each type - vminsert, vmselect and vmstorage,
so it can continue to accept new data and serve incoming requests when a single node is temporary unavailable
during its restart. See cluster availability docs for details.
The cluster has enough compute resources (CPU, RAM, network bandwidth, disk IO) for processing
the current workload when a single node of any type (vminsert, vmselect or vmstorage)
is temporarily unavailable during its restart.
The updated config / upgraded binary is compatible with the remaining components in the cluster.
See the CHANGELOG for compatibility notes between different releases.
If at least a single condition isn’t met, then the rolling restart may result in cluster unavailability
during the config update / version upgrade. In this case the following strategy is recommended.
Gracefully stop all the vminsert and vmselect nodes in parallel.
Gracefully restart all the vmstorage nodes in parallel.
Start all the vminsert and vmselect nodes in parallel.
The cluster is unavailable for data ingestion and querying when performing the steps above.
The downtime is minimized by restarting cluster nodes in parallel at every step above.
The minimum downtime strategy has the following benefits comparing to no downtime strategy:
It allows performing config update / version upgrade with minimum disruption
when the previous config / version is incompatible with the new config / version.
It allows performing config update / version upgrade with minimum disruption
when the cluster has no enough compute resources (CPU, RAM, disk IO, network bandwidth)
for rolling upgrade.
It allows minimizing the duration of config update / version upgrade for clusters with big number of nodes
of for clusters with big vmstorage nodes, which may take long time for graceful restart.
vmstorage nodes may experience increased usage for CPU, RAM and disk IO during
rolling restarts,
since they need to process higher load when some of vmstorage nodes are temporarily unavailable in the cluster.
The following approaches can be used for reducing resource usage at vmstorage nodes during rolling restart:
To pass -disableReroutingOnUnavailable command-line flag to vminsert nodes, so they pause data ingestion when vmstorage nodes are restarted
instead of re-routing the ingested data to other available vmstorage nodes.
Note that the -disableReroutingOnUnavailable flag may pause data ingestion for long time when some vmstorage nodes are unavailable
for long time.
To pass bigger values to -storage.vminsertConnsShutdownDuration (available from v1.95.0)
command-line flag at vmstorage nodes.In this case vmstorage increases the interval between gradual closing of vminsert connections during graceful shutdown.
This reduces data ingestion slowdown during rollout restarts.
Make sure that the -storage.vminsertConnsShutdownDuration is smaller than the graceful shutdown timeout configured at the system which manages vmstorage
(e.g. Docker, Kubernetes, systemd, etc.). Otherwise the system may kill vmstorage node before it finishes gradual closing of vminsert connections.
VictoriaMetrics uses lower amounts of CPU, RAM and storage space on production workloads compared to competing solutions (Prometheus, Thanos, Cortex, TimescaleDB, InfluxDB, QuestDB, M3DB) according to our case studies.
Each node type - vminsert, vmselect and vmstorage - can run on the most suitable hardware. Cluster capacity scales linearly with the available resources. The needed amounts of CPU and RAM per each node type highly depends on the workload - the number of active time series, series churn rate, query types, query qps, etc. It is recommended setting up a test VictoriaMetrics cluster for your production workload and iteratively scaling per-node resources and the number of nodes per node type until the cluster becomes stable. It is recommended setting up monitoring for the cluster. It helps to determine bottlenecks in cluster setup. It is also recommended following the troubleshooting docs.
The needed storage space for the given retention (the retention is set via -retentionPeriod command-line flag at vmstorage) can be extrapolated from disk space usage in a test run. For example, if the storage space usage is 10GB after a day-long test run on a production workload, then it will need at least 10GB*100=1TB of disk space for -retentionPeriod=100d (100-days retention period). Storage space usage can be monitored with the official Grafana dashboard for VictoriaMetrics cluster.
It is recommended leaving the following amounts of spare resources:
50% of free RAM across all the node types for reducing the probability of OOM (out of memory) crashes and slowdowns during temporary spikes in workload.
50% of spare CPU across all the node types for reducing the probability of slowdowns during temporary spikes in workload.
At least 20% of free storage space at the directory pointed by -storageDataPath command-line flag at vmstorage nodes. See also -storage.minFreeDiskSpaceBytes command-line flag description for vmstorage.
Increase free storage space and -storage.minFreeDiskSpaceBytes to match at least the amount of data you plan to ingest in a calendar month: on each vmstorage pod, the monthly final deduplication process will temporarily need as much space as is used for the previous month’s data, before it can free up space. For example, if you have a 3 month retention period and you want to keep at least 10 % space free at all times, you could pick 35 % of your total space as value. When some of your vmstorage pods are in read-only mode, the remaining pods will have a higher share of the total data ingestion, and will therefore need more free space the next month.
Some capacity planning tips for VictoriaMetrics cluster:
The replication increases the amounts of needed resources for the cluster by up to N times where N is replication factor. This is because vminsert stores N copies of every ingested sample on distinct vmstorage nodes. These copies are de-duplicated by vmselect during querying. The most cost-efficient and performant solution for data durability is to rely on replicated durable persistent disks such as Google Compute persistent disks instead of using the replication at VictoriaMetrics level.
It is recommended to run a cluster with big number of small vmstorage nodes instead of a cluster with small number of big vmstorage nodes. This increases chances that the cluster remains available and stable when some of vmstorage nodes are temporarily unavailable during maintenance events such as upgrades, configuration changes or migrations. For example, when a cluster contains 10 vmstorage nodes and a single node becomes temporarily unavailable, then the workload on the remaining 9 nodes increases by 1/9=11%. When a cluster contains 3 vmstorage nodes and a single node becomes temporarily unavailable, then the workload on the remaining 2 nodes increases by 1/2=50%. The remaining vmstorage nodes may have no enough free capacity for handling the increased workload. In this case the cluster may become overloaded, which may result to decreased availability and stability.
Cluster capacity for active time series can be increased by increasing RAM and CPU resources per each vmstorage node or by adding new vmstorage nodes.
Query latency can be reduced by increasing CPU resources per each vmselect node, since each incoming query is processed by a single vmselect node. Performance for heavy queries scales with the number of available CPU cores at vmselect node, since vmselect processes time series referred by the query on all the available CPU cores.
If the cluster needs to process incoming queries at a high rate, then its capacity can be increased by adding more vmselect nodes, so incoming queries could be spread among bigger number of vmselect nodes.
By default vminsert compresses the data it sends to vmstorage in order to reduce network bandwidth usage. The compression takes additional CPU resources at vminsert. If vminsert nodes have limited CPU, then the compression can be disabled by passing -rpc.disableCompression command-line flag at vminsert nodes.
By default vmstorage compresses the data it sends to vmselect during queries in order to reduce network bandwidth usage. The compression takes additional CPU resources at vmstorage. If vmstorage nodes have limited CPU, then the compression can be disabled by passing -rpc.disableCompression command-line flag at vmstorage nodes.
By default, cluster components of VictoriaMetrics are tuned for an optimal resource usage under typical workloads.
Some workloads may need fine-grained resource usage limits. In these cases the following command-line flags may be useful:
-memory.allowedPercent and -memory.allowedBytes limit the amounts of memory, which may be used for various internal caches
at all the cluster components of VictoriaMetrics - vminsert, vmselect and vmstorage. Note that VictoriaMetrics components may use more memory,
since these flags don’t limit additional memory, which may be needed on a per-query basis.
-search.maxMemoryPerQuery limits the amounts of memory, which can be used for processing a single query at vmselect node.
Queries, which need more memory, are rejected. Heavy queries, which select big number of time series,
may exceed the per-query memory limit by a small percent. The total memory limit for concurrently executed queries can be estimated
as -search.maxMemoryPerQuery multiplied by -search.maxConcurrentRequests.
-search.maxUniqueTimeseries at vmstorage component limits the number of unique time series a single query can find and process.
This means that the maximum memory usage and CPU usage a single query can use at vmstorage is proportional to -search.maxUniqueTimeseries.
By default, vmstorage calculates this limit automatically based on the available memory and the maximum number of concurrent read requests (see -search.maxConcurrentRequests).
The calculated limit will be printed during process start-up logs and exposed as vm_search_max_unique_timeseries metric.
-search.maxUniqueTimeseries at vmselect adjusts the limit with the same name at vmstorage. The limit cannot exceed the
value set in vmstorage if the -search.maxUniqueTimeseries flag is explicitly defined there. By default, vmselect doesn’t apply limit adjustments.
-search.maxQueryDuration at vmselect limits the duration of a single query. If the query takes longer than the given duration, then it is canceled.
This allows saving CPU and RAM at vmselect and vmstorage when executing unexpectedly heavy queries.
The limit can be overridden to a smaller value by passing timeout GET parameter.
-search.maxConcurrentRequests at vmselect and vmstorage limits the number of concurrent requests a single vmselect / vmstorage node can process.
Bigger number of concurrent requests usually require bigger amounts of memory at both vmselect and vmstorage.
For example, if a single query needs 100 MiB of additional memory during its execution, then 100 concurrent queries
may need 100 * 100 MiB = 10 GiB of additional memory. So it is better to limit the number of concurrent queries,
while pausing additional incoming queries if the concurrency limit is reached.
vmselect and vmstorage provides -search.maxQueueDuration command-line flag for limiting the maximum wait time for paused queries.
See also -search.maxMemoryPerQuery command-line flag at vmselect.
-search.maxQueueDuration at vmselect and vmstorage limits the maximum duration queries may wait for execution when -search.maxConcurrentRequests
concurrent queries are executed.
-search.ignoreExtraFiltersAtLabelsAPI at vmselect enables ignoring of match[], extra_filters[] and extra_label
query args at /api/v1/labels and
/api/v1/label/…/values.
This may be useful for reducing the load on vmstorage if the provided extra filters match too many time series.
The downside is that the endpoints can return labels and series, which do not match the provided extra filters.
-search.maxSamplesPerSeries at vmselect limits the number of raw samples the query can process per each time series.
vmselect processes raw samples sequentially per each found time series during the query. It unpacks raw samples on the selected time range
per each time series into memory and then applies the given rollup function.
The -search.maxSamplesPerSeries command-line flag allows limiting memory usage at vmselect in the case when the query is executed on a time range,
which contains hundreds of millions of raw samples per each located time series.
-search.maxSamplesPerQuery at vmselect limits the number of raw samples a single query can process. This allows limiting CPU usage at vmselect for heavy queries.
-search.maxResponseSeries at vmselect limits the number of time series a single query can return
from /api/v1/query and /api/v1/query_range.
-search.maxPointsPerTimeseries limits the number of calculated points, which can be returned per each
matching time series from range query.
-search.maxPointsSubqueryPerTimeseries limits the number of calculated points, which can be generated
per each matching time series during subquery evaluation.
-search.maxSeriesPerAggrFunc limits the number of time series, which can be generated by MetricsQL aggregate functions in a single query.
-search.maxSeries at vmselect limits the number of time series, which may be returned from
/api/v1/series.
This endpoint is used mostly by Grafana for auto-completion of metric names, label names and label values.
Queries to this endpoint may take big amounts of CPU time and memory at vmstorage and vmselect when the database contains
big number of unique time series because of high churn rate.
In this case it might be useful to set the -search.maxSeries to quite low value in order limit CPU and memory usage.
-search.maxDeleteSeries at vmselect limits the number of unique time
series that can be deleted by a single
/api/v1/admin/tsdb/delete_series
call. The duration is limited via -search.maxDeleteDuration flagAvailable from v1.110.0. Deleting too many time series may require big
amount of CPU and memory at vmstorage and this limit guards against unplanned resource usage spikes.
Also see How to delete time series section to
learn about different ways of deleting series.
-search.maxTSDBStatusTopNSeries at vmselect limits the number of unique time
series that can be queried with topN argument by a single
/api/v1/status/tsdb?topN=N
call.
-search.maxTagKeys at vmstorage limits the number of items, which may be returned from
/api/v1/labels. This endpoint is used mostly by Grafana
for auto-completion of label names. Queries to this endpoint may take big amounts of CPU time and memory at vmstorage and vmselect
when the database contains big number of unique time series because of high churn rate.
In this case it might be useful to set the -search.maxTagKeys to quite low value in order to limit CPU and memory usage.
See also -search.maxLabelsAPIDuration and -search.maxLabelsAPISeries.
-search.maxTagValues at vmstorage limits the number of items, which may be returned from
/api/v1/label/…/values. This endpoint is used mostly by Grafana
for auto-completion of label values. Queries to this endpoint may take big amounts of CPU time and memory at vmstorage and vmselect
when the database contains big number of unique time series because of high churn rate.
In this case it might be useful to set the -search.maxTagValues to quite low value in order to limit CPU and memory usage.
See also -search.maxLabelsAPIDuration and -search.maxLabelsAPISeries.
-search.maxLabelsAPISeries at vmselect limits the number of time series, which can be scanned
when performing /api/v1/labels or
/api/v1/label/…/values requests.
These endpoints are used mostly by Grafana for auto-completion of label names and label values. Queries to these endpoints may take big amounts of CPU time and memory
when the database contains big number of unique time series because of high churn rate.
In this case it might be useful to set the -search.maxLabelsAPISeries to quite low value in order to limit CPU and memory usage.
See also -search.maxLabelsAPIDuration and -search.ignoreExtraFiltersAtLabelsAPI.
-search.maxLabelsAPIDuration at vmselect limits the duration for requests to /api/v1/labels,
/api/v1/label/…/values
or /api/v1/series.
The limit can be overridden to a smaller value by passing timeout GET parameter.
These endpoints are used mostly by Grafana for auto-completion of label names and label values. Queries to these endpoints may take big amounts of CPU time and memory
when the database contains big number of unique time series because of high churn rate.
In this case it might be useful to set the -search.maxLabelsAPIDuration to quite low value in order to limit CPU and memory usage.
See also -search.maxLabelsAPISeries and -search.ignoreExtraFiltersAtLabelsAPI.
-search.maxFederateSeries at vmselect limits maximum number of time series, which can be returned via /federate API.
The duration of the /federate queries is limited via -search.maxQueryDuration flag. This option allows limiting memory usage.
-search.maxExportSeries at vmselect limits maximum number of time series, which can be returned from /api/v1/export* APIs.
The duration of the export queries is limited via -search.maxExportDuration flag. This option allows limiting memory usage.
-search.maxTSDBStatusSeries at vmselect limits maximum number of time series, which can be processed during the call to /api/v1/status/tsdb.
The duration of the status queries is limited via -search.maxStatusRequestDuration flag. This option allows limiting memory usage.
By default, VictoriaMetrics offloads replication to the underlying storage pointed by -storageDataPath such as Google compute persistent disk,
which guarantees data durability. VictoriaMetrics supports application-level replication if replicated durable persistent disks cannot be used for some reason.
The replication can be enabled by passing -replicationFactor=N command-line flag to vminsert. This instructs vminsert to store N copies for every ingested sample
on N distinct vmstorage nodes. This guarantees that all the stored data remains available for querying if up to N-1vmstorage nodes are unavailable.
Passing -replicationFactor=N command-line flag to vmselect instructs it to not mark responses as partial if less than -replicationFactor vmstorage nodes are unavailable during the query.
See cluster availability docs for details.
The cluster must contain at least 2*N-1vmstorage nodes, where N is replication factor, in order to maintain the given replication factor
for newly ingested data when N-1 of storage nodes are unavailable.
VictoriaMetrics stores timestamps with millisecond precision, so -dedup.minScrapeInterval=1ms command-line flag must be passed to vmselect nodes when the replication is enabled,
so they could de-duplicate replicated samples obtained from distinct vmstorage nodes during querying. If duplicate data is pushed to VictoriaMetrics
from identically configured vmagent instances or Prometheus instances, then the -dedup.minScrapeInterval must be set
to scrape_interval from scrape configs according to deduplication docs.
Note that the replication increases resource usage - CPU, RAM, disk space, network bandwidth - by up to -replicationFactor=N times, because vminsert stores N copies
of incoming data to distinct vmstorage nodes and vmselect needs to de-duplicate the replicated data obtained from vmstorage nodes during querying.
So it is more cost-effective to offload the replication to underlying replicated durable storage pointed by -storageDataPath
such as Google Compute Engine persistent disk, which is protected from data loss and data corruption.
It also provides consistently high performance and may be resized without downtime.
HDD-based persistent disks should be enough for the majority of use cases. It is recommended using durable replicated persistent volumes in Kubernetes.
Cluster version of VictoriaMetrics supports data deduplication in the same way as single-node version do.
See these docs for details. The only difference is that
deduplication can’t be guaranteed when samples and sample duplicates for the same time series end up on different
vmstorage nodes. This could happen in the following scenarios:
when adding/removing vmstorage nodes a new samples for time series will be re-routed to another vmstorage nodes;
when vmstorage nodes are temporarily unavailable (for instance, during their restart). Then new samples are re-routed to the remaining available vmstorage nodes;
when vmstorage node has no enough capacity for processing incoming data stream. Then vminsert re-routes new samples to other vmstorage nodes.
It is recommended to set the same-dedup.minScrapeInterval command-line flag value to both vmselect and vmstorage nodes
to ensure query results consistency, even if storage layer didn’t complete deduplication yet.
It is recommended performing periodical backups from instant snapshots
for protecting from user errors such as accidental data deletion.
The following steps must be performed for each vmstorage node for creating a backup:
Create an instant snapshot by navigating to /snapshot/create HTTP handler. It will create snapshot and return its name.
Archive the created snapshot from <-storageDataPath>/snapshots/<snapshot_name> folder using vmbackup.
The archival process doesn’t interfere with vmstorage work, so it may be performed at any suitable time.
Delete unused snapshots via /snapshot/delete?snapshot=<snapshot_name> or /snapshot/delete_all in order to free up occupied storage space.
There is no need in synchronizing backups among all the vmstorage nodes.
Restoring from backup:
Stop vmstorage node with kill -INT.
Restore data from backup using vmrestore into -storageDataPath directory.
VictoriaMetrics enterprise supports configuring multiple retentions for distinct sets of time series
by passing -retentionFilter command-line flag to vmstorage nodes. See these docs for details on this feature.
Additionally, enterprise version of VictoriaMetrics cluster supports multiple retentions for distinct sets of tenants
by specifying filters on vm_account_id and/or vm_project_id pseudo-labels in -retentionFilter command-line flag.
If the tenant doesn’t match specified -retentionFilter options, then the global -retentionPeriod is used for it.
For example, the following config sets retention to 1 day for tenants with accountID starting from 42,
then sets retention to 3 days for time series with label env="dev" or env="prod" from any tenant,
while the rest of tenants will have 4 weeks retention:
It is OK to mix filters on real labels with filters on vm_account_id and vm_project_id pseudo-labels.
For example, the following config sets retention to 5 days for time series with env="dev" label from tenantaccountID=5:
It is possible to downsample series, which belong to a particular tenant by using filters
on vm_account_id or vm_project_id pseudo-labels in -downsampling.period command-line flag. For example, the following config leaves the last sample per each minute for samples
older than one hour only for tenants with accountID equal to 12 and 42, while series for other tenants are dropped:
It is OK to mix filters on real labels with filters on vm_account_id and vm_project_id pseudo-labels.
For example, the following config instructs leaving the last sample per hour after 30 days for time series with env="dev" label from tenantaccountID=5:
The same flag value must be passed to both vmstorage and vmselect nodes. Configuring vmselect node with -downsampling.period
command-line flag makes query results more consistent, because vmselect uses the maximum configured downsampling interval
on the requested time range if this time range covers multiple downsampling levels.
For example, if -downsampling.period=30d:5m and the query requests the last 60 days of data, then vmselect
downsamples all the raw samples on the requested time range
using 5 minute interval. If -downsampling.period command-line flag isn’t set at vmselect,
then query results can be less consistent because of mixing raw and downsampled data.
for accessing vmalert UI through vmselect Web interface.
For accessing vmalerts UI through vmselect configure -vmalert.proxyURL flag and visit
http://<vmselect>:8481/select/<accountID>/prometheus/vmalert/ link.
-blockcache.missesBeforeCachingintThenumberofcachemissesbeforeputtingtheblockintocache.Highervaluesmayreduceindexdb/dataBlockscachesizeatthecostofhigherCPUanddiskreadusage(default2)-cacheExpireDurationdurationItemsareremovedfromin-memorycachesaftertheyaren't accessed for this duration. Lower values may reduce memory usage at the cost of higher CPU usage. See also -prevCacheRemovalPercent (default 30m0s)-cluster.tlsWhethertouseTLSforconnectionsto-storageNode.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/-cluster.tlsCAFilestringPathtoTLSCAfiletouseforverifyingcertificatesprovidedby-storageNodeif-cluster.tlsflagisset.BydefaultsystemCAisused.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/-cluster.tlsCertFilestringPathtoclient-sideTLScertificatefiletousewhenconnectingto-storageNodeif-cluster.tlsflagisset.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/-cluster.tlsInsecureSkipVerifyWhethertoskipverificationofTLScertificatesprovidedby-storageNodenodesif-cluster.tlsflagisset.NotethatdisabledTLScertificateverificationbreakssecurity.ThisflagisavailableonlyinVictoriaMetricsenterprise.Seehttps://docs.victoriametrics.com/enterprise/-cluster.tlsKeyFilestringPathtoclient-sideTLSkeyfiletousewhenconnectingto-storageNodeif-cluster.tlsflagisset.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/-clusternative.vminsertConnsShutdownDurationdurationThetimeneededforgradualclosingofupstreamvminsertconnectionsduringgracefulshutdown.BiggerdurationreducesspikesinCPU,RAManddiskIOloadontheremaininglower-levelclustersduringrollingrestart.Smallerdurationreducesthetimeneededtoclosealltheupstreamvminsertconnections,thusreducingthetimeforgracefulshutdown.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#improving-re-routing-performance-during-restart (default 25s)-clusternativeListenAddrstringTCPaddresstolistenfordatafromothervminsertnodesinmulti-levelclustersetup.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#multi-level-cluster-setup . Usually :8400 should be set to match default vmstorage port for vminsert. Disabled work if empty-csvTrimTimestampdurationTrimtimestampswhenimportingcsvdatatothisduration.Minimumpracticaldurationis1ms.Higherduration(i.e.1s)maybeusedforreducingdiskspaceusagefortimestampdata(default1ms)-datadog.maxInsertRequestSizesizeThemaximumsizeinbytesofasingleDataDogPOSTrequestto/datadog/api/v2/seriesSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default67108864)-datadog.sanitizeMetricNameSanitizemetricnamesfortheingestedDataDogdatatocomplywithDataDogbehaviourdescribedathttps://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics (default true)-denyQueryTracingWhethertodisabletheabilitytotracequeries.Seehttps://docs.victoriametrics.com/#query-tracing-disableReroutingWhethertodisablere-routingwhensomeofvmstoragenodesacceptincomingdataatslowerspeedcomparedtootherstoragenodes.Disabledre-routinglimitstheingestionratebytheslowestvmstoragenode.Ontheotherside,disabledre-routingminimizesthenumberofactivetimeseriesintheclusterduringrollingrestartsandduringspikesinserieschurnrate.Seealso-disableReroutingOnUnavailableand-dropSamplesOnOverload(defaulttrue)-disableReroutingOnUnavailableWhethertodisablere-routingwhensomeofvmstoragenodesareunavailable.Disabledre-routingstopsingestionwhensomestoragenodesareunavailable.Ontheotherside,disabledre-routingminimizesthenumberofactivetimeseriesintheclusterduringrollingrestartsandduringspikesinserieschurnrate.Seealso-disableRerouting-dropSamplesOnOverloadWhethertodropincomingsamplesifthedestinationvmstoragenodeisoverloadedand/orunavailable.Thisprioritizesclusteravailabilityoverconsistency,e.g.theclustercontinuesacceptingalltheingestedsamples,butsomeofthemmaybedroppedifvmstoragenodesaretemporarilyunavailableand/oroverloaded.Thedropofsampleshappensbeforethereplication,soit's not recommended to use this flag with -replicationFactor enabled.-enableTCP6WhethertoenableIPv6forlisteninganddialing.Bydefault,onlyIPv4TCPandUDPareused-envflag.enableWhethertoenablereadingflagsfromenvironmentvariablesinadditiontothecommandline.Commandlineflagvalueshavepriorityovervaluesfromenvironmentvars.Flagsarereadonlyfromthecommandlineifthisflagisn't set. See https://docs.victoriametrics.com/#environment-variables for more details-envflag.prefixstringPrefixforenvironmentvariablesif-envflag.enableisset-eulaDeprecated,pleaseuse-licenseor-licenseFileflagsinstead.Byspecifyingthisflag,youconfirmthatyouhaveanenterpriselicenseandaccepttheESAhttps://victoriametrics.com/legal/esa/.ThisflagisavailableonlyinEnterprisebinaries.Seehttps://docs.victoriametrics.com/enterprise/-filestream.disableFadviseWhethertodisablefadvise()syscallwhenreadinglargedatafiles.Thefadvise()syscallpreventsfromevictionofrecentlyaccesseddatafromOSpagecacheduringbackgroundmergesandbackups.InsomerarecasesitisbettertodisablethesyscallifitusestoomuchCPU-flagsAuthKeyvalueAuthkeyfor/flagsendpoint.ItmustbepassedviaauthKeyqueryarg.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-flagsAuthKey=file:///abs/path/to/fileor-flagsAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-flagsAuthKey=http://host/pathor-flagsAuthKey=https://host/path-fs.disableMmapWhethertousepread()insteadofmmap()forreadingdatafiles.Bydefault,mmap()isusedfor64-bitarchesandpread()isusedfor32-bitarches,sincetheycannotreaddatafilesbiggerthan2^32bytesinmemory.mmap()isusuallyfasterforreadingsmalldatachunksthanpread()-graphite.sanitizeMetricNameSanitizemetricnamesfortheingestedGraphitedata.Seehttps://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd-graphiteListenAddrstringTCPandUDPaddresstolistenforGraphiteplaintextdata.Usually:2003mustbeset.Doesn't work if empty. See also -graphiteListenAddr.useProxyProtocol-graphiteListenAddr.useProxyProtocolWhethertouseproxyprotocolforconnectionsacceptedat-graphiteListenAddr.Seehttps://www.haproxy.org/download/1.8/doc/proxy-protocol.txt-graphiteTrimTimestampdurationTrimtimestampsforGraphitedatatothisduration.Minimumpracticaldurationis1s.Higherduration(i.e.1m)maybeusedforreducingdiskspaceusagefortimestampdata(default1s)-http.connTimeoutdurationIncomingconnectionsto-httpListenAddrareclosedaftertheconfiguredtimeout.ThismayhelpevenlyspreadingloadamongaclusterofservicesbehindTCP-levelloadbalancer.Zerovaluedisablesclosingofincomingconnections(default2m0s)-http.disableResponseCompressionDisablecompressionofHTTPresponsestosaveCPUresources.Bydefault,compressionisenabledtosavenetworkbandwidth-http.header.cspstringValuefor'Content-Security-Policy'header,recommended:"default-src 'self'"-http.header.frameOptionsstringValuefor'X-Frame-Options'header-http.header.hstsstringValuefor'Strict-Transport-Security'header,recommended:'max-age=31536000; includeSubDomains'-http.idleConnTimeoutdurationTimeoutforincomingidlehttpconnections(default1m0s)-http.maxGracefulShutdownDurationdurationThemaximumdurationforagracefulshutdownoftheHTTPserver.Ahighlyloadedservermayrequireincreasedvalueforagracefulshutdown(default7s)-http.pathPrefixstringAnoptionalprefixtoaddtoallthepathshandledbyhttpserver.Forexample,if'-http.pathPrefix=/foo/bar'isset,thenallthehttprequestswillbehandledon'/foo/bar/*'paths.Thismaybeusefulforproxiedrequests.Seehttps://www.robustperception.io/using-external-urls-and-proxies-with-prometheus-http.shutdownDelaydurationOptionaldelaybeforehttpservershutdown.Duringthisdelay,theserverreturnsnon-OKresponsesfrom/healthpage,soloadbalancerscanroutenewrequeststootherservers-httpAuth.passwordvaluePasswordforHTTPserver's Basic Auth. The authentication is disabled if -httpAuth.username is emptyFlagvaluecanbereadfromthegivenfilewhenusing-httpAuth.password=file:///abs/path/to/fileor-httpAuth.password=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-httpAuth.password=http://host/pathor-httpAuth.password=https://host/path-httpAuth.usernamestringUsernameforHTTPserver's Basic Auth. The authentication is disabled if empty. See also -httpAuth.password-httpListenAddrarrayAddresstolistenforincominghttprequests.Seealso-httpListenAddr.useProxyProtocolSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-httpListenAddr.useProxyProtocolarrayWhethertouseproxyprotocolforconnectionsacceptedatthegiven-httpListenAddr.Seehttps://www.haproxy.org/download/1.8/doc/proxy-protocol.txt.Withenabledproxyprotocolhttpservercannotserveregular/metricsendpoint.Use-pushmetrics.urlformetricspushingSupportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-import.maxLineLensizeThemaximumlengthinbytesofasinglelineacceptedby/api/v1/import;thelinelengthcanbelimitedwith'max_rows_per_line'queryargpassedto/api/v1/exportSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default10485760)-influx.databaseNamesarrayComma-separatedlistofdatabasenamestoreturnfrom/queryand/influx/queryAPI.ThiscanbeneededforacceptingdatafromTelegrafpluginssuchashttps://github.com/fangli/fluent-plugin-influxdbSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-influx.forceStreamModeboolForcestreammodeparsingforingesteddata.Seehttps://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf.-influx.maxLineSizesizeThemaximumsizeinbytesforasingleInfluxDBlineduringparsing.Applicableforstreammodeonly.Seehttps://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegrafSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default262144)-influx.maxRequestSizesizeThemaximumsizeinbytesofasingleInfluxDBrequest.Applicableforbatchmodeonly.Seehttps://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegrafSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default67108864)-influxDBLabelstringDefaultlabelfortheDBnamesentover'?db={db_name}'queryparameter(default"db")-influxListenAddrstringTCPandUDPaddresstolistenforInfluxDBlineprotocoldata.Usually:8089mustbeset.Doesn't work if empty. This flag isn'tneededwheningestingdataoverHTTP-justsendittohttp://<victoriametrics>:8428/write.Seealso-influxListenAddr.useProxyProtocol-influxListenAddr.useProxyProtocolWhethertouseproxyprotocolforconnectionsacceptedat-influxListenAddr.Seehttps://www.haproxy.org/download/1.8/doc/proxy-protocol.txt-influxMeasurementFieldSeparatorstringSeparatorfor'{measurement}{separator}{field_name}'metricnamewheninsertedviaInfluxDBlineprotocol(default"_")-influxSkipMeasurementUses'{field_name}'asametricnamewhileignoring'{measurement}'and'-influxMeasurementFieldSeparator'-influxSkipSingleFieldUses'{measurement}'insteadof'{measurement}{separator}{field_name}'formetricnameifInfluxDBlinecontainsonlyasinglefield-influxTrimTimestampdurationTrimtimestampsforInfluxDBlineprotocoldatatothisduration.Minimumpracticaldurationis1ms.Higherduration(i.e.1s)maybeusedforreducingdiskspaceusagefortimestampdata(default1ms)-insert.maxQueueDurationdurationThemaximumdurationtowaitinthequeuewhen-maxConcurrentInsertsconcurrentinsertrequestsareexecuted(default1m0s)-internStringCacheExpireDurationdurationTheexpirydurationforcachesforinternedstrings.Seehttps://en.wikipedia.org/wiki/String_interning.Seealso-internStringMaxLenand-internStringDisableCache(default6m0s)-internStringDisableCacheWhethertodisablecachesforinternedstrings.ThismayreducememoryusageatthecostofhigherCPUusage.Seehttps://en.wikipedia.org/wiki/String_interning.Seealso-internStringCacheExpireDurationand-internStringMaxLen-internStringMaxLenintThemaximumlengthforstringstointern.AlowerlimitmaysavememoryatthecostofhigherCPUusage.Seehttps://en.wikipedia.org/wiki/String_interning.Seealso-internStringDisableCacheand-internStringCacheExpireDuration(default500)-licensestringLicensekeyforVictoriaMetricsEnterprise.Seehttps://victoriametrics.com/products/enterprise/.TrialEnterpriselicensecanbeobtainedfromhttps://victoriametrics.com/products/enterprise/trial/.ThisflagisavailableonlyinEnterprisebinaries.Thelicensekeycanbealsopassedviafilespecifiedby-licenseFilecommand-lineflag-license.forceOfflineWhethertoenableofflineverificationforVictoriaMetricsEnterpriselicensekey,whichhasbeenpassedeithervia-licenseorvia-licenseFilecommand-lineflag.Theissuedlicensekeymustsupportofflineverificationfeature.Contactinfo@victoriametrics.comifyouneedofflinelicenseverification.ThisflagisavailableonlyinEnterprisebinaries-licenseFilestringPathtofilewithlicensekeyforVictoriaMetricsEnterprise.Seehttps://victoriametrics.com/products/enterprise/.TrialEnterpriselicensecanbeobtainedfromhttps://victoriametrics.com/products/enterprise/trial/.ThisflagisavailableonlyinEnterprisebinaries.Thelicensekeycanbealsopassedinlinevia-licensecommand-lineflag-loggerDisableTimestampsWhethertodisablewritingtimestampsinlogs-loggerErrorsPerSecondLimitintPer-secondlimitonthenumberofERRORmessages.Ifmorethanthegivennumberoferrorsareemittedpersecond,theremainingerrorsaresuppressed.Zerovaluesdisabletheratelimit-loggerFormatstringFormatforlogs.Possiblevalues:default,json(default"default")-loggerJSONFieldsstringAllowsrenamingfieldsinJSONformattedlogs.Example:"ts:timestamp,msg:message"renames"ts"to"timestamp"and"msg"to"message".Supportedfields:ts,level,caller,msg-loggerLevelstringMinimumleveloferrorstolog.Possiblevalues:INFO,WARN,ERROR,FATAL,PANIC(default"INFO")-loggerMaxArgLenintThemaximumlengthofasingleloggedargument.Longerargumentsarereplacedwith'arg_start..arg_end',where'arg_start'and'arg_end'isprefixandsuffixoftheargwiththelengthnotexceeding-loggerMaxArgLen/2(default1000)-loggerOutputstringOutputforthelogs.Supportedvalues:stderr,stdout(default"stderr")-loggerTimezonestringTimezonetousefortimestampsinlogs.TimezonemustbeavalidIANATimeZone.Forexample:America/New_York,Europe/Berlin,Etc/GMT+3orLocal(default"UTC")-loggerWarnsPerSecondLimitintPer-secondlimitonthenumberofWARNmessages.Ifmorethanthegivennumberofwarnsareemittedpersecond,thentheremainingwarnsaresuppressed.Zerovaluesdisabletheratelimit-maxConcurrentInsertsintThemaximumnumberofconcurrentinsertrequests.Sethighervaluewhenclientssenddataoverslownetworks.DefaultvaluedependsonthenumberofavailableCPUcores.Itshouldworkfineinmostcasessinceitminimizesresourceusage.Seealso-insert.maxQueueDuration-maxInsertRequestSizesizeThemaximumsizeinbytesofasinglePrometheusremote_writeAPIrequestSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default33554432)-maxLabelValueLenintThemaximumlengthoflabelvaluesintheacceptedtimeseries.Series,withlongerlabelvaluesareignored.Inthiscasethevm_rows_ignored_total{reason="too_long_label_value"}metricat/metricspageisincremented(default4096)-maxLabelsPerTimeseriesintThemaximumnumberoflabelsacceptedpertimeseries.Superfluouslabelsareignored.Inthiscasethevm_rows_ignored_total{reason="too_many_labels"}metricat/metricspageisincremented(default30)-memory.allowedBytessizeAllowedsizeofsystemmemoryVictoriaMetricscachesmayoccupy.Thisoptionoverrides-memory.allowedPercentifsettoanon-zerovalue.ToolowavaluemayincreasethecachemissrateusuallyresultinginhigherCPUanddiskIOusage.ToohighavaluemayevicttoomuchdatafromtheOSpagecacheresultinginhigherdiskIOusageSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-memory.allowedPercentfloatAllowedpercentofsystemmemoryVictoriaMetricscachesmayoccupy.Seealso-memory.allowedBytes.ToolowavaluemayincreasecachemissrateusuallyresultinginhigherCPUanddiskIOusage.ToohighavaluemayevicttoomuchdatafromtheOSpagecachewhichwillresultinhigherdiskIOusage(default60)-metricNamesStatsResetAuthKeyvalueAuthKeyforresettingmetricnamesusagecachevia/api/v1/admin/status/metric_names_stats/reset.Itoverrides-httpAuth.*Seehttps://docs.victoriametrics.com/#track-ingested-metrics-usageFlagvaluecanbereadfromthegivenfilewhenusing-metricNamesStatsResetAuthKey=file:///abs/path/to/fileor-metricNamesStatsResetAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-metricNamesStatsResetAuthKey=http://host/pathor-metricNamesStatsResetAuthKey=https://host/path-metrics.exposeMetadataWhethertoexposeTYPEandHELPmetadataatthe/metricspage,whichisexposedat-httpListenAddr.Themetadatamaybeneededwhenthe/metricspageisconsumedbysystems,whichrequirethisinformation.Forexample,ManagedPrometheusinGoogleCloud-https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting#missing-metric-type-metricsAuthKeyvalueAuthkeyfor/metricsendpoint.ItmustbepassedviaauthKeyqueryarg.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-metricsAuthKey=file:///abs/path/to/fileor-metricsAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-metricsAuthKey=http://host/pathor-metricsAuthKey=https://host/path-mtlsarrayWhethertorequirevalidclientcertificateforhttpsrequeststothecorresponding-httpListenAddr.Thisflagworksonlyif-tlsflagisset.Seealso-mtlsCAFile.ThisflagisavailableonlyinEnterprisebinaries.Seehttps://docs.victoriametrics.com/enterprise/Supportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-mtlsCAFilearrayOptionalpathtoTLSRootCAforverifyingclientcertificatesatthecorresponding-httpListenAddrwhen-mtlsisenabled.BydefaultthehostsystemTLSRootCAisusedforclientcertificateverification.ThisflagisavailableonlyinEnterprisebinaries.Seehttps://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-newrelic.maxInsertRequestSizesizeThemaximumsizeinbytesofasingleNewRelicrequestto/newrelic/infra/v2/metrics/events/bulkSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default67108864)-opentelemetry.usePrometheusNamingWhethertoconvertmetricnamesandlabelsintoPrometheus-compatibleformatforthemetricsingestedviaOpenTelemetryprotocol;seehttps://docs.victoriametrics.com/#sending-data-via-opentelemetry-opentsdbHTTPListenAddrstringTCPaddresstolistenforOpenTSDBHTTPputrequests.Usually:4242mustbeset.Doesn't work if empty. See also -opentsdbHTTPListenAddr.useProxyProtocol-opentsdbHTTPListenAddr.useProxyProtocolWhethertouseproxyprotocolforconnectionsacceptedat-opentsdbHTTPListenAddr.Seehttps://www.haproxy.org/download/1.8/doc/proxy-protocol.txt-opentsdbListenAddrstringTCPandUDPaddresstolistenforOpenTSDBmetrics.TelnetputmessagesandHTTP/api/putmessagesaresimultaneouslyservedonTCPport.Usually:4242mustbeset.Doesn't work if empty. See also -opentsdbListenAddr.useProxyProtocol-opentsdbListenAddr.useProxyProtocolWhethertouseproxyprotocolforconnectionsacceptedat-opentsdbListenAddr.Seehttps://www.haproxy.org/download/1.8/doc/proxy-protocol.txt-opentsdbTrimTimestampdurationTrimtimestampsforOpenTSDB'telnet put'datatothisduration.Minimumpracticaldurationis1s.Higherduration(i.e.1m)maybeusedforreducingdiskspaceusagefortimestampdata(default1s)-opentsdbhttp.maxInsertRequestSizesizeThemaximumsizeofOpenTSDBHTTPputrequestSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default33554432)-opentsdbhttpTrimTimestampdurationTrimtimestampsforOpenTSDBHTTPdatatothisduration.Minimumpracticaldurationis1ms.Higherduration(i.e.1s)maybeusedforreducingdiskspaceusagefortimestampdata(default1ms)-pprofAuthKeyvalueAuthkeyfor/debug/pprof/*endpoints.ItmustbepassedviaauthKeyqueryarg.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-pprofAuthKey=file:///abs/path/to/fileor-pprofAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-pprofAuthKey=http://host/pathor-pprofAuthKey=https://host/path-prevCacheRemovalPercentfloatItemsinthepreviouscachesareremovedwhenthepercentofrequestsitservesbecomeslowerthanthisvalue.HighervaluesreducememoryusageatthecostofhigherCPUusage.Seealso-cacheExpireDuration(default0.1)-pushmetrics.disableCompressionWhethertodisablerequestbodycompressionwhenpushingmetricstoevery-pushmetrics.url-pushmetrics.extraLabelarrayOptionallabelstoaddtometricspushedtoevery-pushmetrics.url.Forexample,-pushmetrics.extraLabel='instance="foo"'addsinstance="foo"labeltoallthemetricspushedtoevery-pushmetrics.urlSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-pushmetrics.headerarrayOptionalHTTPrequestheadertosendtoevery-pushmetrics.url.Forexample,-pushmetrics.header='Authorization: Basic foobar'adds'Authorization: Basic foobar'headertoeveryrequesttoevery-pushmetrics.urlSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-pushmetrics.intervaldurationIntervalforpushingmetricstoevery-pushmetrics.url(default10s)-pushmetrics.urlarrayOptionalURLtopushmetricsexposedat/metricspage.Seehttps://docs.victoriametrics.com/#push-metrics . By default, metrics exposed at /metrics page aren't pushed to any remote storageSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-relabelConfigstringOptionalpathtoafilewithrelabelingrules,whichareappliedtoalltheingestedmetrics.Thepathcanpointeithertolocalfileortohttpurl.Seehttps://docs.victoriametrics.com/#relabeling for details. The config is reloaded on SIGHUP signal-relabelConfigCheckIntervaldurationIntervalforcheckingforchangesin'-relabelConfig'file.Bydefaultthecheckingisdisabled.SendSIGHUPsignalinordertoforceconfigcheckforchanges-replicationFactorintReplicationfactorfortheingesteddata,i.e.howmanycopiestomakeamongdistinct-storageNodeinstances.Notethatvmselectmustrunwith-dedup.minScrapeInterval=1msfordatade-duplicationwhenreplicationFactorisgreaterthan1.Highervaluesfor-dedup.minScrapeIntervalatvmselectisOK(default1)-rpc.disableCompressionWhethertodisablecompressionforthedatasentfromvminserttovmstorage.ThisreducesCPUusageatthecostofhighernetworkbandwidthusage-sortLabelsWhethertosortlabelsforincomingsamplesbeforewritingthemtostorage.Thismaybeneededforreducingmemoryusageatstoragewhentheorderoflabelsinincomingsamplesisrandom.Forexample,ifm{k1="v1",k2="v2"}maybesentasm{k2="v2",k1="v1"}.Enabledsortingforlabelscanslowdowningestionperformanceabit-storageNodearrayComma-separatedaddressesofvmstoragenodes;usage:-storageNode=vmstorage-host1,...,vmstorage-hostN.EnterpriseversionofVictoriaMetricssupportsautomaticdiscoveryofvmstorageaddressesviaDNSSRVrecords.Forexample,-storageNode=srv+vmstorage.addrs.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#automatic-vmstorage-discoverySupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-storageNode.discoveryIntervaldurationIntervalforrefreshing-storageNodelistbehindDNSSRVrecords.Theminimumsupportedintervalis1s.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#automatic-vmstorage-discovery . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ (default 2s)-storageNode.filterstringAnoptionalregexpfilterfordiscovered-storageNodeaddressesaccordingtohttps://docs.victoriametrics.com/cluster-victoriametrics/#automatic-vmstorage-discovery. Discovered addresses matching the filter are retained, while other addresses are ignored. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/-tlsarrayWhethertoenableTLSforincomingHTTPrequestsatthegiven-httpListenAddr(akahttps).-tlsCertFileand-tlsKeyFilemustbesetif-tlsisset.Seealso-mtlsSupportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-tlsAutocertCacheDirstringDirectorytostoreTLScertificatesissuedviaLet's Encrypt. Certificates are lost on restarts if this flag isn'tset.ThisflagisavailableonlyinEnterprisebinaries.Seehttps://docs.victoriametrics.com/enterprise/-tlsAutocertEmailstringContactemailfortheissuedLet's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/-tlsAutocertHostsarrayOptionalhostnamesforautomaticissuingofLet's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-tlsCertFilearrayPathtofilewithTLScertificateforthecorresponding-httpListenAddrif-tlsisset.PreferECDSAcertsinsteadofRSAcertsasRSAcertsareslower.Theprovidedcertificatefileisautomaticallyre-readeverysecond,soitcanbedynamicallyupdated.Seealso-tlsAutocertHostsSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-tlsCipherSuitesarrayOptionallistofTLSciphersuitesforincomingrequestsoverHTTPSif-tlsisset.Seethelistofsupportedciphersuitesathttps://pkg.go.dev/crypto/tls#pkg-constantsSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-tlsKeyFilearrayPathtofilewithTLSkeyforthecorresponding-httpListenAddrif-tlsisset.Theprovidedkeyfileisautomaticallyre-readeverysecond,soitcanbedynamicallyupdated.Seealso-tlsAutocertHostsSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-tlsMinVersionarrayOptionalminimumTLSversiontouseforthecorresponding-httpListenAddrif-tlsisset.Supportedvalues:TLS10,TLS11,TLS12,TLS13Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-usePromCompatibleNamingWhethertoreplacecharactersunsupportedbyPrometheuswithunderscoresintheingestedmetricnamesandlabelnames.Forexample,foo.bar{a.b='c'}istransformedintofoo_bar{a_b='c'}duringdataingestionifthisflagisset.Seehttps://prometheus.io/docs/concepts/data_model/#metric-names-and-labels-versionShowVictoriaMetricsversion-vmstorageDialTimeoutdurationTimeoutforestablishingRPCconnectionsfromvminserttovmstorage.Seealso-vmstorageUserTimeout(default3s)-vmstorageUserTimeoutdurationNetworktimeoutforRPCconnectionsfromvminserttovmstorage(Linuxonly).Lowervaluesspeedupre-reroutingrecoverywhensomeofvmstoragenodesbecomeunavailablebecauseofnetworkingissues.ReadmoreaboutTCP_USER_TIMEOUTathttps://blog.cloudflare.com/when-tcp-sockets-refuse-to-die/.Seealso-vmstorageDialTimeout(default3s)
-blockcache.missesBeforeCachingintThenumberofcachemissesbeforeputtingtheblockintocache.Highervaluesmayreduceindexdb/dataBlockscachesizeatthecostofhigherCPUanddiskreadusage(default2)-cacheDataPathstringPathtodirectoryforcachefilesandtemporaryqueryresults.Bydefault,thecachewon't be persisted, and temporary query results will be placed under /tmp/searchResults. If set, the cache will be persisted under cacheDataPath/rollupResult, and temporary query results will be placed under cacheDataPath/tmp/searchResults.-cacheExpireDurationdurationItemsareremovedfromin-memorycachesaftertheyaren't accessed for this duration. Lower values may reduce memory usage at the cost of higher CPU usage. See also -prevCacheRemovalPercent (default 30m0s)-cluster.tlsWhethertouseTLSforconnectionsto-storageNode.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/-cluster.tlsCAFilestringPathtoTLSCAfiletouseforverifyingcertificatesprovidedby-storageNodeif-cluster.tlsflagisset.BydefaultsystemCAisused.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/-cluster.tlsCertFilestringPathtoclient-sideTLScertificatefiletousewhenconnectingto-storageNodeif-cluster.tlsflagisset.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/-cluster.tlsInsecureSkipVerifyWhethertoskipverificationofTLScertificatesprovidedby-storageNodenodesif-cluster.tlsflagisset.NotethatdisabledTLScertificateverificationbreakssecurity.ThisflagisavailableonlyinVictoriaMetricsenterprise.Seehttps://docs.victoriametrics.com/enterprise/-cluster.tlsKeyFilestringPathtoclient-sideTLSkeyfiletousewhenconnectingto-storageNodeif-cluster.tlsflagisset.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/-clusternative.disableCompressionWhethertodisablecompressionofthedatasenttovmselectvia-clusternativeListenAddr.ThisreducesCPUusageatthecostofhighernetworkbandwidthusage-clusternative.maxConcurrentRequestsintThemaximumnumberofconcurrentvmselectrequeststheservercanprocessat-clusternativeListenAddr.Itshouldn't be high, since a single request usually saturates a CPU core at the underlying vmstorage nodes, and many concurrently executed requests may require high amounts of memory. See also -clusternative.maxQueueDuration-clusternative.maxQueueDurationdurationThemaximumtimetheincomingqueryto-clusternativeListenAddrwaitsforexecutionwhen-clusternative.maxConcurrentRequestslimitisreached(default10s)-clusternative.maxTagKeysintThemaximumnumberoftagkeysreturnedpersearchat-clusternativeListenAddr(default100000)-clusternative.maxTagValueSuffixesPerSearchintThemaximumnumberoftagvaluesuffixesreturnedfrom/metrics/findat-clusternativeListenAddr(default100000)-clusternative.maxTagValuesintThemaximumnumberoftagvaluesreturnedpersearchat-clusternativeListenAddr(default100000)-clusternative.tlsWhethertouseTLSwhenacceptingconnectionsat-clusternativeListenAddr.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/-clusternative.tlsCAFilestringPathtoTLSCAfiletouseforverifyingcertificatesprovidedbyvmselect,whichconnectsat-clusternativeListenAddrif-clusternative.tlsflagisset.BydefaultsystemCAisused.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/-clusternative.tlsCertFilestringPathtoserver-sideTLScertificatefiletousewhenacceptingconnectionsat-clusternativeListenAddrif-clusternative.tlsflagisset.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/-clusternative.tlsCipherSuitesarrayOptionallistofTLSciphersuitesusedforconnectionsat-clusternativeListenAddrif-clusternative.tlsflagisset.Seethelistofsupportedciphersuitesathttps://pkg.go.dev/crypto/tls#pkg-constants . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-clusternative.tlsInsecureSkipVerifyWhethertoskipverificationofTLScertificatesprovidedbyvmselect,whichconnectsto-clusternativeListenAddrif-clusternative.tlsflagisset.NotethatdisabledTLScertificateverificationbreakssecurity.ThisflagisavailableonlyinVictoriaMetricsenterprise.Seehttps://docs.victoriametrics.com/enterprise/-clusternative.tlsKeyFilestringPathtoserver-sideTLSkeyfiletousewhenacceptingconnectionsat-clusternativeListenAddrif-clusternative.tlsflagisset.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/-clusternativeListenAddrstringTCPaddresstolistenforrequestsfromothervmselectnodesinmulti-levelclustersetup.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#multi-level-cluster-setup . Usually :8401 should be set to match default vmstorage port for vmselect. Disabled work if empty-dedup.minScrapeIntervaldurationLeaveonlythelastsampleineverytimeseriespereachdiscreteintervalequalto-dedup.minScrapeInterval>0.Seehttps://docs.victoriametrics.com/#deduplication for details-deleteAuthKeyvalueauthKeyformetrics' deletion via /prometheus/api/v1/admin/tsdb/delete_series and /graphite/tags/delSeries. It could be passed via authKey query arg.Flagvaluecanbereadfromthegivenfilewhenusing-deleteAuthKey=file:///abs/path/to/fileor-deleteAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-deleteAuthKey=http://host/pathor-deleteAuthKey=https://host/path-denyQueryTracingWhethertodisabletheabilitytotracequeries.Seehttps://docs.victoriametrics.com/#query-tracing-downsampling.periodarrayComma-separateddownsamplingperiodsintheformat'offset:period'.Forexample,'30d:10m'instructstoleaveasinglesampleper10minutesforsamplesolderthan30days.Whensettingmultipledownsamplingperiods,itisnecessaryfortheperiodstobemultiplesofeachother.Seehttps://docs.victoriametrics.com/#downsampling for details. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-enableTCP6WhethertoenableIPv6forlisteninganddialing.Bydefault,onlyIPv4TCPandUDPareused-envflag.enableWhethertoenablereadingflagsfromenvironmentvariablesinadditiontothecommandline.Commandlineflagvalueshavepriorityovervaluesfromenvironmentvars.Flagsarereadonlyfromthecommandlineifthisflagisn't set. See https://docs.victoriametrics.com/#environment-variables for more details-envflag.prefixstringPrefixforenvironmentvariablesif-envflag.enableisset-eulaDeprecated,pleaseuse-licenseor-licenseFileflagsinstead.Byspecifyingthisflag,youconfirmthatyouhaveanenterpriselicenseandaccepttheESAhttps://victoriametrics.com/legal/esa/.ThisflagisavailableonlyinEnterprisebinaries.Seehttps://docs.victoriametrics.com/enterprise/-filestream.disableFadviseWhethertodisablefadvise()syscallwhenreadinglargedatafiles.Thefadvise()syscallpreventsfromevictionofrecentlyaccesseddatafromOSpagecacheduringbackgroundmergesandbackups.InsomerarecasesitisbettertodisablethesyscallifitusestoomuchCPU-flagsAuthKeyvalueAuthkeyfor/flagsendpoint.ItmustbepassedviaauthKeyqueryarg.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-flagsAuthKey=file:///abs/path/to/fileor-flagsAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-flagsAuthKey=http://host/pathor-flagsAuthKey=https://host/path-fs.disableMmapWhethertousepread()insteadofmmap()forreadingdatafiles.Bydefault,mmap()isusedfor64-bitarchesandpread()isusedfor32-bitarches,sincetheycannotreaddatafilesbiggerthan2^32bytesinmemory.mmap()isusuallyfasterforreadingsmalldatachunksthanpread()-globalReplicationFactorintHowmanycopiesofeveryingestedsampleisavailableacrossvmstoragegroups.vmselectcontinuesreturningfullresponseswhenuptoglobalReplicationFactor-1vmstoragegroupsaretemporarilyunavailable.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#vmstorage-groups-at-vmselect . See also -replicationFactor (default 1)-http.connTimeoutdurationIncomingconnectionsto-httpListenAddrareclosedaftertheconfiguredtimeout.ThismayhelpevenlyspreadingloadamongaclusterofservicesbehindTCP-levelloadbalancer.Zerovaluedisablesclosingofincomingconnections(default2m0s)-http.disableResponseCompressionDisablecompressionofHTTPresponsestosaveCPUresources.Bydefault,compressionisenabledtosavenetworkbandwidth-http.header.cspstringValuefor'Content-Security-Policy'header,recommended:"default-src 'self'"-http.header.frameOptionsstringValuefor'X-Frame-Options'header-http.header.hstsstringValuefor'Strict-Transport-Security'header,recommended:'max-age=31536000; includeSubDomains'-http.idleConnTimeoutdurationTimeoutforincomingidlehttpconnections(default1m0s)-http.maxGracefulShutdownDurationdurationThemaximumdurationforagracefulshutdownoftheHTTPserver.Ahighlyloadedservermayrequireincreasedvalueforagracefulshutdown(default7s)-http.pathPrefixstringAnoptionalprefixtoaddtoallthepathshandledbyhttpserver.Forexample,if'-http.pathPrefix=/foo/bar'isset,thenallthehttprequestswillbehandledon'/foo/bar/*'paths.Thismaybeusefulforproxiedrequests.Seehttps://www.robustperception.io/using-external-urls-and-proxies-with-prometheus-http.shutdownDelaydurationOptionaldelaybeforehttpservershutdown.Duringthisdelay,theserverreturnsnon-OKresponsesfrom/healthpage,soloadbalancerscanroutenewrequeststootherservers-httpAuth.passwordvaluePasswordforHTTPserver's Basic Auth. The authentication is disabled if -httpAuth.username is emptyFlagvaluecanbereadfromthegivenfilewhenusing-httpAuth.password=file:///abs/path/to/fileor-httpAuth.password=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-httpAuth.password=http://host/pathor-httpAuth.password=https://host/path-httpAuth.usernamestringUsernameforHTTPserver's Basic Auth. The authentication is disabled if empty. See also -httpAuth.password-httpListenAddrarrayAddresstolistenforincominghttprequests.Seealso-httpListenAddr.useProxyProtocolSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-httpListenAddr.useProxyProtocolarrayWhethertouseproxyprotocolforconnectionsacceptedatthegiven-httpListenAddr.Seehttps://www.haproxy.org/download/1.8/doc/proxy-protocol.txt.Withenabledproxyprotocolhttpservercannotserveregular/metricsendpoint.Use-pushmetrics.urlformetricspushingSupportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-internStringCacheExpireDurationdurationTheexpirydurationforcachesforinternedstrings.Seehttps://en.wikipedia.org/wiki/String_interning.Seealso-internStringMaxLenand-internStringDisableCache(default6m0s)-internStringDisableCacheWhethertodisablecachesforinternedstrings.ThismayreducememoryusageatthecostofhigherCPUusage.Seehttps://en.wikipedia.org/wiki/String_interning.Seealso-internStringCacheExpireDurationand-internStringMaxLen-internStringMaxLenintThemaximumlengthforstringstointern.AlowerlimitmaysavememoryatthecostofhigherCPUusage.Seehttps://en.wikipedia.org/wiki/String_interning.Seealso-internStringDisableCacheand-internStringCacheExpireDuration(default500)-licensestringLicensekeyforVictoriaMetricsEnterprise.Seehttps://victoriametrics.com/products/enterprise/.TrialEnterpriselicensecanbeobtainedfromhttps://victoriametrics.com/products/enterprise/trial/.ThisflagisavailableonlyinEnterprisebinaries.Thelicensekeycanbealsopassedviafilespecifiedby-licenseFilecommand-lineflag-license.forceOfflineWhethertoenableofflineverificationforVictoriaMetricsEnterpriselicensekey,whichhasbeenpassedeithervia-licenseorvia-licenseFilecommand-lineflag.Theissuedlicensekeymustsupportofflineverificationfeature.Contactinfo@victoriametrics.comifyouneedofflinelicenseverification.ThisflagisavailableonlyinEnterprisebinaries-licenseFilestringPathtofilewithlicensekeyforVictoriaMetricsEnterprise.Seehttps://victoriametrics.com/products/enterprise/.TrialEnterpriselicensecanbeobtainedfromhttps://victoriametrics.com/products/enterprise/trial/.ThisflagisavailableonlyinEnterprisebinaries.Thelicensekeycanbealsopassedinlinevia-licensecommand-lineflag-loggerDisableTimestampsWhethertodisablewritingtimestampsinlogs-loggerErrorsPerSecondLimitintPer-secondlimitonthenumberofERRORmessages.Ifmorethanthegivennumberoferrorsareemittedpersecond,theremainingerrorsaresuppressed.Zerovaluesdisabletheratelimit-loggerFormatstringFormatforlogs.Possiblevalues:default,json(default"default")-loggerJSONFieldsstringAllowsrenamingfieldsinJSONformattedlogs.Example:"ts:timestamp,msg:message"renames"ts"to"timestamp"and"msg"to"message".Supportedfields:ts,level,caller,msg-loggerLevelstringMinimumleveloferrorstolog.Possiblevalues:INFO,WARN,ERROR,FATAL,PANIC(default"INFO")-loggerMaxArgLenintThemaximumlengthofasingleloggedargument.Longerargumentsarereplacedwith'arg_start..arg_end',where'arg_start'and'arg_end'isprefixandsuffixoftheargwiththelengthnotexceeding-loggerMaxArgLen/2(default1000)-loggerOutputstringOutputforthelogs.Supportedvalues:stderr,stdout(default"stderr")-loggerTimezonestringTimezonetousefortimestampsinlogs.TimezonemustbeavalidIANATimeZone.Forexample:America/New_York,Europe/Berlin,Etc/GMT+3orLocal(default"UTC")-loggerWarnsPerSecondLimitintPer-secondlimitonthenumberofWARNmessages.Ifmorethanthegivennumberofwarnsareemittedpersecond,thentheremainingwarnsaresuppressed.Zerovaluesdisabletheratelimit-memory.allowedBytessizeAllowedsizeofsystemmemoryVictoriaMetricscachesmayoccupy.Thisoptionoverrides-memory.allowedPercentifsettoanon-zerovalue.ToolowavaluemayincreasethecachemissrateusuallyresultinginhigherCPUanddiskIOusage.ToohighavaluemayevicttoomuchdatafromtheOSpagecacheresultinginhigherdiskIOusageSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-memory.allowedPercentfloatAllowedpercentofsystemmemoryVictoriaMetricscachesmayoccupy.Seealso-memory.allowedBytes.ToolowavaluemayincreasecachemissrateusuallyresultinginhigherCPUanddiskIOusage.ToohighavaluemayevicttoomuchdatafromtheOSpagecachewhichwillresultinhigherdiskIOusage(default60)-metrics.exposeMetadataWhethertoexposeTYPEandHELPmetadataatthe/metricspage,whichisexposedat-httpListenAddr.Themetadatamaybeneededwhenthe/metricspageisconsumedbysystems,whichrequirethisinformation.Forexample,ManagedPrometheusinGoogleCloud-https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting#missing-metric-type-metricsAuthKeyvalueAuthkeyfor/metricsendpoint.ItmustbepassedviaauthKeyqueryarg.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-metricsAuthKey=file:///abs/path/to/fileor-metricsAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-metricsAuthKey=http://host/pathor-metricsAuthKey=https://host/path-mtlsarrayWhethertorequirevalidclientcertificateforhttpsrequeststothecorresponding-httpListenAddr.Thisflagworksonlyif-tlsflagisset.Seealso-mtlsCAFile.ThisflagisavailableonlyinEnterprisebinaries.Seehttps://docs.victoriametrics.com/enterprise/Supportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-mtlsCAFilearrayOptionalpathtoTLSRootCAforverifyingclientcertificatesatthecorresponding-httpListenAddrwhen-mtlsisenabled.BydefaultthehostsystemTLSRootCAisusedforclientcertificateverification.ThisflagisavailableonlyinEnterprisebinaries.Seehttps://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-pprofAuthKeyvalueAuthkeyfor/debug/pprof/*endpoints.ItmustbepassedviaauthKeyqueryarg.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-pprofAuthKey=file:///abs/path/to/fileor-pprofAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-pprofAuthKey=http://host/pathor-pprofAuthKey=https://host/path-prevCacheRemovalPercentfloatItemsinthepreviouscachesareremovedwhenthepercentofrequestsitservesbecomeslowerthanthisvalue.HighervaluesreducememoryusageatthecostofhigherCPUusage.Seealso-cacheExpireDuration(default0.1)-pushmetrics.disableCompressionWhethertodisablerequestbodycompressionwhenpushingmetricstoevery-pushmetrics.url-pushmetrics.extraLabelarrayOptionallabelstoaddtometricspushedtoevery-pushmetrics.url.Forexample,-pushmetrics.extraLabel='instance="foo"'addsinstance="foo"labeltoallthemetricspushedtoevery-pushmetrics.urlSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-pushmetrics.headerarrayOptionalHTTPrequestheadertosendtoevery-pushmetrics.url.Forexample,-pushmetrics.header='Authorization: Basic foobar'adds'Authorization: Basic foobar'headertoeveryrequesttoevery-pushmetrics.urlSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-pushmetrics.intervaldurationIntervalforpushingmetricstoevery-pushmetrics.url(default10s)-pushmetrics.urlarrayOptionalURLtopushmetricsexposedat/metricspage.Seehttps://docs.victoriametrics.com/#push-metrics . By default, metrics exposed at /metrics page aren't pushed to any remote storageSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-replicationFactorarrayHowmanycopiesofeveryingestedsampleisavailableacross-storageNodenodes.vmselectcontinuesreturningfullresponseswhenuptoreplicationFactor-1vmstoragenodesaretemporarilyunavailable.Seealso-globalReplicationFactorand-search.skipSlowReplicas(default1)Supportsanarrayof`key:value`entriesseparatedbycommaorspecifiedviamultipleflags.-search.cacheTimestampOffsetdurationThemaximumdurationsincethecurrenttimeforresponsedata,whichisalwaysqueriedfromtheoriginalrawdata,withoutusingtheresponsecache.IncreasethisvalueifyouseegapsinresponsesduetotimesynchronizationissuesbetweenVictoriaMetricsanddatasources(default5m0s)-search.denyPartialResponseWhethertodenypartialresponsesifapartof-storageNodeinstancesfailtoperformqueries;thistradesavailabilityoverconsistency;seealso-search.maxQueryDuration-search.disableCacheWhethertodisableresponsecaching.Thismaybeusefulwheningestinghistoricaldata.Seehttps://docs.victoriametrics.com/#backfilling . See also -search.resetRollupResultCacheOnStartup-search.disableImplicitConversionWhethertoreturnanerrorforqueriesthatrelyonimplicitsubqueryconversions,seehttps://docs.victoriametrics.com/metricsql/#subqueries for details. See also -search.logImplicitConversion-search.graphiteMaxPointsPerSeriesintThemaximumnumberofpointsperseriesGraphiterenderAPIcanreturn(default1000000)-search.graphiteStorageStepdurationTheintervalbetweendatapointsstoredinthedatabase.ItisusedatGraphiteRenderAPIhandlerfornormalizingtheintervalbetweendatapointsincaseitisn't normalized. It can be overridden by sending 'storage_step' query arg to /render API or by sending the desired interval via 'Storage-Step' http header during querying /render API (default 10s)-search.ignoreExtraFiltersAtLabelsAPIWhethertoignorematch[],extra_filters[]andextra_labelqueryargsat/api/v1/labelsand/api/v1/label/.../values.ThismaybeusefulfordecreasingloadonVictoriaMetricswhenextrafiltersmatchtoomanytimeseries.Thedownsideisthatsuperfluouslabelsorseriescouldbereturned,whichdonotmatchtheextrafilters.Seealso-search.maxLabelsAPISeriesand-search.maxLabelsAPIDuration-search.latencyOffsetdurationThetimewhendatapointsbecomevisibleinqueryresultsafterthecollection.Itcanbeoverriddenonper-querybasisvialatency_offsetarg.Toosmallvaluecanresultinincompletelastpointsforqueryresults(default30s)-search.logImplicitConversionWhethertologquerieswithimplicitsubqueryconversions,seehttps://docs.victoriametrics.com/metricsql/#subqueries for details. Such conversion can be disabled using -search.disableImplicitConversion-search.logQueryMemoryUsagesizeLogqueryandincrementvm_memory_intensive_queries_totalmetriceachtimethequeryrequiresmorememorythanspecifiedbythisflag.Thismayhelpdetectingandoptimizingheavyqueries.Queryloggingisdisabledbydefault.Seealso-search.logSlowQueryDurationand-search.maxMemoryPerQuerySupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-search.logSlowQueryDurationdurationLogquerieswithexecutiontimeexceedingthisvalue.Zerodisablesslowquerylogging.Seealso-search.logQueryMemoryUsage(default5s)-search.maxBinaryOpPushdownLabelValuesintThemaximumnumberofvaluesforalabelinthefirstexpressionthatcanbeextractedasacommonlabelfilterandpusheddowntothesecondexpressioninabinaryoperation.Alargervaluemakesthepushed-downfiltermorecomplexbutfewertimeserieswillbereturned.Thisflagisusefulwhenselectivelabelcontainsnumerousvalues,forexample`instance`,andstorageresourcesareabundant.(default100)-search.maxConcurrentRequestsintThemaximumnumberofconcurrentsearchrequests.Itshouldn't be high, since a single request can saturate all the CPU cores, while many concurrently executed requests may require high amounts of memory. See also -search.maxQueueDuration and -search.maxMemoryPerQuery (default 16)-search.maxDeleteDurationdurationThemaximumdurationfor/api/v1/admin/tsdb/delete_seriescall(default5m)-search.maxDeleteSeriesintThemaximumnumberoftimeseries,whichcanbedeletedusing/api/v1/admin/tsdb/delete_series.Thisoptionallowslimitingmemoryusage(default1000000)-search.maxTSDBStatusTopNSeriesintThemaximumnumberoftimeseriesthatcanbereturnedfrom/api/v1/status/tsdb.Thisoptionallowslimitingmemoryusage(default1000)-search.maxExportDurationdurationThemaximumdurationfor/api/v1/exportcall(default720h0m0s)-search.maxExportSeriesintThemaximumnumberoftimeseries,whichcanbereturnedfrom/api/v1/export*APIs.Thisoptionallowslimitingmemoryusage(default10000000)-search.maxFederateSeriesintThemaximumnumberoftimeseries,whichcanbereturnedfrom/federate.Thisoptionallowslimitingmemoryusage(default1000000)-search.maxGraphiteSeriesintThemaximumnumberoftimeseries,whichcanbescannedduringqueriestoGraphiteRenderAPI.Seehttps://docs.victoriametrics.com/#graphite-render-api-usage (default 300000)-search.maxGraphiteTagKeysintThemaximumnumberoftagkeysreturnedfromGraphiteAPI,whichreturnstags.Seehttps://docs.victoriametrics.com/#graphite-tags-api-usage (default 100000)-search.maxGraphiteTagValuesintThemaximumnumberoftagvaluesreturnedfromGraphiteAPI,whichreturnstagvalues.Seehttps://docs.victoriametrics.com/#graphite-tags-api-usage (default 100000)-search.maxLabelsAPIDurationdurationThemaximumdurationfor/api/v1/labels,/api/v1/label/.../valuesand/api/v1/seriesrequests.Seealso-search.maxLabelsAPISeriesand-search.ignoreExtraFiltersAtLabelsAPI(default5s)-search.maxLabelsAPISeriesintThemaximumnumberoftimeseries,whichcouldbescannedwhensearchingforthematchingtimeseriesat/api/v1/labelsand/api/v1/label/.../values.ThisoptionallowslimitingmemoryusageandCPUusage.Seealso-search.maxLabelsAPIDuration,-search.maxTagKeys,-search.maxTagValuesand-search.ignoreExtraFiltersAtLabelsAPI(default1000000)-search.maxLookbackdurationSynonymto-query.lookback-deltafromPrometheus.Thevalueisdynamicallydetectedfromintervalbetweentimeseriesdatapointsifnotset.Itcanbeoverriddenonper-querybasisviamax_lookbackarg.Seealso'-search.maxStalenessInterval'flag,whichhasthesamemeaningduetohistoricalreasons-search.maxMemoryPerQuerysizeThemaximumamountsofmemoryasinglequerymayconsume.Queriesrequiringmorememoryarerejected.Thetotalmemorylimitforconcurrentlyexecutedqueriescanbeestimatedas-search.maxMemoryPerQuerymultipliedby-search.maxConcurrentRequests.Seealso-search.logQueryMemoryUsageSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-search.maxPointsPerTimeseriesintThemaximumpointsperasingletimeseriesreturnedfrom/api/v1/query_range.Thisoptiondoesn't limit the number of scanned raw samples in the database. The main purpose of this option is to limit the number of per-series points returned to graphing UI such as VMUI or Grafana. There is no sense in setting this limit to values bigger than the horizontal resolution of the graph. See also -search.maxResponseSeries (default 30000)-search.maxPointsSubqueryPerTimeseriesintThemaximumnumberofpointsperseries,whichcanbegeneratedbysubquery.Seehttps://valyala.medium.com/prometheus-subqueries-in-victoriametrics-9b1492b720b3(default100000)-search.maxQueryDurationdurationThemaximumdurationforqueryexecution.Itcanbeoverriddentoasmallervalueonaper-querybasisvia'timeout'queryarg(default30s)-search.maxQueryLensizeThemaximumsearchquerylengthinbytesSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default16384)-search.maxQueueDurationdurationThemaximumtimetherequestwaitsforexecutionwhen-search.maxConcurrentRequestslimitisreached;seealso-search.maxQueryDuration(default10s)-search.maxResponseSeriesintThemaximumnumberoftimeserieswhichcanbereturnedfrom/api/v1/queryand/api/v1/query_range.Thelimitisdisabledifitequalsto0.Seealso-search.maxPointsPerTimeseriesand-search.maxUniqueTimeseries-search.maxSamplesPerQueryintThemaximumnumberofrawsamplesasinglequerycanprocessacrossalltimeseries.Thisprotectsfromheavyqueries,whichselectunexpectedlyhighnumberofrawsamples.Seealso-search.maxSamplesPerSeries(default1000000000)-search.maxSamplesPerSeriesintThemaximumnumberofrawsamplesasinglequerycanscanpereachtimeseries.Seealso-search.maxSamplesPerQuery(default30000000)-search.maxSeriesintThemaximumnumberoftimeseries,whichcanbereturnedfrom/api/v1/series.Thisoptionallowslimitingmemoryusage(default30000)-search.maxSeriesPerAggrFuncintThemaximumnumberoftimeseriesanaggregateMetricsQLfunctioncangenerate(default1000000)-search.maxStalenessIntervaldurationThemaximumintervalforstalenesscalculations.Bydefault,itisautomaticallycalculatedfromthemedianintervalbetweensamples.ThisflagcouldbeusefulfortuningPrometheusdatamodelclosertoInflux-styledatamodel.Seehttps://prometheus.io/docs/prometheus/latest/querying/basics/#staleness for details. See also '-search.setLookbackToStep' flag-search.maxStatusRequestDurationdurationThemaximumdurationfor/api/v1/status/*requests(default5m0s)-search.maxStepForPointsAdjustmentdurationThemaximumstepwhen/api/v1/query_rangehandleradjustspointswithtimestampscloserthan-search.latencyOffsettothecurrenttime.Theadjustmentisneededbecausesuchpointsmaycontainincompletedata(default1m0s)-search.maxTSDBStatusSeriesintThemaximumnumberoftimeseries,whichcanbeprocessedduringthecallto/api/v1/status/tsdb.Thisoptionallowslimitingmemoryusage(default10000000)-search.maxTagValueSuffixesPerSearchintThemaximumnumberoftagvaluesuffixesreturnedfrom/metrics/find(default100000)-search.maxUniqueTimeseriesintThemaximumnumberofuniquetimeseries,whichcanbeselectedduring/api/v1/queryand/api/v1/query_rangequeries.Thisoptionallowslimitingmemoryusage.Thelimitcan't exceed the corresponding value set in vmstorage if the `-search.maxUniqueTimeseries` flag is explicitly defined there. (default 0)-search.maxWorkersPerQueryintThemaximumnumberofCPUcoresasinglequerycanuse.Thedefaultvalueshouldworkgoodformostcases.Theflagcanbesettolowervaluesforimprovingperformanceofbignumberofconcurrentlyexecutedqueries.Theflagcanbesettobiggervaluesforimprovingperformanceofheavyqueries,whichscanbignumberoftimeseries(>10K)and/orbignumberofsamples(>100M).ThereisnosenseinsettingthisflagtovaluesbiggerthanthenumberofCPUcoresavailableonthesystem(default16)-search.minStalenessIntervaldurationTheminimumintervalforstalenesscalculations.Thisflagcouldbeusefulforremovinggapsongraphsgeneratedfromtimeserieswithirregularintervalsbetweensamples.Seealso'-search.maxStalenessInterval'-search.minWindowForInstantRollupOptimizationvalueEnablecache-basedoptimizationforrepeatedqueriesto/api/v1/query(akainstantqueries),whichcontainrollupfunctionswithlookbehindwindowexceedingthegivenvalueThefollowingoptionalsuffixesaresupported:s(second),h(hour),d(day),w(week),y(year).Ifsuffixisn't set, then the duration is counted in months (default 3h)-search.noStaleMarkersSetthisflagtotrueifthedatabasedoesn't contain Prometheus stale markers, so there is no need in spending additional CPU time on its handling. Staleness markers may exist only in data obtained from Prometheus scrape targets-search.queryStats.lastQueriesCountintQuerystatsfor/api/v1/status/top_queriesistrackedonthisnumberoflastqueries.Zerovaluedisablesquerystatstracking(default20000)-search.queryStats.minQueryDurationdurationTheminimumdurationforqueriestotrackinquerystatsat/api/v1/status/top_queries.Querieswithlowerdurationareignoredinquerystats(default1ms)-search.resetCacheAuthKeyvalueOptionalauthKeyforresettingrollupcachevia/internal/resetRollupResultCachecall.ItcouldbepassedviaauthKeyqueryarg.Flagvaluecanbereadfromthegivenfilewhenusing-search.resetCacheAuthKey=file:///abs/path/to/fileor-search.resetCacheAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-search.resetCacheAuthKey=http://host/pathor-search.resetCacheAuthKey=https://host/path-search.resetRollupResultCacheOnStartupWhethertoresetrollupresultcacheonstartup.Seehttps://docs.victoriametrics.com/#rollup-result-cache . See also -search.disableCache-search.setLookbackToStepWhethertofixlookbackintervalto'step'queryargvalue.Ifsettotrue,thequerymodelbecomesclosertoInfluxDBdatamodel.Ifsettotrue,then-search.maxLookbackand-search.maxStalenessIntervalareignored-search.skipSlowReplicasWhethertoskip-replicationFactor-1slowestvmstoragenodesduringquerying.Enablingthissettingmayimprovequeryspeed,butitcouldalsoleadtoincompleteresultsifsomequerieddatahaslessthan-replicationFactorcopiesatvmstoragenodes.Considerenablingthissettingonlyifallthequerieddatacontains-replicationFactorcopiesinthecluster-search.inmemoryBufSizeBytessizeSizeforin-memorydatablocksusedduringprocessingsearchrequests.Bydefault,thesizeisautomaticallycalculatedbasedonavailablememory.Adjustthisflagvalueifyouobservethatvm_tmp_blocks_max_inmemory_file_size_bytesmetricconstantlyshowsmuchhighervaluesthanvm_tmp_blocks_inmemory_file_size_bytes.Seehttps://github.com/VictoriaMetrics/VictoriaMetrics/pull/6851Supportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-search.tenantCacheExpireDurationdurationTheexpirydurationforlistoftenantsformulti-tenantqueries.(default5m0s)-search.treatDotsAsIsInRegexpsWhethertotreatdotsasisinregexplabelfiltersusedinqueries.Forexample,foo{bar=~"a.b.c"}willbeautomaticallyconvertedtofoo{bar=~"a\\.b\\.c"},i.e.allthedotsinregexpfilterswillbeautomaticallyescapedinordertomatchonlydotcharinsteadofmatchinganychar.Dotsin".+",".*"and".{n}"regexpsaren't escaped. This option is DEPRECATED in favor of {__graphite__="a.*.c"} syntax for selecting metrics matching the given Graphite metrics filter-selectNodearrayComma-separatedaddressesofvmselectnodes;usage:-selectNode=vmselect-host1,...,vmselect-hostNSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-storageNodearrayComma-separatedaddressesofvmstoragenodes;usage:-storageNode=vmstorage-host1,...,vmstorage-hostN.EnterpriseversionofVictoriaMetricssupportsautomaticdiscoveryofvmstorageaddressesviaDNSSRVrecords.Forexample,-storageNode=srv+vmstorage.addrs.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#automatic-vmstorage-discoverySupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-storageNode.discoveryIntervaldurationIntervalforrefreshing-storageNodelistbehindDNSSRVrecords.Theminimumsupportedintervalis1s.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#automatic-vmstorage-discovery . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ (default 2s)-storageNode.filterstringAnoptionalregexpfilterfordiscovered-storageNodeaddressesaccordingtohttps://docs.victoriametrics.com/cluster-victoriametrics/#automatic-vmstorage-discovery. Discovered addresses matching the filter are retained, while other addresses are ignored. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/-tlsarrayWhethertoenableTLSforincomingHTTPrequestsatthegiven-httpListenAddr(akahttps).-tlsCertFileand-tlsKeyFilemustbesetif-tlsisset.Seealso-mtlsSupportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-tlsAutocertCacheDirstringDirectorytostoreTLScertificatesissuedviaLet's Encrypt. Certificates are lost on restarts if this flag isn'tset.ThisflagisavailableonlyinEnterprisebinaries.Seehttps://docs.victoriametrics.com/enterprise/-tlsAutocertEmailstringContactemailfortheissuedLet's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/-tlsAutocertHostsarrayOptionalhostnamesforautomaticissuingofLet's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-tlsCertFilearrayPathtofilewithTLScertificateforthecorresponding-httpListenAddrif-tlsisset.PreferECDSAcertsinsteadofRSAcertsasRSAcertsareslower.Theprovidedcertificatefileisautomaticallyre-readeverysecond,soitcanbedynamicallyupdated.Seealso-tlsAutocertHostsSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-tlsCipherSuitesarrayOptionallistofTLSciphersuitesforincomingrequestsoverHTTPSif-tlsisset.Seethelistofsupportedciphersuitesathttps://pkg.go.dev/crypto/tls#pkg-constantsSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-tlsKeyFilearrayPathtofilewithTLSkeyforthecorresponding-httpListenAddrif-tlsisset.Theprovidedkeyfileisautomaticallyre-readeverysecond,soitcanbedynamicallyupdated.Seealso-tlsAutocertHostsSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-tlsMinVersionarrayOptionalminimumTLSversiontouseforthecorresponding-httpListenAddrif-tlsisset.Supportedvalues:TLS10,TLS11,TLS12,TLS13Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-versionShowVictoriaMetricsversion-vmalert.proxyURLstringOptionalURLforproxyingrequeststovmalert.Forexample,if-vmalert.proxyURL=http://vmalert:8880,thenalertingAPIrequestssuchas/api/v1/rulesfromGrafanawillbeproxiedtohttp://vmalert:8880/api/v1/rules-vmstorageDialTimeoutdurationTimeoutforestablishingRPCconnectionsfromvmselecttovmstorage.Seealso-vmstorageUserTimeout(default3s)-vmstorageUserTimeoutdurationNetworktimeoutforRPCconnectionsfromvmselecttovmstorage(Linuxonly).Lowervaluesreducethemaximumquerydurationswhensomevmstoragenodesbecomeunavailablebecauseofnetworkingissues.ReadmoreaboutTCP_USER_TIMEOUTathttps://blog.cloudflare.com/when-tcp-sockets-refuse-to-die/.Seealso-vmstorageDialTimeout(default3s)-vmui.customDashboardsPathstringOptionalpathtovmuidashboards.Seehttps://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmui/packages/vmui/public/dashboards-vmui.defaultTimezonestringThedefaulttimezonetobeusedinvmui.TimezonemustbeavalidIANATimeZone.Forexample:America/New_York,Europe/Berlin,Etc/GMT+3orLocal
-bigMergeConcurrencyintDeprecated:thisflagdoesnothing-blockcache.missesBeforeCachingintThenumberofcachemissesbeforeputtingtheblockintocache.Highervaluesmayreduceindexdb/dataBlockscachesizeatthecostofhigherCPUanddiskreadusage(default2)-cacheExpireDurationdurationItemsareremovedfromin-memorycachesaftertheyaren't accessed for this duration. Lower values may reduce memory usage at the cost of higher CPU usage. See also -prevCacheRemovalPercent (default 30m0s)-cluster.tlsWhethertouseTLSwhenacceptingconnectionsfromvminsertandvmselect.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/-cluster.tlsCAFilestringPathtoTLSCAfiletouseforverifyingcertificatesprovidedbyvminsertandvmselectif-cluster.tlsflagisset.BydefaultsystemCAisused.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/-cluster.tlsCertFilestringPathtoserver-sideTLScertificatefiletousewhenacceptingconnectionsfromvminsertandvmselectif-cluster.tlsflagisset.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/-cluster.tlsCipherSuitesarrayOptionallistofTLSciphersuitesusedforconnectionsfromvminsertandvmselectif-cluster.tlsflagisset.Seethelistofsupportedciphersuitesathttps://pkg.go.dev/crypto/tls#pkg-constants . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-cluster.tlsInsecureSkipVerifyWhethertoskipverificationofTLScertificatesprovidedbyvminsertandvmselectif-cluster.tlsflagisset.NotethatdisabledTLScertificateverificationbreakssecurity.ThisflagisavailableonlyinVictoriaMetricsenterprise.Seehttps://docs.victoriametrics.com/enterprise/-cluster.tlsKeyFilestringPathtoserver-sideTLSkeyfiletousewhenacceptingconnectionsfromvminsertandvmselectif-cluster.tlsflagisset.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/-dedup.minScrapeIntervaldurationLeaveonlythelastsampleineverytimeseriespereachdiscreteintervalequalto-dedup.minScrapeInterval>0.Seehttps://docs.victoriametrics.com/#deduplication for details-denyQueriesOutsideRetentionWhethertodenyqueriesoutsideoftheconfigured-retentionPeriod.Whenset,then/api/v1/query_rangewouldreturn'503 Service Unavailable'errorforquerieswith'from'valueoutside-retentionPeriod.Thismaybeusefulwhenmultipledatasourceswithdistinctretentionsarehiddenbehindquery-tee-denyQueryTracingWhethertodisabletheabilitytotracequeries.Seehttps://docs.victoriametrics.com/#query-tracing-downsampling.periodarrayComma-separateddownsamplingperiodsintheformat'offset:period'.Forexample,'30d:10m'instructstoleaveasinglesampleper10minutesforsamplesolderthan30days.Whensettingmultipledownsamplingperiods,itisnecessaryfortheperiodstobemultiplesofeachother.Seehttps://docs.victoriametrics.com/#downsampling for details. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-enableTCP6WhethertoenableIPv6forlisteninganddialing.Bydefault,onlyIPv4TCPandUDPareused-envflag.enableWhethertoenablereadingflagsfromenvironmentvariablesinadditiontothecommandline.Commandlineflagvalueshavepriorityovervaluesfromenvironmentvars.Flagsarereadonlyfromthecommandlineifthisflagisn't set. See https://docs.victoriametrics.com/#environment-variables for more details-envflag.prefixstringPrefixforenvironmentvariablesif-envflag.enableisset-eulaDeprecated,pleaseuse-licenseor-licenseFileflagsinstead.Byspecifyingthisflag,youconfirmthatyouhaveanenterpriselicenseandaccepttheESAhttps://victoriametrics.com/legal/esa/.ThisflagisavailableonlyinEnterprisebinaries.Seehttps://docs.victoriametrics.com/enterprise/-filestream.disableFadviseWhethertodisablefadvise()syscallwhenreadinglargedatafiles.Thefadvise()syscallpreventsfromevictionofrecentlyaccesseddatafromOSpagecacheduringbackgroundmergesandbackups.InsomerarecasesitisbettertodisablethesyscallifitusestoomuchCPU-finalMergeDelaydurationDeprecated:thisflagdoesnothing-flagsAuthKeyvalueAuthkeyfor/flagsendpoint.ItmustbepassedviaauthKeyqueryarg.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-flagsAuthKey=file:///abs/path/to/fileor-flagsAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-flagsAuthKey=http://host/pathor-flagsAuthKey=https://host/path-forceFlushAuthKeyvalueauthKey,whichmustbepassedinquerystringto/internal/force_flushpagesFlagvaluecanbereadfromthegivenfilewhenusing-forceFlushAuthKey=file:///abs/path/to/fileor-forceFlushAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-forceFlushAuthKey=http://host/pathor-forceFlushAuthKey=https://host/path-forceMergeAuthKeyvalueauthKey,whichmustbepassedinquerystringto/internal/force_mergepagesFlagvaluecanbereadfromthegivenfilewhenusing-forceMergeAuthKey=file:///abs/path/to/fileor-forceMergeAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-forceMergeAuthKey=http://host/pathor-forceMergeAuthKey=https://host/path-fs.disableMmapWhethertousepread()insteadofmmap()forreadingdatafiles.Bydefault,mmap()isusedfor64-bitarchesandpread()isusedfor32-bitarches,sincetheycannotreaddatafilesbiggerthan2^32bytesinmemory.mmap()isusuallyfasterforreadingsmalldatachunksthanpread()-http.connTimeoutdurationIncomingconnectionsto-httpListenAddrareclosedaftertheconfiguredtimeout.ThismayhelpevenlyspreadingloadamongaclusterofservicesbehindTCP-levelloadbalancer.Zerovaluedisablesclosingofincomingconnections(default2m0s)-http.disableResponseCompressionDisablecompressionofHTTPresponsestosaveCPUresources.Bydefault,compressionisenabledtosavenetworkbandwidth-http.header.cspstringValuefor'Content-Security-Policy'header,recommended:"default-src 'self'"-http.header.frameOptionsstringValuefor'X-Frame-Options'header-http.header.hstsstringValuefor'Strict-Transport-Security'header,recommended:'max-age=31536000; includeSubDomains'-http.idleConnTimeoutdurationTimeoutforincomingidlehttpconnections(default1m0s)-http.maxGracefulShutdownDurationdurationThemaximumdurationforagracefulshutdownoftheHTTPserver.Ahighlyloadedservermayrequireincreasedvalueforagracefulshutdown(default7s)-http.pathPrefixstringAnoptionalprefixtoaddtoallthepathshandledbyhttpserver.Forexample,if'-http.pathPrefix=/foo/bar'isset,thenallthehttprequestswillbehandledon'/foo/bar/*'paths.Thismaybeusefulforproxiedrequests.Seehttps://www.robustperception.io/using-external-urls-and-proxies-with-prometheus-http.shutdownDelaydurationOptionaldelaybeforehttpservershutdown.Duringthisdelay,theserverreturnsnon-OKresponsesfrom/healthpage,soloadbalancerscanroutenewrequeststootherservers-httpAuth.passwordvaluePasswordforHTTPserver's Basic Auth. The authentication is disabled if -httpAuth.username is emptyFlagvaluecanbereadfromthegivenfilewhenusing-httpAuth.password=file:///abs/path/to/fileor-httpAuth.password=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-httpAuth.password=http://host/pathor-httpAuth.password=https://host/path-httpAuth.usernamestringUsernameforHTTPserver's Basic Auth. The authentication is disabled if empty. See also -httpAuth.password-httpListenAddrarrayAddresstolistenforincominghttprequests.Seealso-httpListenAddr.useProxyProtocolSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-httpListenAddr.useProxyProtocolarrayWhethertouseproxyprotocolforconnectionsacceptedatthegiven-httpListenAddr.Seehttps://www.haproxy.org/download/1.8/doc/proxy-protocol.txt.Withenabledproxyprotocolhttpservercannotserveregular/metricsendpoint.Use-pushmetrics.urlformetricspushingSupportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-inmemoryDataFlushIntervaldurationTheintervalforguaranteedsavingofin-memorydatatodisk.ThesaveddatasurvivesuncleanshutdownssuchasOOMcrash,hardwarereset,SIGKILL,etc.Biggerintervalsmayhelpincreasethelifetimeofflashstoragewithlimitedwritecycles(e.g.RaspberryPI).SmallerintervalsincreasediskIOload.Minimumsupportedvalueis1s(default5s)-insert.maxQueueDurationdurationThemaximumdurationtowaitinthequeuewhen-maxConcurrentInsertsconcurrentinsertrequestsareexecuted(default1m0s)-internStringCacheExpireDurationdurationTheexpirydurationforcachesforinternedstrings.Seehttps://en.wikipedia.org/wiki/String_interning.Seealso-internStringMaxLenand-internStringDisableCache(default6m0s)-internStringDisableCacheWhethertodisablecachesforinternedstrings.ThismayreducememoryusageatthecostofhigherCPUusage.Seehttps://en.wikipedia.org/wiki/String_interning.Seealso-internStringCacheExpireDurationand-internStringMaxLen-internStringMaxLenintThemaximumlengthforstringstointern.AlowerlimitmaysavememoryatthecostofhigherCPUusage.Seehttps://en.wikipedia.org/wiki/String_interning.Seealso-internStringDisableCacheand-internStringCacheExpireDuration(default500)-licensestringLicensekeyforVictoriaMetricsEnterprise.Seehttps://victoriametrics.com/products/enterprise/.TrialEnterpriselicensecanbeobtainedfromhttps://victoriametrics.com/products/enterprise/trial/.ThisflagisavailableonlyinEnterprisebinaries.Thelicensekeycanbealsopassedviafilespecifiedby-licenseFilecommand-lineflag-license.forceOfflineWhethertoenableofflineverificationforVictoriaMetricsEnterpriselicensekey,whichhasbeenpassedeithervia-licenseorvia-licenseFilecommand-lineflag.Theissuedlicensekeymustsupportofflineverificationfeature.Contactinfo@victoriametrics.comifyouneedofflinelicenseverification.ThisflagisavailableonlyinEnterprisebinaries-licenseFilestringPathtofilewithlicensekeyforVictoriaMetricsEnterprise.Seehttps://victoriametrics.com/products/enterprise/.TrialEnterpriselicensecanbeobtainedfromhttps://victoriametrics.com/products/enterprise/trial/.ThisflagisavailableonlyinEnterprisebinaries.Thelicensekeycanbealsopassedinlinevia-licensecommand-lineflag-logNewSeriesWhethertolognewseries.Thisoptionisfordebugpurposesonly.ItcanleadtoperformanceissueswhenbignumberofnewseriesareingestedintoVictoriaMetrics-loggerDisableTimestampsWhethertodisablewritingtimestampsinlogs-loggerErrorsPerSecondLimitintPer-secondlimitonthenumberofERRORmessages.Ifmorethanthegivennumberoferrorsareemittedpersecond,theremainingerrorsaresuppressed.Zerovaluesdisabletheratelimit-loggerFormatstringFormatforlogs.Possiblevalues:default,json(default"default")-loggerJSONFieldsstringAllowsrenamingfieldsinJSONformattedlogs.Example:"ts:timestamp,msg:message"renames"ts"to"timestamp"and"msg"to"message".Supportedfields:ts,level,caller,msg-loggerLevelstringMinimumleveloferrorstolog.Possiblevalues:INFO,WARN,ERROR,FATAL,PANIC(default"INFO")-loggerMaxArgLenintThemaximumlengthofasingleloggedargument.Longerargumentsarereplacedwith'arg_start..arg_end',where'arg_start'and'arg_end'isprefixandsuffixoftheargwiththelengthnotexceeding-loggerMaxArgLen/2(default1000)-loggerOutputstringOutputforthelogs.Supportedvalues:stderr,stdout(default"stderr")-loggerTimezonestringTimezonetousefortimestampsinlogs.TimezonemustbeavalidIANATimeZone.Forexample:America/New_York,Europe/Berlin,Etc/GMT+3orLocal(default"UTC")-loggerWarnsPerSecondLimitintPer-secondlimitonthenumberofWARNmessages.Ifmorethanthegivennumberofwarnsareemittedpersecond,thentheremainingwarnsaresuppressed.Zerovaluesdisabletheratelimit-maxConcurrentInsertsintThemaximumnumberofconcurrentinsertrequests.Sethighervaluewhenclientssenddataoverslownetworks.DefaultvaluedependsonthenumberofavailableCPUcores.Itshouldworkfineinmostcasessinceitminimizesresourceusage.Seealso-insert.maxQueueDuration-memory.allowedBytessizeAllowedsizeofsystemmemoryVictoriaMetricscachesmayoccupy.Thisoptionoverrides-memory.allowedPercentifsettoanon-zerovalue.ToolowavaluemayincreasethecachemissrateusuallyresultinginhigherCPUanddiskIOusage.ToohighavaluemayevicttoomuchdatafromtheOSpagecacheresultinginhigherdiskIOusageSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-memory.allowedPercentfloatAllowedpercentofsystemmemoryVictoriaMetricscachesmayoccupy.Seealso-memory.allowedBytes.ToolowavaluemayincreasecachemissrateusuallyresultinginhigherCPUanddiskIOusage.ToohighavaluemayevicttoomuchdatafromtheOSpagecachewhichwillresultinhigherdiskIOusage(default60)-metrics.exposeMetadataWhethertoexposeTYPEandHELPmetadataatthe/metricspage,whichisexposedat-httpListenAddr.Themetadatamaybeneededwhenthe/metricspageisconsumedbysystems,whichrequirethisinformation.Forexample,ManagedPrometheusinGoogleCloud-https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting#missing-metric-type-metricsAuthKeyvalueAuthkeyfor/metricsendpoint.ItmustbepassedviaauthKeyqueryarg.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-metricsAuthKey=file:///abs/path/to/fileor-metricsAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-metricsAuthKey=http://host/pathor-metricsAuthKey=https://host/path-mtlsarrayWhethertorequirevalidclientcertificateforhttpsrequeststothecorresponding-httpListenAddr.Thisflagworksonlyif-tlsflagisset.Seealso-mtlsCAFile.ThisflagisavailableonlyinEnterprisebinaries.Seehttps://docs.victoriametrics.com/enterprise/Supportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-mtlsCAFilearrayOptionalpathtoTLSRootCAforverifyingclientcertificatesatthecorresponding-httpListenAddrwhen-mtlsisenabled.BydefaultthehostsystemTLSRootCAisusedforclientcertificateverification.ThisflagisavailableonlyinEnterprisebinaries.Seehttps://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-pprofAuthKeyvalueAuthkeyfor/debug/pprof/*endpoints.ItmustbepassedviaauthKeyqueryarg.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-pprofAuthKey=file:///abs/path/to/fileor-pprofAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-pprofAuthKey=http://host/pathor-pprofAuthKey=https://host/path-precisionBitsintThenumberofprecisionbitstostorepereachvalue.Lowerprecisionbitsimprovesdatacompressionatthecostofprecisionloss(default64)-prevCacheRemovalPercentfloatItemsinthepreviouscachesareremovedwhenthepercentofrequestsitservesbecomeslowerthanthisvalue.HighervaluesreducememoryusageatthecostofhigherCPUusage.Seealso-cacheExpireDuration(default0.1)-pushmetrics.disableCompressionWhethertodisablerequestbodycompressionwhenpushingmetricstoevery-pushmetrics.url-pushmetrics.extraLabelarrayOptionallabelstoaddtometricspushedtoevery-pushmetrics.url.Forexample,-pushmetrics.extraLabel='instance="foo"'addsinstance="foo"labeltoallthemetricspushedtoevery-pushmetrics.urlSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-pushmetrics.headerarrayOptionalHTTPrequestheadertosendtoevery-pushmetrics.url.Forexample,-pushmetrics.header='Authorization: Basic foobar'adds'Authorization: Basic foobar'headertoeveryrequesttoevery-pushmetrics.urlSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-pushmetrics.intervaldurationIntervalforpushingmetricstoevery-pushmetrics.url(default10s)-pushmetrics.urlarrayOptionalURLtopushmetricsexposedat/metricspage.Seehttps://docs.victoriametrics.com/#push-metrics . By default, metrics exposed at /metrics page aren't pushed to any remote storageSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-retentionFilterarrayRetentionfilterintheformat'filter:retention'.Forexample,'{env="dev"}:3d'configurestheretentionfortimeserieswithenv="dev"labelto3days.Seehttps://docs.victoriametrics.com/#retention-filters for details. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-retentionPeriodvalueDatawithtimestampsoutsidetheretentionPeriodisautomaticallydeleted.TheminimumretentionPeriodis24hor1d.Seealso-retentionFilterThefollowingoptionalsuffixesaresupported:s(second),h(hour),d(day),w(week),y(year).Ifsuffixisn't set, then the duration is counted in months (default 1)-retentionTimezoneOffsetdurationTheoffsetforperformingindexdbrotation.Ifsetto0,thentheindexdbrotationisperformedat4amUTCtimepereach-retentionPeriod.Ifsetto2h,thentheindexdbrotationisperformedat4amEETtime(thetimezonewith+2hoffset)-rpc.disableCompressionWhethertodisablecompressionofthedatasentfromvmstoragetovmselect.ThisreducesCPUusageatthecostofhighernetworkbandwidthusage-search.maxConcurrentRequestsintThemaximumnumberofconcurrentvmselectrequeststhevmstoragecanprocessat-vmselectAddr.Itshouldn't be high, since a single request usually saturates a CPU core, and many concurrently executed requests may require high amounts of memory. See also -search.maxQueueDuration-search.maxQueueDurationdurationThemaximumtimetheincomingvmselectrequestwaitsforexecutionwhen-search.maxConcurrentRequestslimitisreached(default10s)-search.maxTagKeysintThemaximumnumberoftagkeysreturnedpersearch.Seealso-search.maxLabelsAPISeriesand-search.maxLabelsAPIDuration(default100000)-search.maxTagValueSuffixesPerSearchintThemaximumnumberoftagvaluesuffixesreturnedfrom/metrics/find(default100000)-search.maxTagValuesintThemaximumnumberoftagvaluesreturnedpersearch.Seealso-search.maxLabelsAPISeriesand-search.maxLabelsAPIDuration(default100000)-search.maxUniqueTimeseriesintThemaximumnumberofuniquetimeseries,whichcanbescannedduringeveryquery.Thisallowsprotectingagainstheavyqueries,whichselectunexpectedlyhighnumberofseries.Whensettozero,thelimitisautomaticallycalculatedbasedon-search.maxConcurrentRequests(inverselyproportional)andmemoryavailabletotheprocess(proportional).Seealso-search.max*command-lineflagsatvmselect.-smallMergeConcurrencyintDeprecated:thisflagdoesnothing-snapshotAuthKeyvalueauthKey,whichmustbepassedinquerystringto/snapshot*pagesFlagvaluecanbereadfromthegivenfilewhenusing-snapshotAuthKey=file:///abs/path/to/fileor-snapshotAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-snapshotAuthKey=http://host/pathor-snapshotAuthKey=https://host/path-snapshotCreateTimeoutdurationDeprecated:thisflagdoesnothing-snapshotsMaxAgevalueAutomaticallydeletesnapshotsolderthan-snapshotsMaxAgeifitissettonon-zeroduration.MakesurethatbackupprocesshasenoughtimetofinishthebackupbeforethecorrespondingsnapshotisautomaticallydeletedThefollowingoptionalsuffixesaresupported:s(second),h(hour),d(day),w(week),y(year).Ifsuffixisn't set, then the duration is counted in months (default 0)-storage.cacheSizeIndexDBDataBlockssizeOverridesmaxsizeforindexdb/dataBlockscache.Seehttps://docs.victoriametrics.com/single-server-victoriametrics/#cache-tuningSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-storage.cacheSizeIndexDBDataBlocksSparsesizeOverridesmaxsizeforindexdb/dataBlocksSparsecache.Seehttps://docs.victoriametrics.com/single-server-victoriametrics/#cache-tuningSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-storage.cacheSizeIndexDBIndexBlockssizeOverridesmaxsizeforindexdb/indexBlockscache.Seehttps://docs.victoriametrics.com/single-server-victoriametrics/#cache-tuningSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-storage.cacheSizeIndexDBTagFilterssizeOverridesmaxsizeforindexdb/tagFiltersToMetricIDscache.Seehttps://docs.victoriametrics.com/single-server-victoriametrics/#cache-tuningSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-storage.cacheSizeMetricNamesStatssizeOverridesmaxsizeforstorage/metricNamesStatsTrackercache.Seehttps://docs.victoriametrics.com/single-server-victoriametrics/#cache-tuningSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-storage.cacheSizeStorageTSIDsizeOverridesmaxsizeforstorage/tsidcache.Seehttps://docs.victoriametrics.com/single-server-victoriametrics/#cache-tuningSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-storage.finalDedupScheduleCheckIntervaldurationTheintervalforcheckingwhenfinaldeduplicationprocessshouldbestarted.Storageunconditionallyadds25%jittertotheintervalvalueoneachcheckevaluation.Changingtheintervaltothebiggervaluesmaydelaydownsampling,deduplicationforhistoricaldata.Seealsohttps://docs.victoriametrics.com/#deduplication (default 1h0m0s)-storage.maxDailySeriesintThemaximumnumberofuniqueseriescanbeaddedtothestorageduringthelast24hours.Excessseriesareloggedanddropped.Thiscanbeusefulforlimitingserieschurnrate.Seehttps://docs.victoriametrics.com/#cardinality-limiter . See also -storage.maxHourlySeries-storage.maxHourlySeriesintThemaximumnumberofuniqueseriescanbeaddedtothestorageduringthelasthour.Excessseriesareloggedanddropped.Thiscanbeusefulforlimitingseriescardinality.Seehttps://docs.victoriametrics.com/#cardinality-limiter . See also -storage.maxDailySeries-storage.minFreeDiskSpaceBytessizeTheminimumfreediskspaceat-storageDataPathafterwhichthestoragestopsacceptingnewdataSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default10000000)-storage.trackMetricNamesStatsWhethertotrackingestandqueryrequestsfortimeseriesmetricnames.Thisfeatureallowstotrackmetricnamesunusedatqueryrequests.Seehttps://docs.victoriametrics.com/#track-ingested-metrics-usage-storage.vminsertConnsShutdownDurationdurationThetimeneededforgradualclosingofvminsertconnectionsduringgracefulshutdown.BiggerdurationreducesspikesinCPU,RAManddiskIOloadontheremainingvmstoragenodesduringrollingrestart.Smallerdurationreducesthetimeneededtocloseallthevminsertconnections,thusreducingthetimeforgracefulshutdown.Seehttps://docs.victoriametrics.com/cluster-victoriametrics/#improving-re-routing-performance-during-restart (default 25s)-storageDataPathstringPathtostoragedata(default"vmstorage-data")-tlsarrayWhethertoenableTLSforincomingHTTPrequestsatthegiven-httpListenAddr(akahttps).-tlsCertFileand-tlsKeyFilemustbesetif-tlsisset.Seealso-mtlsSupportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-tlsAutocertCacheDirstringDirectorytostoreTLScertificatesissuedviaLet's Encrypt. Certificates are lost on restarts if this flag isn'tset.ThisflagisavailableonlyinEnterprisebinaries.Seehttps://docs.victoriametrics.com/enterprise/-tlsAutocertEmailstringContactemailfortheissuedLet's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/-tlsAutocertHostsarrayOptionalhostnamesforautomaticissuingofLet's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-tlsCertFilearrayPathtofilewithTLScertificateforthecorresponding-httpListenAddrif-tlsisset.PreferECDSAcertsinsteadofRSAcertsasRSAcertsareslower.Theprovidedcertificatefileisautomaticallyre-readeverysecond,soitcanbedynamicallyupdated.Seealso-tlsAutocertHostsSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-tlsCipherSuitesarrayOptionallistofTLSciphersuitesforincomingrequestsoverHTTPSif-tlsisset.Seethelistofsupportedciphersuitesathttps://pkg.go.dev/crypto/tls#pkg-constantsSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-tlsKeyFilearrayPathtofilewithTLSkeyforthecorresponding-httpListenAddrif-tlsisset.Theprovidedkeyfileisautomaticallyre-readeverysecond,soitcanbedynamicallyupdated.Seealso-tlsAutocertHostsSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-tlsMinVersionarrayOptionalminimumTLSversiontouseforthecorresponding-httpListenAddrif-tlsisset.Supportedvalues:TLS10,TLS11,TLS12,TLS13Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-versionShowVictoriaMetricsversion-vminsertAddrstringTCPaddresstoacceptconnectionsfromvminsertservices(default":8400")-vmselectAddrstringTCPaddresstoacceptconnectionsfromvmselectservices(default":8401")
