While VictoriaMetrics provides an efficient solution to store and observe metrics, our users needed something fast
and RAM friendly to scrape metrics from Prometheus-compatible exporters into VictoriaMetrics.
Also, we found that our user’s infrastructure are like snowflakes in that no two are alike. Therefore, we decided to add more flexibility
to vmagent such as the ability to accept metrics via popular push protocols
additionally to discovering Prometheus-compatible targets and scraping metrics from them.
Can be used as a drop-in replacement for Prometheus for discovering and scraping targets such as node_exporter.
Note that single-node VictoriaMetrics can also discover and scrape Prometheus-compatible targets in the same way as vmagent does -
see these docs.
Can add, remove and modify labels (aka tags) via Prometheus relabeling. Can filter data before sending it to remote storage. See these docs for details.
Can accept data via all the ingestion protocols supported by VictoriaMetrics - see these docs.
Can aggregate incoming samples by time and by labels before sending them to remote storage - see these docs.
Can replicate collected metrics simultaneously to multiple Prometheus-compatible remote storage systems - see these docs.
Works smoothly in environments with unstable connections to remote storage. If the remote storage is unavailable, the collected metrics
are buffered at -remoteWrite.tmpDataPath. The buffered metrics are sent to remote storage as soon as the connection
to the remote storage is repaired. The maximum disk usage for the buffer can be limited with -remoteWrite.maxDiskUsagePerURL.
Uses much lower amounts of RAM, CPU, disk IO and network bandwidth than Prometheus. The RAM usage and CPU usage can be reduced further
if needed according to these docs.
Scrape targets can be spread among multiple vmagent instances when big number of targets must be scraped. See these docs.
Can load scrape configs from multiple files. See these docs.
Can deal with high cardinality
and high churn rate issues by limiting the number of unique time series at scrape time
and before sending them to remote storage systems. See these docs.
Can write collected metrics to multiple tenants. See these docs.
Can read and write data from / to Kafka. See these docs.
Can read and write data from / to Google PubSub. See these docs.
Please download vmutils-* archive from releases page (
vmagent is also available in docker images Docker Hub and Quay),
unpack it and pass the following flags to the vmagent binary in order to start scraping Prometheus-compatible targets
and sending the data to the Prometheus-compatible remote storage:
-promscrape.config with the path to Prometheus config file (usually located at /etc/prometheus/prometheus.yml).
The path can point either to local file or to http url. See scrape config examples.
vmagent doesn’t support some sections of Prometheus config file, so you may need either to delete these sections or
to run vmagent with -promscrape.config.strictParse=false command-line flag.
In this case vmagent ignores unsupported sections. See the list of unsupported sections.
-remoteWrite.url with Prometheus-compatible remote storage endpoint such as VictoriaMetrics, where to send the data to.
The -remoteWrite.url may refer to DNS SRV address. See these docs for details.
If you use single-node VictoriaMetrics, then you can discover and scrape Prometheus-compatible targets directly from VictoriaMetrics
without the need to use vmagent - see these docs.
vmagent can run and collect metrics in IoT environments and industrial networks with unreliable or scheduled connections to their remote storage.
It buffers the collected data in local files until the connection to remote storage becomes available and then sends the buffered
data to the remote storage. It re-tries sending the data to remote storage until errors are resolved.
The maximum on-disk size for the buffered metrics can be limited with -remoteWrite.maxDiskUsagePerURL.
vmagent works on various architectures from the IoT world - 32-bit arm, 64-bit arm, ppc64, 386, amd64.
If you use Prometheus only for scraping metrics from various targets and forwarding these metrics to remote storage
then vmagent can replace Prometheus. Typically, vmagent requires lower amounts of RAM, CPU and network bandwidth compared with Prometheus.
See these docs for details.
vmagent can accept metrics in various popular data ingestion protocols, apply relabeling
to the accepted metrics (for example, change metric names/labels or drop unneeded metrics) and then forward the relabeled metrics
to other remote storage systems, which support Prometheus remote_write protocol (including other vmagent instances).
vmagent replicates the collected metrics among multiple remote storage instances configured via -remoteWrite.url args.
If a single remote storage instance temporarily is out of service, then the collected data remains available in another remote storage instance.
vmagent buffers the collected data in files at -remoteWrite.tmpDataPath until the remote storage becomes available again,
and then it sends the buffered data to the remote storage in order to prevent data gaps.
VictoriaMetrics cluster already supports replication,
so there is no need in specifying multiple -remoteWrite.url flags when writing data to the same cluster.
See these docs.
By default vmagent replicates data among remote storage systems enumerated via -remoteWrite.url command-line flag.
If the -remoteWrite.shardByURL command-line flag is set, then vmagent spreads evenly
the outgoing time series among all the remote storage systems
enumerated via -remoteWrite.url.
It is possible to replicate samples among remote storage systems by passing -remoteWrite.shardByURLReplicas=N
command-line flag to vmagent additionally to -remoteWrite.shardByURL command-line flag.
This instructs vmagent writing every outgoing sample to N distinct remote storage systems enumerated via -remoteWrite.url
in addition to sharding.
Samples for the same time series are routed to the same remote storage system if -remoteWrite.shardByURL flag is specified.
This allows building scalable data processing pipelines when a single remote storage cannot keep up with the data ingestion workload.
For example, this allows building horizontally scalable stream aggregation
by routing outgoing samples for the same time series of counter
and histogram types from top-level vmagent instances
to the same second-level vmagent instance, so they are aggregated properly.
If -remoteWrite.shardByURL command-line flag is set, then all the metric labels are used for even sharding
among remote storage systems specified in -remoteWrite.url.
Sometimes it may be needed to use only a particular set of labels for sharding. For example, it may be needed to route all the metrics with the same instance label
to the same -remoteWrite.url. In this case you can specify comma-separated list of these labels in the -remoteWrite.shardByURL.labels
command-line flag. For example, -remoteWrite.shardByURL.labels=instance,__name__ would shard metrics with the same name and instance
label to the same -remoteWrite.url.
Sometimes, it may be necessary to ignore some labels when sharding samples across multiple -remoteWrite.url backends.
For example, if all the raw samples with the same set of labels
except of instance and pod labels must be routed to the same backend. In this case the list of ignored labels must be passed to
-remoteWrite.shardByURL.ignoreLabels command-line flag: -remoteWrite.shardByURL.ignoreLabels=instance,pod.
vmagent can add, remove or update labels on the collected data before sending it to the remote storage. Additionally,
it can remove unwanted samples via Prometheus-like relabeling before sending the collected data to remote storage.
Please see these docs for details.
vmagent supports splitting the collected data between multiple destinations with the help of -remoteWrite.urlRelabelConfig,
which is applied independently for each configured -remoteWrite.url destination. For example, it is possible to replicate or split
data among long-term remote storage, short-term remote storage and a real-time analytical system built on top of Kafka.
Note that each destination can receive its own subset of the collected data due to per-destination relabeling via -remoteWrite.urlRelabelConfig.
For example, let’s assume all the scraped or received metrics by vmagent have label env with values dev or prod.
To route metrics env=dev to destination dev and metrics with env=prod to destination prod apply the following config:
Create relabeling config file relabelDev.yml to drop all metrics that don’t have label env=dev:
1
2
3
- action:keepsource_labels:[env]regex:"dev"
Create relabeling config file relabelProd.yml to drop all metrics that don’t have label env=prod:
1
2
3
- action:keepsource_labels:[env]regex:"prod"
Configure vmagent with 2 -remoteWrite.url flags pointing to destinations dev and prod with corresponding
-remoteWrite.urlRelabelConfig configs:
With this configuration vmagent will forward to http://<dev-url> only metrics that have env=dev label.
And to http://<prod-url> it will forward only metrics that have env=prod label.
Please note, order of flags is important: 1st mentioned -remoteWrite.urlRelabelConfig will be applied to the
1st mentioned -remoteWrite.url, and so on.
vmagent can be used as a proxy for Prometheus data sent via Prometheus remote_write protocol. It can accept data via the remote_write API
at the/api/v1/write endpoint. Then apply relabeling and filtering and proxy it to another remote_write system .
The vmagent can be configured to encrypt the incoming remote_write requests with -tls* command-line flags.
Also, Basic Auth can be enabled for the incoming remote_write requests with -httpAuth.* command-line flags.
While vmagent can accept data in several supported protocols (OpenTSDB, Influx, Prometheus, Graphite) and scrape data from various targets,
writes are always performed in Prometheus remote_write protocol. Therefore, for the clustered version,
the -remoteWrite.url command-line flag should be configured as <schema>://<vminsert-host>:8480/insert/<accountID>/prometheus/api/v1/write
according to these docs.
There is also support for multitenant writes. See these docs.
The following command instructs vmagent to merge time series with different replica label values
and then to send only the last sample per each merged series per every 60 seconds:
Specify the path to prometheus.yml file via -promscrape.config command-line flag. vmagent takes into account the following
sections from Prometheus config file:
The file pointed by -promscrape.config may contain %{ENV_VAR} placeholders which are substituted by the corresponding ENV_VAR environment variable values.
vmagent supports the following additional options in scrape_configs section:
headers - a list of HTTP headers to send to scrape target with each scrape request. This can be used when the scrape target
needs custom authorization and authentication. For example:
disable_compression: true for disabling response compression on a per-job basis. By default, vmagent requests compressed responses
from scrape targets for saving network bandwidth.
disable_keepalive: true for disabling HTTP keep-alive connections
on a per-job basis. By default, vmagent uses keep-alive connections to scrape targets for reducing overhead on connection re-establishing.
series_limit: N for limiting the number of unique time series a single scrape target can expose. See these docs.
stream_parse: true for scraping targets in a streaming manner. This may be useful when targets export big number of metrics. See these docs.
scrape_align_interval: duration for aligning scrapes to the given interval instead of using random offset
in the range [0 ... scrape_interval] for scraping each target. The random offset helps to spread scrapes evenly in time.
scrape_offset: duration for specifying the exact offset for scraping instead of using random offset in the range [0 ... scrape_interval].
vmagent supports loading scrape configs from multiple files specified
in the scrape_config_files section of -promscrape.config file. For example, the following -promscrape.config instructs vmagent
loading scrape configs from all the *.yml files under configs directory, from single_scrape_config.yml local file
and from https://config-server/scrape_config.yml url:
Every referred file can contain arbitrary number of supported scrape configs.
There is no need in specifying top-level scrape_configs section in these files. For example:
vmagent doesn’t support the following sections in Prometheus config file passed to -promscrape.config command-line flag:
remote_write. This section is substituted
with various -remoteWrite* command-line flags. See the full list of flags. The remote_write section isn’t supported
in order to reduce possible confusion when vmagent is used for accepting incoming metrics via supported push protocols.
In this case the -promscrape.config file isn’t needed.
remote_read. This section isn’t supported at all, since vmagent doesn’t provide Prometheus querying API.
It is expected that the querying API is provided by the remote storage specified via -remoteWrite.url such as VictoriaMetrics.
See Prometheus querying API docs for VictoriaMetrics.
rule_files and alerting. These sections are supported by vmalert.
The list of supported service discovery types is available here.
Additionally, vmagent doesn’t support refresh_interval option at service discovery sections.
This option is substituted with -promscrape.*CheckInterval command-line flags, which are specific per each service discovery type.
See the full list of command-line flags for vmagent.
vmagent should be restarted in order to update config options set via command-line args.
vmagent supports multiple approaches for reloading configs from updated config files such as
-promscrape.config, -remoteWrite.relabelConfig, -remoteWrite.urlRelabelConfig, -streamAggr.config
and -remoteWrite.streamAggr.config:
Sending SIGHUP signal to vmagent process:
1
kill -SIGHUP `pidof vmagent`
Sending HTTP request to http://vmagent:8429/-/reload endpoint. This endpoint can be protected with -reloadAuthKey command-line flag.
There is also -promscrape.configCheckInterval command-line flag, which can be used for automatic reloading configs from updated -promscrape.config file.
If vmagent encounters urls with srv+ prefix in hostname (such as http://srv+some-addr/some/path), then it resolves some-addrDNS SRV
record into TCP address with hostname and TCP port, and then uses the resulting url when it needs connecting to it.
SRV urls are supported in the following places:
In -remoteWrite.url command-line flag. For example, if victoria-metricsDNS SRV record contains
victoria-metrics-host:8428 TCP address, then -remoteWrite.url=http://srv+victoria-metrics/api/v1/write is automatically resolved into
-remoteWrite.url=http://victoria-metrics-host:8428/api/v1/write. If the DNS SRV record is resolved into multiple TCP addresses, then vmagent
uses randomly chosen address per each connection it establishes to the remote storage.
In scrape target addresses aka __address__ label - see these docs for details.
vmagent supports sending data to the configured -remoteWrite.url either via Prometheus remote write protocol
or via VictoriaMetrics remote write protocol.
VictoriaMetrics remote write protocol provides the following benefits comparing to Prometheus remote write protocol:
Reduced network bandwidth usage by 2x-5x. This allows saving network bandwidth usage costs when vmagent and
the configured remote storage systems are located in different datacenters, availability zones or regions.
Reduced disk read/write IO and disk space usage at vmagent when the remote storage is temporarily unavailable.
In this case vmagent buffers the incoming data to disk using the VictoriaMetrics remote write format.
This reduces disk read/write IO and disk space usage by 2x-5x comparing to Prometheus remote write format.
vmagent uses VictoriaMetrics remote write protocol by default
Available from the next releasewhen it sends data to VictoriaMetrics components such as other vmagent instances,
single-node VictoriaMetrics
or vminsert at cluster version. If needed, It can automatically downgrade to a Prometheus protocol at runtime.
It is possible to force switch to VictoriaMetrics remote write protocol by specifying -remoteWrite.forceVMProto
command-line flag for the corresponding -remoteWrite.url.
It is possible to tune the compression level for VictoriaMetrics remote write protocol with -remoteWrite.vmProtoCompressLevel command-line flag.
Bigger values reduce network usage at the cost of higher CPU usage. Negative values reduce CPU usage at the cost of higher network usage.
The default value for the compression level is 0, the minimum value is -22 and the maximum value is 22. The default value works optimally
in most cases, so it isn’t recommended changing it.
vmagent automatically switches to Prometheus remote write protocol when it sends data to old versions of VictoriaMetrics components
or to other Prometheus-compatible remote storage systems. It is possible to force switch to Prometheus remote write protocol
by specifying -remoteWrite.forcePromProto command-line flag for the corresponding -remoteWrite.url.
By default vmagent collects the data without tenant identifiers
and routes it to the remote storage specified via -remoteWrite.url command-line flag. The -remoteWrite.url can point to /insert/<tenant_id>/prometheus/api/v1/write path
at vminsert according to these docs. In this case all the metrics are written to the given <tenant_id> tenant.
The easiest way to write data to multiple distinct tenants is to specify the needed tenants via vm_account_id and vm_project_id labels
and then to push metrics with these labels to multitenant url at VictoriaMetrics cluster.
The vm_account_id and vm_project_id labels can be specified via relabeling before sending the metrics to -remoteWrite.url.
For example, the following relabeling rule instructs sending metrics to <account_id>:0tenant
defined in the prometheus.io/account_id annotation of Kubernetes pod deployment:
vmagent can accept data via the same multitenant endpoints (/insert/<accountID>/<suffix>) as vminsert at VictoriaMetrics cluster
does according to these docs if -enableMultitenantHandlers command-line flag is set.
In this case, vmagent automatically converts tenant identifiers from the URL to vm_account_id and vm_project_id labels.
These tenant labels are added before applying relabeling specified via -remoteWrite.relabelConfig
and -remoteWrite.urlRelabelConfig command-line flags. Metrics with vm_account_id and vm_project_id labels can be routed to the corresponding tenants
when specifying -remoteWrite.url to multitenant url at VictoriaMetrics cluster.
Extra labels can be added to metrics collected by vmagent via the following mechanisms:
The global -> external_labels section in -promscrape.config file. These labels are added only to metrics scraped from targets configured
in the -promscrape.config file. They aren’t added to metrics collected via other data ingestion protocols.
The -remoteWrite.label command-line flag. These labels are added to all the collected metrics before sending them to -remoteWrite.url.
For example, the following command starts vmagent, which adds {datacenter="foobar"} label to all the metrics pushed
to all the configured remote storage systems (all the -remoteWrite.url flag values):
vmagent automatically generates the following metrics per each scrape of every Prometheus-compatible target
and attaches instance, job and other target-specific labels to these metrics:
up - this metric exposes 1 value on successful scrape and 0 value on unsuccessful scrape. This allows monitoring
failing scrapes with the following MetricsQL query:
1
up == 0
scrape_duration_seconds - the duration of the scrape for the given target. This allows monitoring slow scrapes.
For example, the following MetricsQL query returns scrapes,
which take more than 1.5 seconds to complete:
1
scrape_duration_seconds > 1.5
scrape_timeout_seconds - the configured timeout for the current scrape target (aka scrape_timeout).
This allows detecting targets with scrape durations close to the configured scrape timeout.
For example, the following MetricsQL query returns targets (identified by instance label),
which take more than 80% of the configured scrape_timeout during scrapes:
scrape_response_size_bytes - response size in bytes for the given target. This allows to monitor amount of data scraped
and to adjust max_scrape_size option for scraped targets.
For example, the following MetricsQL query returns targets with scrape response
bigger than 10MiB:
1
scrape_response_size_bytes > 10MiB
scrape_samples_scraped - the number of samples parsed per each scrape. This allows detecting targets,
which expose too many series. For example, the following MetricsQL query
returns targets, which expose more than 10000 metrics:
1
scrape_samples_scraped > 10000
scrape_samples_limit - the configured limit on the number of samples the given target can expose.
The limit can be set via sample_limit option at scrape_configs.
This metric is exposed only if the sample_limit is set. This allows detecting targets,
which expose too many metrics compared to the configured sample_limit. For example, the following query
returns targets (identified by instance label), which expose more than 80% metrics compared to the configured sample_limit:
scrape_samples_post_metric_relabeling - the number of samples left after applying metric-level relabeling
from metric_relabel_configs section (see relabeling docs for more details).
This allows detecting targets with too many series after the relabeling.
For example, the following MetricsQL query returns targets
with more than 10000 metrics after the relabeling:
1
scrape_samples_post_metric_relabeling > 10000
scrape_series_added - an approximate number of new series the given target generates during the current scrape.
This metric allows detecting targets (identified by instance label),
which lead to high churn rate.
For example, the following MetricsQL query returns targets,
which generate more than 1000 new series during the last hour:
1
sum_over_time(scrape_series_added[1h]) > 1000
vmagent sets scrape_series_added to zero when it runs with -promscrape.noStaleMarkers command-line flag
or when it scrapes target with no_stale_markers: true option, e.g. when staleness markers are disabled.
scrape_series_limit - the limit on the number of unique series the given target can expose according to these docs.
This metric is exposed only if the series limit is set.
scrape_series_current - the number of unique series the given target exposed so far.
This metric is exposed only if the series limit is set according to these docs.
This metric allows alerting when the number of exposed series by the given target reaches the limit.
For example, the following query would alert when the target exposes more than 90% of unique series compared to the configured limit.
1
scrape_series_current / scrape_series_limit > 0.9
scrape_series_limit_samples_dropped - exposes the number of dropped samples during the scrape because of the exceeded limit
on the number of unique series. This metric is exposed only if the series limit is set according to these docs.
This metric allows alerting when scraped samples are dropped because of the exceeded limit.
For example, the following query alerts when at least a single sample is dropped because of the exceeded limit during the last hour:
If the target exports metrics with names clashing with the automatically generated metric names, then vmagent automatically
adds exported_ prefix to these metric names, so they don’t clash with automatically generated metric names.
Relabeling defined in relabel_configs or metric_relabel_configs of scrape config isn’t applied to automatically
generated metrics. But they still can be relabeled via -remoteWrite.relabelConfig before sending metrics to remote address.
At the global -> relabel_configs section in -promscrape.config file
Available from v1.106.0.
This relabeling is used for modifying labels in discovered targets and for dropping unneeded targets.
Configuration from global section will be prepended to the relabel_config of targets from scrape_config section.
See relabeling cookbook for details.
At the global -> metric_relabel_configs section in -promscrape.config file
Available from v1.106.0.
This relabeling is used for modifying labels in scraped metrics and for dropping unneeded metrics.
Configuration from global section will be prepended to the metric_relabel_config of targets from scrape_config section.
See relabeling cookbook for details.
At the scrape_config -> relabel_configs section in -promscrape.config file.
This relabeling is used for modifying labels in discovered targets and for dropping unneeded targets.
See relabeling cookbook for details.
This relabeling can be debugged by clicking the debug link at the corresponding target on the http://vmagent:8429/targets page
or on the http://vmagent:8429/service-discovery page. See these docs for details.
The link is unavailable if vmagent runs with -promscrape.dropOriginalLabels command-line flag.
At the scrape_config -> metric_relabel_configs section in -promscrape.config file.
This relabeling is used for modifying labels in scraped metrics and for dropping unneeded metrics.
See relabeling cookbook for details.
This relabeling can be debugged via http://vmagent:8429/metric-relabel-debug page. See these docs for details.
At the -remoteWrite.relabelConfig file. This relabeling is used for modifying labels for all the collected metrics
(including metrics obtained via push-based protocols) and for dropping unneeded metrics
before sending them to all the configured -remoteWrite.url addresses.
This relabeling can be debugged via http://vmagent:8429/metric-relabel-debug page. See these docs for details.
At the -remoteWrite.urlRelabelConfig files. This relabeling is used for modifying labels for metrics
and for dropping unneeded metrics before sending them to the particular -remoteWrite.url.
This relabeling can be debugged via http://vmagent:8429/metric-relabel-debug page. See these docs for details.
All the files with relabeling configs can contain special placeholders in the form %{ENV_VAR},
which are replaced by the corresponding environment variable values.
Streaming aggregation, if configured,
is performed after applying all the relabeling stages mentioned above.
The following articles contain useful information about Prometheus relabeling:
vmagent provides the following enhancements on top of Prometheus-compatible relabeling:
The replacement option can refer arbitrary labels via {{label_name}} placeholders.
Such placeholders are substituted with the corresponding label value. For example, the following relabeling rule
sets instance-job label value to host123-foo when applied to the metric with {instance="host123",job="foo"} labels:
An optional if filter can be used for conditional relabeling. The if filter may contain
arbitrary time series selector.
The action is performed only for samples, which match the provided if filter.
For example, the following relabeling rule keeps metrics matching foo{bar="baz"} series selector, while dropping the rest of metrics:
1
2
- if:'foo{bar="baz"}'action:keep
This is equivalent to less clear Prometheus-compatible relabeling rule:
The if option may contain more than one filter. In this case the action is performed if at least a single filter
matches the given sample.
For example, the following relabeling rule adds foo="bar" label to samples with job="foo" or instance="bar" labels:
The regex value can be split into multiple lines for improved readability and maintainability.
These lines are automatically joined with | char when parsed. For example, the following configs are equivalent:
VictoriaMetrics provides the following additional relabeling actions on top of standard actions
from the Prometheus relabeling:
replace_all replaces all the occurrences of regex in the values of source_labels with the replacement
and stores the results in the target_label. For example, the following relabeling config replaces all the occurrences
of - char in metric names with _ char (e.g. foo-bar-baz metric name is transformed into foo_bar_baz):
labelmap_all replaces all the occurrences of regex in all the label names with the replacement.
For example, the following relabeling config replaces all the occurrences of - char in all the label names
with _ char (e.g. foo-bar-baz label name is transformed into foo_bar_baz):
1
2
3
- action:labelmap_allregex:"-"replacement:"_"
keep_if_equal: keeps the entry if all the label values from source_labels are equal,
while dropping all the other entries. For example, the following relabeling config keeps targets
if they contain equal values for instance and host labels, while dropping all the other targets:
drop_if_equal: drops the entry if all the label values from source_labels are equal,
while keeping all the other entries. For example, the following relabeling config drops targets
if they contain equal values for instance and host labels, while keeping all the other targets:
keep_if_contains: keeps the entry if target_label contains all the label values listed in source_labels,
while dropping all the other entries. For example, the following relabeling config keeps targets
if __meta_consul_tags contains value from the required_consul_tag label:
drop_if_contains: drops the entry if target_label contains all the label values listed in source_labels,
while keeping all the other entries. For example, the following relabeling config drops targets
if __meta_consul_tag contains value from the denied_consul_tag label:
keep_metrics: keeps all the metrics with names matching the given regex,
while dropping all the other metrics. For example, the following relabeling config keeps metrics
with foo and bar names, while dropping all the other metrics:
1
2
- action:keep_metricsregex:"foo|bar"
drop_metrics: drops all the metrics with names matching the given regex, while keeping all the other metrics.
For example, the following relabeling config drops metrics with foo and bar names, while leaving all the other metrics:
1
2
- action:drop_metricsregex:"foo|bar"
graphite: applies Graphite-style relabeling to metric name. See these docs for details.
VictoriaMetrics components support action: graphite relabeling rules, which allow extracting various parts from Graphite-style metrics
into the configured labels with the syntax similar to Glob matching in statsd_exporter.
Note that the name field must be substituted with explicit __name__ option under labels section.
If __name__ option is missing under labels section, then the original Graphite-style metric name is left unchanged.
For example, the following relabeling rule generates requests_total{job="app42",instance="host124:8080"} metric
from app42.host123.requests.total Graphite-style metric:
Important notes about action: graphite relabeling rules:
The relabeling rule is applied only to metrics, which match the given match expression. Other metrics remain unchanged.
The * matches the maximum possible number of chars until the next dot or until the next part of the match expression whichever comes first.
It may match zero chars if the next char is ..
For example, match: "app*foo.bar" matches app42foo.bar and 42 becomes available to use at labels section via $1 capture group.
The $0 capture group matches the original metric name.
The relabeling rules are executed in order defined in the original config.
The action: graphite relabeling rules are easier to write and maintain than action: replace for labels extraction from Graphite-style metric names.
Additionally, the action: graphite relabeling rules usually work much faster than the equivalent action: replace rules.
vmagent and single-node VictoriaMetrics
provide the following tools for debugging target-level and metric-level relabeling:
Target-level debugging (e.g. relabel_configs section at scrape_configs)
can be performed by navigating to http://vmagent:8429/targets page (http://victoriametrics:8428/targets page for single-node VictoriaMetrics)
and clicking the debug target relabeling link at the target, which must be debugged.
The link is unavailable if vmagent runs with -promscrape.dropOriginalLabels command-line flag.
The opened page shows step-by-step results for the actual target relabeling rules applied to the discovered target labels.
The page shows also the target URL generated after applying all the relabeling rules.
The http://vmagent:8429/targets page shows only active targets. If you need to understand why some target
is dropped during the relabeling, then navigate to http://vmagent:8428/service-discovery page
(http://victoriametrics:8428/service-discovery for single-node VictoriaMetrics), find the dropped target
and click the debug link there. The link is unavailable if vmagent runs with -promscrape.dropOriginalLabels command-line flag.
The opened page shows step-by-step results for the actual relabeling rules, which result to target drop.
Metric-level debugging (e.g. metric_relabel_configs section at scrape_configs
can be performed by navigating to http://vmagent:8429/targets page (http://victoriametrics:8428/targets page for single-node VictoriaMetrics)
and clicking the debug metrics relabeling link at the target, which must be debugged.
The link is unavailable if vmagent runs with -promscrape.dropOriginalLabels command-line flag.
The opened page shows step-by-step results for the actual metric relabeling rules applied to the given target labels.
http://vmagent:8429/targets page, which contains information about all the targets, which are scraped at the moment.
This page helps answering the following questions:
Why some targets cannot be scraped? The last error column contains the reason why the given target cannot be scraped.
You can also click the endpoint link in order open the target url in your browser.
You can also click the response link in order to open the target url on behalf of vmagent. This may be helpful when vmagent
is located in some isolated network.
Which labels the particular target has? The labels column shows per-target labels. These labels are attached to all the metrics
scraped from the given target. You can also click on the target labels in order to see the original labels of the target
before applying the relabeling. The original labels are unavailable if vmagent runs with -promscrape.dropOriginalLabels command-line flag.
Why the given target has the given set of labels? Click the target link at debug relabeling column for the particular target in order to see step-by-step
execution of target relabeling rules applied to the original labels. This link is unavailable if vmagent runs
with -promscrape.dropOriginalLabels command-line flag.
How the given metrics relabeling rules are applied to scraped metrics? Click the metrics link at debug relabeling column
for the particular target in order to see step-by-step execution of metric relabeling rules applied to the scraped metrics.
How many failed scrapes were for the particular target? The errors column shows this value.
How many metrics the given target exposes? The samples column shows the number of metrics scraped per each target during the last scrape.
How long does it take to scrape the given target? The duration column shows last scrape duration per each target.
When was the last scrape for the given target? The last scrape column shows the last time the given target was scraped.
How many times the given target was scraped? The scrapes column shows this information.
What is the current state of the particular target? The state column shows the current state of the particular target.
http://vmagent:8429/service-discovery page, which contains information about all the discovered targets.
This page doesn’t work if vmagent runs with -promscrape.dropOriginalLabels command-line flag.
This pages helps answering the following questions:
Why some targets are dropped during service discovery? Click debug link at debug relabeling on the dropped target in order to see step-by-step
execution of target relabeling rules applied to the original labels of discovered target.
Why some targets contain unexpected labels? Click debug link at debug relabeling on the dropped target in order to see step-by-step
execution of target relabeling rules applied to the original labels of discovered target.
What were the original labels before relabeling for a particular target? The discovered labels column contains the original labels per each discovered target.
If the metric disappears from the list of scraped metrics, then stale marker is sent to this particular metric.
If the scrape target becomes temporarily unavailable, then stale markers are sent for all the metrics scraped from this target.
If the scrape target is removed from the list of targets, then stale markers are sent for all the metrics scraped from this target.
Prometheus staleness markers’ tracking needs additional memory, since it must store the previous response body per each scrape target
in order to compare it to the current response body. The memory usage may be reduced by disabling staleness tracking in the following ways:
By passing -promscrape.noStaleMarkers command-line flag to vmagent. This disables staleness tracking across all the targets.
By specifying no_stale_markers: true option in the scrape_config for the corresponding target.
When staleness tracking is disabled, then vmagent doesn’t track the number of new time series per each scrape,
e.g. it sets scrape_series_added metric to zero. See these docs for details.
By default, vmagent parses the full response from the scrape target, applies relabeling
and then pushes the resulting metrics to the configured -remoteWrite.url in one go. This mode works good for the majority of cases
when the scrape target exposes small number of metrics (e.g. less than 10K). But this mode may take big amounts of memory
when the scrape target exposes big number of metrics (for example, when vmagent scrapes kube-state-metrics
in large Kubernetes cluster). It is recommended enabling stream parsing mode for such targets.
When this mode is enabled, vmagent processes the response from the scrape target in chunks.
This allows saving memory when scraping targets that expose millions of metrics.
Stream parsing mode is automatically enabled for scrape targets returning response bodies with sizes bigger than
the -promscrape.minResponseSizeForStreamParse command-line flag value. Additionally,
stream parsing mode can be explicitly enabled in the following places:
Via -promscrape.streamParse command-line flag. In this case all the scrape targets defined
in the file pointed by -promscrape.config are scraped in stream parsing mode.
Via stream_parse: true option at scrape_configs section. In this case all the scrape targets defined
in this section are scraped in stream parsing mode.
Via __stream_parse__=true label, which can be set via relabeling at relabel_configs section.
In this case stream parsing mode is enabled for the corresponding scrape targets.
Typical use case: to set the label via Kubernetes annotations
for targets exposing big number of metrics.
Note that vmagent in stream parsing mode stores up to sample_limit samples to the configured -remoteStorage.url
instead of dropping all the samples read from the target, because the parsed data is sent to the remote storage
as soon as it is parsed in stream parsing mode.
A single vmagent instance can scrape tens of thousands of scrape targets. Sometimes this isn’t enough due to limitations on CPU, network, RAM, etc.
In this case scrape targets can be split among multiple vmagent instances (aka vmagent horizontal scaling, sharding and clustering).
The number of vmagent instances in the cluster must be passed to -promscrape.cluster.membersCount command-line flag.
Each vmagent instance in the cluster must use identical -promscrape.config files with distinct -promscrape.cluster.memberNum values
in the range 0 ... N-1, where N is the number of vmagent instances in the cluster specified via -promscrape.cluster.membersCount.
For example, the following commands spread scrape targets among a cluster of two vmagent instances:
The -promscrape.cluster.memberNum can be set to a StatefulSet pod name when vmagent runs in Kubernetes.
The pod name must end with a number in the range 0 ... promscrape.cluster.membersCount-1. For example, -promscrape.cluster.memberNum=vmagent-0.
By default, each scrape target is scraped only by a single vmagent instance in the cluster. If there is a need for replicating scrape targets among multiple vmagent instances,
then -promscrape.cluster.replicationFactor command-line flag must be set to the desired number of replicas. For example, the following commands
start a cluster of three vmagent instances, where each target is scraped by two vmagent instances:
Every vmagent in the cluster exposes all the discovered targets at http://vmagent:8429/service-discovery page.
Each discovered target on this page contains its status (UP, DOWN or DROPPED with the reason why the target has been dropped).
If the target is dropped because of sharding to other vmagent instances in the cluster, then the status column contains
-promscrape.cluster.memberNum values for vmagent instances where the given target is scraped.
The /service-discovery page provides links to the corresponding vmagent instances if -promscrape.cluster.memberURLTemplate command-line flag is set.
Every occurrence of %d inside the -promscrape.cluster.memberURLTemplate is substituted with the -promscrape.cluster.memberNum
for the corresponding vmagent instance. For example, -promscrape.cluster.memberURLTemplate='http://vmagent-instance-%d:8429/targets'
generates http://vmagent-instance-42:8429/targets url for vmagent instance, which runs with -promscrape.cluster.memberNum=42.
Note that vmagent shows up to -promscrape.maxDroppedTargets dropped targets on the /service-discovery page.
Increase the -promscrape.maxDroppedTargets command-line flag value if the /service-discovery page misses some dropped targets.
If each target is scraped by multiple vmagent instances, then data deduplication must be enabled at remote storage pointed by -remoteWrite.url.
The -dedup.minScrapeInterval must be set to the scrape_interval configured at -promscrape.config.
See these docs for details.
The -promscrape.cluster.memberLabel command-line flag allows specifying a name for member num label to add to all the scraped metrics.
The value of the member num label is set to -promscrape.cluster.memberNum. For example, the following config instructs adding vmagent_instance="0" label
to all the metrics scraped by the given vmagent instance:
It is possible to run multiple identically configuredvmagent instances or vmagentclusters, so they scrape
the same set of targets and push the collected data to the same set of VictoriaMetrics remote storage systems.
Two identically configured vmagent instances or clusters is usually called an HA pair.
When running HA pairs, deduplication must be configured
at VictoriaMetrics side in order to de-duplicate received samples.
See these docs for details.
It is also recommended passing different values to -promscrape.cluster.name command-line flag per each vmagent
instance or per each vmagent cluster in HA setup. This is needed for proper data de-duplication.
See this issue for details.
vmagent supports scraping targets via http, https and socks5 proxies. Proxy address must be specified in proxy_url option. For example, the following scrape config instructs
target scraping via https proxy at https://proxy-addr:1234:
vmagent stores pending data that cannot be sent to the configured remote storage systems in a timely manner.
By default, vmagent writes all the pending data to folder configured via -remoteWrite.tmpDataPath cmd-line flag
until this data is sent to the configured -remoteWrite.url systems or until the folder becomes full.
The maximum data size that can be saved to -remoteWrite.tmpDataPath per every configured -remoteWrite.url can be
limited via -remoteWrite.maxDiskUsagePerURL command-line flag. When this limit is reached, vmagent drops the oldest
data from disk in order to save newly ingested data.
The folder structure of persistence data is as follows:
vmagent will generate the following persistent queue folders:
1
2
3
4
# 1_<hash(http://example-1:8428/prometheus/api/v1/write)>, query parameters foo=bar and fragment baz are removed.1_BA6E4303DCFA0D45
# 2_<hash(http://user:pass@example-2:8428/prometheus/api/v1/write)>, query parameters qux=quux and fragment quuz are removed.2_0AAFDF53E314A72A
When it is better to drop pending data instead of buffering it.
In this case -remoteWrite.disableOnDiskQueue command-line flag can be passed to vmagent per each configured -remoteWrite.url.
vmagent works in the following way if the corresponding remote storage system at -remoteWrite.url cannot keep up with the data ingestion rate
and the -remoteWrite.disableOnDiskQueue command-line flag is set:
It returns 429 Too Many Requests HTTP error to clients, which send data to vmagent via supported HTTP endpoints.
If -remoteWrite.dropSamplesOnOverload command-line flag is set or if multiple -remoteWrite.disableOnDiskQueue command-line flags are set
for different -remoteWrite.url options, then the ingested samples are silently dropped instead of returning the error to clients.
It suspends consuming data from Kafka side or Google PubSub side until the remote storage becomes available.
If -remoteWrite.dropSamplesOnOverload command-line flag is set or if multiple -remoteWrite.disableOnDiskQueue command-line flags are set
for different -remoteWrite.url options, then the fetched samples are silently dropped instead of suspending data consumption from Kafka or Google PubSub.
It drops samples pushed to vmagent via non-HTTP protocols and logs the error. Pass -remoteWrite.dropSamplesOnOverload command-line flag in order
to suppress error messages in this case.
It drops stream aggregation output samples, because it is better from operations perspective
to drop output samples instead of blocking the stream aggregation process.
The number of dropped samples because of overloaded remote storage can be monitored via vmagent_remotewrite_samples_dropped_total metric.
The number of unsuccessful attempts to send data to overloaded remote storage can be monitored via vmagent_remotewrite_push_failures_total metric.
Running vmagent on hosts with more RAM or increasing the value for -memory.allowedPercent may reduce the number of unsuccessful attempts or dropped samples
on spiky workloads, since vmagent may buffer more data in memory before returning the error or dropping data.
vmagent still may write pending in-memory data to -remoteWrite.tmpDataPath on graceful shutdown
if -remoteWrite.disableOnDiskQueue command-line flag is specified. It may also read buffered data from -remoteWrite.tmpDataPath
on startup.
When -remoteWrite.disableOnDiskQueue command-line flag is set, vmagent may send the same samples multiple times to the configured remote storage
if it cannot keep up with the data ingestion rate. In this case the deduplication
must be enabled on all the configured remote storage systems.
By default, vmagent doesn’t limit the number of time series each scrape target can expose.
The limit can be enforced in the following places:
Via -promscrape.seriesLimitPerTarget command-line flag. This limit is applied individually
to all the scrape targets defined in the file pointed by -promscrape.config.
Via series_limit config option at scrape_config section.
The series_limit allows overriding the -promscrape.seriesLimitPerTarget on a per-scrape_config basis.
If series_limit is set to 0 or to negative value, then it isn’t applied to the given scrape_config,
even if -promscrape.seriesLimitPerTarget command-line flag is set.
Via __series_limit__ label, which can be set with relabeling at relabel_configs section.
The __series_limit__ allows overriding the series_limit on a per-target basis.
If __series_limit__ is set to 0 or to negative value, then it isn’t applied to the given target.
Typical use case: to set the limit via Kubernetes annotations
for targets, which may expose too high number of time series.
Scraped metrics are dropped for time series exceeding the given limit on the time window of 24h.
vmagent creates the following additional per-target metrics for targets with non-zero series limit:
scrape_series_limit_samples_dropped - the number of dropped samples during the scrape when the unique series limit is exceeded.
scrape_series_limit - the series limit for the given target.
scrape_series_current - the current number of series for the given target.
These metrics are automatically sent to the configured -remoteWrite.url alongside with the scraped per-target metrics.
These metrics allow building the following alerting rules:
scrape_series_current / scrape_series_limit > 0.9 - alerts when the number of series exposed by the target reaches 90% of the limit.
sum_over_time(scrape_series_limit_samples_dropped[1h]) > 0 - alerts when some samples are dropped because the series limit on a particular target is reached.
By default, vmagent doesn’t limit the number of time series written to remote storage systems specified at -remoteWrite.url.
The limit can be enforced by setting the following command-line flags:
-remoteWrite.maxHourlySeries - limits the number of unique time series vmagent can write to remote storage systems during the last hour.
Useful for limiting the number of active time series.
-remoteWrite.maxDailySeries - limits the number of unique time series vmagent can write to remote storage systems during the last day.
Useful for limiting daily churn rate.
Both limits can be set simultaneously. If any of these limits is reached, then samples for new time series are dropped instead of sending
them to remote storage systems. A sample of dropped series is put in the log with WARNING level.
vmagent exposes the following metrics at http://vmagent:8429/metrics page (see monitoring docs for details):
vmagent_hourly_series_limit_rows_dropped_total - the number of metrics dropped due to exceeded hourly limit on the number of unique time series.
vmagent_hourly_series_limit_max_series - the hourly series limit set via -remoteWrite.maxHourlySeries.
vmagent_hourly_series_limit_current_series - the current number of unique series registered during the last hour.
vmagent_daily_series_limit_rows_dropped_total - the number of metrics dropped due to exceeded daily limit on the number of unique time series.
vmagent_daily_series_limit_max_series - the daily series limit set via -remoteWrite.maxDailySeries.
vmagent_daily_series_limit_current_series - the current number of unique series registered during the last day.
These limits are approximate, so vmagent can underflow/overflow the limit by a small percentage (usually less than 1%).
vmagent exports various metrics in Prometheus exposition format at http://vmagent-host:8429/metrics page.
We recommend setting up regular scraping of this page either through vmagent itself or by Prometheus-compatible scraper,
so that the exported metrics may be analyzed later.
If you use Google Cloud Managed Prometheus for scraping metrics from VictoriaMetrics components, then pass -metrics.exposeMetadata
command-line to them, so they add TYPE and HELP comments per each exposed metric at /metrics page.
See these docs for details.
Use official Grafana dashboard for vmagent state overview.
Graphs on this dashboard contain useful hints - hover the i icon at the top left corner of each graph in order to read it.
If you have suggestions for improvements or have found a bug - please open an issue on github or add a review to the dashboard.
vmagent also exports the status for various targets at the following pages:
http://vmagent-host:8429/targets. This pages shows the current status for every active target.
http://vmagent-host:8429/service-discovery. This pages shows the list of discovered targets with the discovered __meta_* labels
according to these docs.
This page may help debugging target relabeling.
http://vmagent-host:8429/ready. This handler returns http 200 status code when vmagent finishes
its initialization for all the service_discovery configs.
It may be useful to perform vmagent rolling update without any scrape loss.
It is recommended increasing the maximum number of open files in the system (ulimit -n) when scraping a big number of targets,
as vmagent establishes at least a single TCP connection per target.
When vmagent scrapes many unreliable targets, it can flood the error log with scrape errors. It is recommended investigating and fixing these errors.
If it is unfeasible to fix all the reported errors, then they can be suppressed by passing -promscrape.suppressScrapeErrors command-line flag to vmagent.
The most recent scrape error per each target can be observed at http://vmagent-host:8429/targets and http://vmagent-host:8429/api/v1/targets.
The http://vmagent-host:8429/service-discovery page could be useful for debugging relabeling process for scrape targets.
This page contains original labels for targets dropped during relabeling.
By default, the -promscrape.maxDroppedTargets targets are shown here. If your setup drops more targets during relabeling,
then increase -promscrape.maxDroppedTargets command-line flag value to see all the dropped targets.
Note that tracking each dropped target requires up to 10Kb of RAM. Therefore, big values for -promscrape.maxDroppedTargets
may result in increased memory usage if a big number of scrape targets are dropped during relabeling.
It is recommended increasing -remoteWrite.queues if vmagent_remotewrite_pending_data_bytesmetric
grows constantly. It is also recommended increasing -remoteWrite.maxBlockSize and -remoteWrite.maxRowsPerBlock command-line flags in this case.
This can improve data ingestion performance to the configured remote storage systems at the cost of higher memory usage.
If you see gaps in the data pushed by vmagent to remote storage when -remoteWrite.maxDiskUsagePerURL is set,
try increasing -remoteWrite.queues. Such gaps may appear because vmagent cannot keep up with sending the collected data to remote storage.
Therefore, it starts dropping the buffered data if the on-disk buffer size exceeds -remoteWrite.maxDiskUsagePerURL.
vmagent drops data blocks if remote storage replies with 400 Bad Request and 409 Conflict HTTP responses.
The number of dropped blocks can be monitored via vmagent_remotewrite_packets_dropped_total metric exported at /metrics page.
Use -remoteWrite.queues=1 when -remoteWrite.url points to remote storage, which doesn’t accept out-of-order samples (aka data backfilling).
Such storage systems include Prometheus, Mimir, Cortex and Thanos, which typically emit out of order sample errors.
The best solution is to use remote storage with backfilling support such as VictoriaMetrics.
vmagent buffers scraped data at the -remoteWrite.tmpDataPath directory until it is sent to -remoteWrite.url.
The directory can grow large when remote storage is unavailable for extended periods of time and if the maximum directory size isn’t limited
with -remoteWrite.maxDiskUsagePerURL command-line flag.
If you don’t want to send all the buffered data from the directory to remote storage then simply stop vmagent and delete the directory.
If vmagent runs on a host with slow persistent storage, which cannot keep up with the volume of processed samples, then it is possible to disable
the persistent storage with -remoteWrite.disableOnDiskQueue command-line flag. See these docs for more details.
By default vmagent masks -remoteWrite.url with secret-url values in logs and at /metrics page because
the url may contain sensitive information such as auth tokens or passwords.
Pass -remoteWrite.showURL command-line flag when starting vmagent in order to see all the valid urls.
By default vmagent evenly spreads scrape load in time. If a particular scrape target must be scraped at the beginning of some interval,
then scrape_align_interval option must be used. For example, the following config aligns hourly scrapes to the beginning of hour:
By default vmagent evenly spreads scrape load in time. If a particular scrape target must be scraped at specific offset, then scrape_offset option must be used.
For example, the following config instructs vmagent to scrape the target at 10 seconds of every minute:
If you see skipping duplicate scrape target with identical labels errors when scraping Kubernetes pods, then it is likely these pods listen to multiple ports
or they use an init container. These errors can either be fixed or suppressed with the -promscrape.suppressDuplicateScrapeTargetErrors command-line flag.
See the available options below if you prefer fixing the root cause of the error:
The following relabeling rule may be added to relabel_configs section in order to filter out pods with unneeded ports:
vmagent buffers collected metrics on disk at the directory specified via -remoteWrite.tmpDataPath command-line flag
until the metrics are sent to remote storage configured via -remoteWrite.url command-line flag.
The -remoteWrite.tmpDataPath directory can grow large when remote storage is unavailable for extended
periods of time and if the maximum directory size isn’t limited with -remoteWrite.maxDiskUsagePerURL command-line flag.
To estimate the allocated disk size for persistent queue, or to estimate -remoteWrite.maxDiskUsagePerURL command-line flag value,
take into account the following attributes:
The size in bytes of data stream sent by vmagent:
Run the query sum(rate(vmagent_remotewrite_bytes_sent_total[1h])) by(instance,url) in vmui
or Grafana to get the amount of bytes sent by each vmagent instance per second.
The amount of time a persistent queue should keep the data before starting to drop it.
For example, if vmagent should be able to buffer the data for at least 6 hours, then the following query
can be used for estimating the needed amounts of disk space in gigabytes:
Enterprise version of vmagent can read metrics in various formats from Google PubSub messages.
-gcp.pubsub.subscribe.defaultMessageFormat and -gcp.pubsub.subscribe.topicSubscription.messageFormat command-line flags allow configuring the needed message format.
The following message formats are supported:
Every PubSub message may contain multiple lines in influx, prometheus, graphite and jsonline format delimited by \n.
vmagent consumes messages from PubSub topic subscriptions specified by -gcp.pubsub.subscribe.topicSubscription command-line flag.
Multiple topics can be specified by passing multiple -gcp.pubsub.subscribe.topicSubscription command-line flags to vmagent.
vmagent uses standard Google authorization mechanism for topic access. It’s possible to specify credentials directly via -gcp.pubsub.subscribe.credentialsFile command-line flag.
For example, the following command starts vmagent, which reads metrics in InfluxDB line protocol format
from PubSub projects/victoriametrics-vmagent-pub-sub-test/subscriptions/telegraf-testing and sends them to remote storage at http://localhost:8428/api/v1/write:
It is expected that Telegraf sends metrics to the telegraf-testing topic at the victoriametrics-vmagent-pub-sub-test project
with the following config:
vmagent buffers messages read from Google PubSub topic on local disk if the remote storage at -remoteWrite.url cannot keep up with the data ingestion rate.
In this case it may be useful to disable on-disk data persistence in order to prevent from unbounded growth of the on-disk queue.
See these docs.
vmagent can read messages from different topics in different formats. For example, the following command starts vmagent, which reads plaintext
Influx messages from telegraf-testing topic
and gzipped JSON line messages from json-line-testing topic:
These command-line flags are available only in enterprise version of vmagent,
which can be downloaded for evaluation from releases page
(see vmutils-...-enterprise.tar.gz archives) and from docker images Docker Hub and Quay with tags containing enterprise suffix.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
-gcp.pubsub.subscribe.credentialsFile string
Path to file with GCP credentials to use for PubSub client. If not set, default credentials are used (see Workload Identity for K8S or https://cloud.google.com/docs/authentication/application-default-credentials ). See https://docs.victoriametrics.com/vmagent/#reading-metrics-from-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
-gcp.pubsub.subscribe.defaultMessageFormat string
Default message format if -gcp.pubsub.subscribe.topicSubscription.messageFormat is missing. See https://docs.victoriametrics.com/vmagent/#reading-metrics-from-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ (default "promremotewrite") -gcp.pubsub.subscribe.topicSubscription array
GCP PubSub topic subscription in the format: projects/<project-id>/subscriptions/<subscription-name>. See https://docs.victoriametrics.com/vmagent/#reading-metrics-from-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Supports an array of values separated by comma or specified via multiple flags.
-gcp.pubsub.subscribe.topicSubscription.concurrency array
The number of concurrently processed messages for topic subscription specified via -gcp.pubsub.subscribe.topicSubscription flag. See https://docs.victoriametrics.com/vmagent/#reading-metrics-from-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ (default 0) Supports array of values separated by comma or specified via multiple flags.
-gcp.pubsub.subscribe.topicSubscription.isGzipped array
Enables gzip decompression for messages payload at the corresponding -gcp.pubsub.subscribe.topicSubscription. Only prometheus, jsonline, graphite and influx formats accept gzipped messages. See https://docs.victoriametrics.com/vmagent/#reading-metrics-from-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Supports array of values separated by comma or specified via multiple flags.
-gcp.pubsub.subscribe.topicSubscription.messageFormat array
Message format for the corresponding -gcp.pubsub.subscribe.topicSubscription. Valid formats: influx, prometheus, promremotewrite, graphite, jsonline . See https://docs.victoriametrics.com/vmagent/#reading-metrics-from-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Supports an array of values separated by comma or specified via multiple flags.
Enterprise version of vmagent writes data into Google PubSub if -remoteWrite.url command-line flag starts with pubsub: prefix.
For example, -remoteWrite.url=pubsub:projects/victoriametrics-vmagent-publish-test/topics/testing-pubsub-push.
These messages can be read later from Google PubSub by another vmagent instance - see these docs for details.
vmagent uses standard Google authorization mechanism for PubSub topic access. Custom auth credentials can be specified via -gcp.pubsub.subscribe.credentialsFile command-line flag.
These command-line flags are available only in enterprise version of vmagent,
which can be downloaded for evaluation from releases page
(see vmutils-...-enterprise.tar.gz archives) and from docker images Docker Hub and Quay with tags containing enterprise suffix.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
-gcp.pubsub.publish.byteThreshold int
Publish a batch when its size in bytes reaches this value. See https://docs.victoriametrics.com/vmagent/#writing-metrics-to-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ (default 1000000) -gcp.pubsub.publish.countThreshold int
Publish a batch when it has this many messages. See https://docs.victoriametrics.com/vmagent/#writing-metrics-to-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ (default 100) -gcp.pubsub.publish.credentialsFile string
Path to file with GCP credentials to use for PubSub client. If not set, default credentials will be used (see Workload Identity for K8S or https://cloud.google.com/docs/authentication/application-default-credentials). See https://docs.victoriametrics.com/vmagent/#writing-metrics-to-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
-gcp.pubsub.publish.delayThreshold value
Publish a non-empty batch after this delay has passed. See https://docs.victoriametrics.com/vmagent/#writing-metrics-to-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
The following optional suffixes are supported: s (second), h (hour), d (day), w (week), y (year). If suffix isn't set, then the duration is counted in months (default 10ms)
-gcp.pubsub.publish.maxOutstandingBytes int
The maximum size of buffered messages to be published. If less than or equal to zero, this is disabled. See https://docs.victoriametrics.com/vmagent/#writing-metrics-to-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ (default -1)
-gcp.pubsub.publish.maxOutstandingMessages int
The maximum number of buffered messages to be published. If less than or equal to zero, this is disabled. See https://docs.victoriametrics.com/vmagent/#writing-metrics-to-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ (default 100)
-gcp.pubsub.publish.timeout value
The maximum time that the client will attempt to publish a bundle of messages. See https://docs.victoriametrics.com/vmagent/#writing-metrics-to-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
The following optional suffixes are supported: s (second), h (hour), d (day), w (week), y (year). If suffix isn't set, then the duration is counted in months (default 60s)
The enterprise version of vmagent is available for evaluation at releases page
in vmutils-...-enterprise.tar.gz archives and in docker images Docker Hub and Quay with tags containing enterprise suffix.
See how to request a free trial license here.
Enterprise version of vmagent can read metrics in various formats from Kafka messages.
These formats can be configured with -kafka.consumer.topic.defaultFormat or -kafka.consumer.topic.format command-line flags. The following formats are supported:
Every Kafka message may contain multiple lines in influx, prometheus, graphite and jsonline format delimited by \n.
vmagent consumes messages from Kafka topics specified by -kafka.consumer.topic command-line flag. Multiple topics can be specified
by passing multiple -kafka.consumer.topic command-line flags to vmagent.
vmagent consumes messages from Kafka brokers specified by -kafka.consumer.topic.brokers command-line flag.
Multiple brokers can be specified per each -kafka.consumer.topic by passing a list of brokers delimited by ;.
For example:
This command starts vmagent which reads messages from topic-a at host1:9092 and host2:9092 brokers and messages
from topic-b at host3:9092 and host4:9092 brokers.
Note that when using YAML configuration (for example, when using Helm charts or Kubernetes operator)
keys provided in extraArgs must be unique, so in order to achieve the same configuration as in the example above, the following configuration must be used:
Note that list of brokers for the same topic is separated by ; and different groups of brokers are separated by ,.
The following command starts vmagent, which reads metrics in InfluxDB line protocol format from Kafka broker at localhost:9092
from the topic metrics-by-telegraf and sends them to remote storage at http://localhost:8428/api/v1/write:
vmagent buffers messages read from Kafka topic on local disk if the remote storage at -remoteWrite.url cannot keep up with the data ingestion rate.
In this case it may be useful to disable on-disk data persistence in order to prevent from unbounded growth of the on-disk queue.
See these docs.
These command-line flags are available only in enterprise version of vmagent,
which can be downloaded for evaluation from releases page
(see vmutils-...-enterprise.tar.gz archives) and from docker images Docker Hub and Quay with tags containing enterprise suffix.
-kafka.consumer.topic array
Kafka topic names for data consumption. See https://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Supports an array of values separated by comma or specified via multiple flags.
-kafka.consumer.topic.basicAuth.password array
Optional basic auth password for -kafka.consumer.topic. Must be used in conjunction with any supported auth methods for kafka client, specified by flag -kafka.consumer.topic.options='security.protocol=SASL_SSL;sasl.mechanisms=PLAIN' . See https://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Supports an array of values separated by comma or specified via multiple flags.
-kafka.consumer.topic.basicAuth.username array
Optional basic auth username for -kafka.consumer.topic. Must be used in conjunction with any supported auth methods for kafka client, specified by flag -kafka.consumer.topic.options='security.protocol=SASL_SSL;sasl.mechanisms=PLAIN' . See https://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Supports an array of values separated by comma or specified via multiple flags.
-kafka.consumer.topic.brokers array
List of brokers to connect for given topic, e.g. -kafka.consumer.topic.broker=host-1:9092;host-2:9092 . See https://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Supports an array of values separated by comma or specified via multiple flags.
-kafka.consumer.topic.concurrency array
Configures consumer concurrency for topic specified via -kafka.consumer.topic flag. See https://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ (default 1) Supports array of values separated by comma or specified via multiple flags.
-kafka.consumer.topic.defaultFormat string
Expected data format in the topic if -kafka.consumer.topic.format is skipped. See https://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ (default "promremotewrite") -kafka.consumer.topic.format array
data format for corresponding kafka topic. Valid formats: influx, prometheus, promremotewrite, graphite, jsonline . See https://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Supports an array of values separated by comma or specified via multiple flags.
-kafka.consumer.topic.groupID array
Defines group.id for topic. See https://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Supports an array of values separated by comma or specified via multiple flags.
-kafka.consumer.topic.isGzipped array
Enables gzip setting for topic messages payload. Only prometheus, jsonline, graphite and influx formats accept gzipped messages.See https://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Supports array of values separated by comma or specified via multiple flags.
-kafka.consumer.topic.options array
Optional key=value;key1=value2 settings for topic consumer. See full configuration options at https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md . See https://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Supports an array of values separated by comma or specified via multiple flags.
Enterprise version of vmagent writes data to Kafka with at-least-once
semantics if -remoteWrite.url contains e.g. Kafka url. For example, if vmagent is started with -remoteWrite.url=kafka://localhost:9092/?topic=prom-rw,
then it would send Prometheus remote_write messages to Kafka bootstrap server at localhost:9092 with the topic prom-rw.
These messages can be read later from Kafka by another vmagent - see these docs for details.
Additional Kafka options can be passed as query params to -remoteWrite.url. For instance, kafka://localhost:9092/?topic=prom-rw&client.id=my-favorite-id
sets client.id Kafka option to my-favorite-id. The full list of Kafka options is available here.
By default, vmagent sends compressed messages using Google’s Snappy, as defined in the Prometheus remote write protocol.
To switch to the VictoriaMetrics remote write protocol and reduce network bandwidth,
simply set the -remoteWrite.forceVMProto=true flag. It is also possible to adjust the compression level for the VictoriaMetrics remote write protocol using the -remoteWrite.vmProtoCompressLevel
command-line flag.
If you are migrating from remote write to Kafka, the request rate and request body size of remote write can roughly correspond to the message rate and size of Kafka.
vmagent organizes scraped/ingested data into blocks. A block contains multiple time series and samples.
Each block is compressed with Snappy or zstd before being sent out by the remote write or the Kafka producer.
In order to get the request rate of remote write (as the estimated produce rate of Kafka), use this MetricsQL:
Please note that the remote write body and Kafka message need to use the same compression algorithm in order to serve as
estimation references. See more in the VictoriaMetrics remote write protocol.
By default vmagent accepts http requests at 8429 port (this port can be changed via -httpListenAddr command-line flags),
since it is expected it runs in an isolated trusted network.
Enterprise version of vmagent supports the ability to accept mTLS
requests at this port, by specifying -tls and -mtls command-line flags. For example, the following command runs vmagent, which accepts only mTLS requests at port 8429:
1
./vmagent -tls -mtls -remoteWrite.url=...
By default, system-wide TLS Root CA is used for verifying client certificates if -mtls command-line flag is specified.
It is possible to specify custom TLS Root CA via -mtlsCAFile command-line flag.
vmagent is optimized for low CPU usage and low RAM usage without the need to tune any configs. Sometimes it is needed to optimize CPU / RAM usage of vmagent even more.
For example, if vmagent needs to scrape thousands of targets in resource-constrained environments. Then the following options may help reducing CPU usage and RAM usage of vmagent:
Set GOGC environment variable to 100. This reduces CPU usage at the cost of higher RAM usage.
Set GOMAXPROCS environment variable to the value slightly bigger than the number of CPU cores used by vmagent.
Another option is to set CPU limit in Kubernetes / Docker to the integer value bigger than the number of CPU cores used by vmagent.
This reduces RAM usage and CPU usage when vmagent runs in an environment with bigger number of available CPU cores. Note that it may be needed to increase the -remoteWrite.queues
command-line flag to bigger values if GOMAXPROCS is set to too small values, since by default -remoteWrite.queues is proportional to GOMAXPROCS.
Disable response compression at scrape targets via -promscrape.disableCompression command-line flag or via disable_compression: true option
in the scrape_config. This reduces CPU usage at the cost of higher network bandwidth usage
between vmagent and scrape targets.
Disable tracking of original labels for the discovered targets via -promscrape.dropOriginalLabels command-line flag. This helps reducing RAM usage when vmagent
discovers big number of scrape targets and the majority of these targets are dropped.
This is a typical case when vmagent discovers Kubernetes targets. The downside of using -promscrape.dropOriginalLabels command-line flag
is the reduced debuggability for improperly configured per-target relabeling.
Disable staleness markers via -promscrape.noStaleMarkers command-line flag
or via no_stale_markers: true option in the scrape_config. This reduces RAM usage and CPU usage.
Note that disabling staleness markers may result in unexpected query results when scrape targets are frequently rotated (this is a typical case in Kubernetes).
Set -memory.allowedBytes command-line flag to the value close to the actual memory usage of vmagent. Another option is to set memory limit in Kubernetes / Docker
to the value 50% bigger than the actual memory usage of vmagent. This should reduce memory usage spikes for vmagent running in the environment with bigger available memory
when the remote storage cannot keep up with the data ingestion rate. Increasing -remoteWrite.queues command-line flag value may help in this case too.
In extreme cases it may be useful to set -promscrape.disableKeepAlive command-line flag in order to save RAM on HTTP keep-alive connections to thousands of scrape targets.
Increase scrape_interval option in the global section of the -promscrape.config and/or at every scrape_config
for reducing CPU usage. For example, increasing the scrape_interval from 10s to 30s across all the targets decreases CPU usage at vmagent by up to 3x.
Example command, which runs vmagent in an optimized mode:
Run make package-vmagent. It builds victoriametrics/vmagent:<PKG_TAG> docker image locally.
<PKG_TAG> is an auto-generated image tag, which depends on source code in the repository.
The <PKG_TAG> may be manually set via PKG_TAG=foobar make package-vmagent.
The base docker image is alpine but it is possible to use any other base image
by setting it via <ROOT_IMAGE> environment variable. For example, the following command builds the image on top of scratch image:
Install Go. The minimum supported version is Go 1.22.
Run make vmagent-linux-arm or make vmagent-linux-arm64 from the root folder of the repository
It builds vmagent-linux-arm or vmagent-linux-arm64 binary respectively and puts it into the bin folder.
Run make vmagent-linux-arm-prod or make vmagent-linux-arm64-prod from the root folder of the repository.
It builds vmagent-linux-arm-prod or vmagent-linux-arm64-prod binary respectively and puts it into the bin folder.
vmagent can be fine-tuned with various command-line flags. Run ./vmagent -help in order to see the full list of these flags with their descriptions and default values:
./vmagent-helpvmagentcollectsmetricsdataviapopulardataingestionprotocolsandroutesthemtoVictoriaMetrics.Seethedocsathttps://docs.victoriametrics.com/vmagent/.-blockcache.missesBeforeCachingintThenumberofcachemissesbeforeputtingtheblockintocache.Highervaluesmayreduceindexdb/dataBlockscachesizeatthecostofhigherCPUanddiskreadusage(default2)-cacheExpireDurationdurationItemsareremovedfromin-memorycachesaftertheyaren't accessed for this duration. Lower values may reduce memory usage at the cost of higher CPU usage. See also -prevCacheRemovalPercent (default 30m0s)-configAuthKeyvalueAuthorizationkeyforaccessing/configpage.ItmustbepassedviaauthKeyqueryarg.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-configAuthKey=file:///abs/path/to/fileor-configAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-configAuthKey=http://host/pathor-configAuthKey=https://host/path-csvTrimTimestampdurationTrimtimestampswhenimportingcsvdatatothisduration.Minimumpracticaldurationis1ms.Higherduration(i.e.1s)maybeusedforreducingdiskspaceusagefortimestampdata(default1ms)-datadog.maxInsertRequestSizesizeThemaximumsizeinbytesofasingleDataDogPOSTrequestto/datadog/api/v2/seriesSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default67108864)-datadog.sanitizeMetricNameSanitizemetricnamesfortheingestedDataDogdatatocomplywithDataDogbehaviourdescribedathttps://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics (default true)-denyQueryTracingWhethertodisabletheabilitytotracequeries.Seehttps://docs.victoriametrics.com/#query-tracing-dryRunWhethertocheckconfigfileswithoutrunningvmagent.Thefollowingfilesarechecked:-promscrape.config,-remoteWrite.relabelConfig,-remoteWrite.urlRelabelConfig,-remoteWrite.streamAggr.config.Unknownconfigentriesaren't allowed in -promscrape.config by default. This can be changed by passing -promscrape.config.strictParse=false command-line flag-enableMultitenantHandlersWhethertoprocessincomingdataviamultitenantinserthandlersaccordingtohttps://docs.victoriametrics.com/cluster-victoriametrics/#url-format . By default incoming data is processed via single-node insert handlers according to https://docs.victoriametrics.com/#how-to-import-time-series-data .See https://docs.victoriametrics.com/vmagent/#multitenancy for details-enableTCP6WhethertoenableIPv6forlisteninganddialing.Bydefault,onlyIPv4TCPandUDPareused-envflag.enableWhethertoenablereadingflagsfromenvironmentvariablesinadditiontothecommandline.Commandlineflagvalueshavepriorityovervaluesfromenvironmentvars.Flagsarereadonlyfromthecommandlineifthisflagisn't set. See https://docs.victoriametrics.com/#environment-variables for more details-envflag.prefixstringPrefixforenvironmentvariablesif-envflag.enableisset-eulaDeprecated,pleaseuse-licenseor-licenseFileflagsinstead.Byspecifyingthisflag,youconfirmthatyouhaveanenterpriselicenseandaccepttheESAhttps://victoriametrics.com/legal/esa/.ThisflagisavailableonlyinEnterprisebinaries.Seehttps://docs.victoriametrics.com/enterprise/-filestream.disableFadviseWhethertodisablefadvise()syscallwhenreadinglargedatafiles.Thefadvise()syscallpreventsfromevictionofrecentlyaccesseddatafromOSpagecacheduringbackgroundmergesandbackups.InsomerarecasesitisbettertodisablethesyscallifitusestoomuchCPU-flagsAuthKeyvalueAuthkeyfor/flagsendpoint.ItmustbepassedviaauthKeyqueryarg.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-flagsAuthKey=file:///abs/path/to/fileor-flagsAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-flagsAuthKey=http://host/pathor-flagsAuthKey=https://host/path-fs.disableMmapWhethertousepread()insteadofmmap()forreadingdatafiles.Bydefault,mmap()isusedfor64-bitarchesandpread()isusedfor32-bitarches,sincetheycannotreaddatafilesbiggerthan2^32bytesinmemory.mmap()isusuallyfasterforreadingsmalldatachunksthanpread()-gcp.pubsub.publish.byteThresholdintPublishabatchwhenitssizeinbytesreachesthisvalue.Seehttps://docs.victoriametrics.com/vmagent/#writing-metrics-to-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ (default 1000000)-gcp.pubsub.publish.countThresholdintPublishabatchwhenithasthismanymessages.Seehttps://docs.victoriametrics.com/vmagent/#writing-metrics-to-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ (default 100)-gcp.pubsub.publish.credentialsFilestringPathtofilewithGCPcredentialstouseforPubSubclient.Ifnotset,defaultcredentialswillbeused(seeWorkloadIdentityforK8Sorhttps://cloud.google.com/docs/authentication/application-default-credentials).Seehttps://docs.victoriametrics.com/vmagent/#writing-metrics-to-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/-gcp.pubsub.publish.delayThresholdvaluePublishanon-emptybatchafterthisdelayhaspassed.Seehttps://docs.victoriametrics.com/vmagent/#writing-metrics-to-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/Thefollowingoptionalsuffixesaresupported:s(second),h(hour),d(day),w(week),y(year).Ifsuffixisn't set, then the duration is counted in months (default 10ms)-gcp.pubsub.publish.maxOutstandingBytesintThemaximumsizeofbufferedmessagestobepublished.Iflessthanorequaltozero,thisisdisabled.Seehttps://docs.victoriametrics.com/vmagent/#writing-metrics-to-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ (default -1)-gcp.pubsub.publish.maxOutstandingMessagesintThemaximumnumberofbufferedmessagestobepublished.Iflessthanorequaltozero,thisisdisabled.Seehttps://docs.victoriametrics.com/vmagent/#writing-metrics-to-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ (default 100)-gcp.pubsub.publish.timeoutvalueThemaximumtimethattheclientwillattempttopublishabundleofmessages.Seehttps://docs.victoriametrics.com/vmagent/#writing-metrics-to-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/Thefollowingoptionalsuffixesaresupported:s(second),h(hour),d(day),w(week),y(year).Ifsuffixisn't set, then the duration is counted in months (default 60s)-gcp.pubsub.subscribe.credentialsFilestringPathtofilewithGCPcredentialstouseforPubSubclient.Ifnotset,defaultcredentialsareused(seeWorkloadIdentityforK8Sorhttps://cloud.google.com/docs/authentication/application-default-credentials).Seehttps://docs.victoriametrics.com/vmagent/#reading-metrics-from-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/-gcp.pubsub.subscribe.defaultMessageFormatstringDefaultmessageformatif-gcp.pubsub.subscribe.topicSubscription.messageFormatismissing.Seehttps://docs.victoriametrics.com/vmagent/#reading-metrics-from-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ (default "promremotewrite")-gcp.pubsub.subscribe.topicSubscriptionarrayGCPPubSubtopicsubscriptionintheformat:projects/<project-id>/subscriptions/<subscription-name>.Seehttps://docs.victoriametrics.com/vmagent/#reading-metrics-from-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-gcp.pubsub.subscribe.topicSubscription.concurrencyarrayThenumberofconcurrentlyprocessedmessagesfortopicsubscriptionspecifiedvia-gcp.pubsub.subscribe.topicSubscriptionflag.Seehttps://docs.victoriametrics.com/vmagent/#reading-metrics-from-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ (default 0)Supportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettodefaultvalue.-gcp.pubsub.subscribe.topicSubscription.isGzippedarrayEnablesgzipdecompressionformessagespayloadatthecorresponding-gcp.pubsub.subscribe.topicSubscription.Onlyprometheus,jsonline,graphiteandinfluxformatsacceptgzippedmessages.Seehttps://docs.victoriametrics.com/vmagent/#reading-metrics-from-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/Supportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-gcp.pubsub.subscribe.topicSubscription.messageFormatarrayMessageformatforthecorresponding-gcp.pubsub.subscribe.topicSubscription.Validformats:influx,prometheus,promremotewrite,graphite,jsonline.Seehttps://docs.victoriametrics.com/vmagent/#reading-metrics-from-pubsub . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-graphiteListenAddrstringTCPandUDPaddresstolistenforGraphiteplaintextdata.Usually:2003mustbeset.Doesn't work if empty. See also -graphiteListenAddr.useProxyProtocol-graphiteListenAddr.useProxyProtocolWhethertouseproxyprotocolforconnectionsacceptedat-graphiteListenAddr.Seehttps://www.haproxy.org/download/1.8/doc/proxy-protocol.txt-graphiteTrimTimestampdurationTrimtimestampsforGraphitedatatothisduration.Minimumpracticaldurationis1s.Higherduration(i.e.1m)maybeusedforreducingdiskspaceusagefortimestampdata(default1s)-http.connTimeoutdurationIncomingconnectionsto-httpListenAddrareclosedaftertheconfiguredtimeout.ThismayhelpevenlyspreadingloadamongaclusterofservicesbehindTCP-levelloadbalancer.Zerovaluedisablesclosingofincomingconnections(default2m0s)-http.disableResponseCompressionDisablecompressionofHTTPresponsestosaveCPUresources.Bydefault,compressionisenabledtosavenetworkbandwidth-http.header.cspstringValuefor'Content-Security-Policy'header,recommended:"default-src 'self'"-http.header.frameOptionsstringValuefor'X-Frame-Options'header-http.header.hstsstringValuefor'Strict-Transport-Security'header,recommended:'max-age=31536000; includeSubDomains'-http.idleConnTimeoutdurationTimeoutforincomingidlehttpconnections(default1m0s)-http.maxGracefulShutdownDurationdurationThemaximumdurationforagracefulshutdownoftheHTTPserver.Ahighlyloadedservermayrequireincreasedvalueforagracefulshutdown(default7s)-http.pathPrefixstringAnoptionalprefixtoaddtoallthepathshandledbyhttpserver.Forexample,if'-http.pathPrefix=/foo/bar'isset,thenallthehttprequestswillbehandledon'/foo/bar/*'paths.Thismaybeusefulforproxiedrequests.Seehttps://www.robustperception.io/using-external-urls-and-proxies-with-prometheus-http.shutdownDelaydurationOptionaldelaybeforehttpservershutdown.Duringthisdelay,theserverreturnsnon-OKresponsesfrom/healthpage,soloadbalancerscanroutenewrequeststootherservers-httpAuth.passwordvaluePasswordforHTTPserver's Basic Auth. The authentication is disabled if -httpAuth.username is emptyFlagvaluecanbereadfromthegivenfilewhenusing-httpAuth.password=file:///abs/path/to/fileor-httpAuth.password=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-httpAuth.password=http://host/pathor-httpAuth.password=https://host/path-httpAuth.usernamestringUsernameforHTTPserver's Basic Auth. The authentication is disabled if empty. See also -httpAuth.password-httpListenAddrarrayTCPaddresstolistenforincominghttprequests.Setthisflagtoemptyvalueinordertodisablelisteningonanyport.Thismodemaybeusefulforrunningmultiplevmagentinstancesonthesameserver.Notethat/targetsand/metricspagesaren't available if -httpListenAddr=''. See also -tls and -httpListenAddr.useProxyProtocolSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-httpListenAddr.useProxyProtocolarrayWhethertouseproxyprotocolforconnectionsacceptedatthecorresponding-httpListenAddr.Seehttps://www.haproxy.org/download/1.8/doc/proxy-protocol.txt.Withenabledproxyprotocolhttpservercannotserveregular/metricsendpoint.Use-pushmetrics.urlformetricspushingSupportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-import.maxLineLensizeThemaximumlengthinbytesofasinglelineacceptedby/api/v1/import;thelinelengthcanbelimitedwith'max_rows_per_line'queryargpassedto/api/v1/exportSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default10485760)-influx.databaseNamesarrayComma-separatedlistofdatabasenamestoreturnfrom/queryand/influx/queryAPI.ThiscanbeneededforacceptingdatafromTelegrafpluginssuchashttps://github.com/fangli/fluent-plugin-influxdbSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-influx.forceStreamModeboolForcestreammodeparsingforingesteddata.Seehttps://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf.-influx.maxLineSizesizeThemaximumsizeinbytesforasingleInfluxDBlineduringparsing.Applicableforstreammodeonly.Seehttps://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegrafSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default262144)-influx.maxRequestSizesizeThemaximumsizeinbytesofasingleInfluxDBrequest.Applicableforbatchmodeonly.Seehttps://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegrafSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default67108864)-influxDBLabelstringDefaultlabelfortheDBnamesentover'?db={db_name}'queryparameter(default"db")-influxListenAddrstringTCPandUDPaddresstolistenforInfluxDBlineprotocoldata.Usually:8089mustbeset.Doesn't work if empty. This flag isn'tneededwheningestingdataoverHTTP-justsendittohttp://<vmagent>:8429/write.Seealso-influxListenAddr.useProxyProtocol-influxListenAddr.useProxyProtocolWhethertouseproxyprotocolforconnectionsacceptedat-influxListenAddr.Seehttps://www.haproxy.org/download/1.8/doc/proxy-protocol.txt-influxMeasurementFieldSeparatorstringSeparatorfor'{measurement}{separator}{field_name}'metricnamewheninsertedviaInfluxDBlineprotocol(default"_")-influxSkipMeasurementUses'{field_name}'asametricnamewhileignoring'{measurement}'and'-influxMeasurementFieldSeparator'-influxSkipSingleFieldUses'{measurement}'insteadof'{measurement}{separator}{field_name}'formetricnameifInfluxDBlinecontainsonlyasinglefield-influxTrimTimestampdurationTrimtimestampsforInfluxDBlineprotocoldatatothisduration.Minimumpracticaldurationis1ms.Higherduration(i.e.1s)maybeusedforreducingdiskspaceusagefortimestampdata(default1ms)-insert.maxQueueDurationdurationThemaximumdurationtowaitinthequeuewhen-maxConcurrentInsertsconcurrentinsertrequestsareexecuted(default1m0s)-internStringCacheExpireDurationdurationTheexpirydurationforcachesforinternedstrings.Seehttps://en.wikipedia.org/wiki/String_interning.Seealso-internStringMaxLenand-internStringDisableCache(default6m0s)-internStringDisableCacheWhethertodisablecachesforinternedstrings.ThismayreducememoryusageatthecostofhigherCPUusage.Seehttps://en.wikipedia.org/wiki/String_interning.Seealso-internStringCacheExpireDurationand-internStringMaxLen-internStringMaxLenintThemaximumlengthforstringstointern.AlowerlimitmaysavememoryatthecostofhigherCPUusage.Seehttps://en.wikipedia.org/wiki/String_interning.Seealso-internStringDisableCacheand-internStringCacheExpireDuration(default500)-kafka.consumer.topicarrayKafkatopicnamesfordataconsumption.Seehttps://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-kafka.consumer.topic.basicAuth.passwordarrayOptionalbasicauthpasswordfor-kafka.consumer.topic.Mustbeusedinconjunctionwithanysupportedauthmethodsforkafkaclient,specifiedbyflag-kafka.consumer.topic.options='security.protocol=SASL_SSL;sasl.mechanisms=PLAIN'.Seehttps://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-kafka.consumer.topic.basicAuth.usernamearrayOptionalbasicauthusernamefor-kafka.consumer.topic.Mustbeusedinconjunctionwithanysupportedauthmethodsforkafkaclient,specifiedbyflag-kafka.consumer.topic.options='security.protocol=SASL_SSL;sasl.mechanisms=PLAIN'.Seehttps://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-kafka.consumer.topic.brokersarrayListofbrokerstoconnectforgiventopic,e.g.-kafka.consumer.topic.broker=host-1:9092;host-2:9092.Seehttps://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-kafka.consumer.topic.concurrencyarrayConfiguresconsumerconcurrencyfortopicspecifiedvia-kafka.consumer.topicflag.Seehttps://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ (default 1)Supportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettodefaultvalue.-kafka.consumer.topic.defaultFormatstringExpecteddataformatinthetopicif-kafka.consumer.topic.formatisskipped.Seehttps://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ (default "promremotewrite")-kafka.consumer.topic.formatarraydataformatforcorrespondingkafkatopic.Validformats:influx,prometheus,promremotewrite,graphite,jsonline.Seehttps://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-kafka.consumer.topic.groupIDarrayDefinesgroup.idfortopic.Seehttps://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-kafka.consumer.topic.isGzippedarrayEnablesgzipsettingfortopicmessagespayload.Onlyprometheus,jsonline,graphiteandinfluxformatsacceptgzippedmessages.Seehttps://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/Supportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-kafka.consumer.topic.optionsarrayOptionalkey=value;key1=value2settingsfortopicconsumer.Seefullconfigurationoptionsathttps://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md.Seehttps://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-licensestringLicensekeyforVictoriaMetricsEnterprise.Seehttps://victoriametrics.com/products/enterprise/.TrialEnterpriselicensecanbeobtainedfromhttps://victoriametrics.com/products/enterprise/trial/.ThisflagisavailableonlyinEnterprisebinaries.Thelicensekeycanbealsopassedviafilespecifiedby-licenseFilecommand-lineflag-license.forceOfflineWhethertoenableofflineverificationforVictoriaMetricsEnterpriselicensekey,whichhasbeenpassedeithervia-licenseorvia-licenseFilecommand-lineflag.Theissuedlicensekeymustsupportofflineverificationfeature.Contactinfo@victoriametrics.comifyouneedofflinelicenseverification.ThisflagisavailableonlyinEnterprisebinaries-licenseFilestringPathtofilewithlicensekeyforVictoriaMetricsEnterprise.Seehttps://victoriametrics.com/products/enterprise/.TrialEnterpriselicensecanbeobtainedfromhttps://victoriametrics.com/products/enterprise/trial/.ThisflagisavailableonlyinEnterprisebinaries.Thelicensekeycanbealsopassedinlinevia-licensecommand-lineflag-loggerDisableTimestampsWhethertodisablewritingtimestampsinlogs-loggerErrorsPerSecondLimitintPer-secondlimitonthenumberofERRORmessages.Ifmorethanthegivennumberoferrorsareemittedpersecond,theremainingerrorsaresuppressed.Zerovaluesdisabletheratelimit-loggerFormatstringFormatforlogs.Possiblevalues:default,json(default"default")-loggerJSONFieldsstringAllowsrenamingfieldsinJSONformattedlogs.Example:"ts:timestamp,msg:message"renames"ts"to"timestamp"and"msg"to"message".Supportedfields:ts,level,caller,msg-loggerLevelstringMinimumleveloferrorstolog.Possiblevalues:INFO,WARN,ERROR,FATAL,PANIC(default"INFO")-loggerMaxArgLenintThemaximumlengthofasingleloggedargument.Longerargumentsarereplacedwith'arg_start..arg_end',where'arg_start'and'arg_end'isprefixandsuffixoftheargwiththelengthnotexceeding-loggerMaxArgLen/2(default1000)-loggerOutputstringOutputforthelogs.Supportedvalues:stderr,stdout(default"stderr")-loggerTimezonestringTimezonetousefortimestampsinlogs.TimezonemustbeavalidIANATimeZone.Forexample:America/New_York,Europe/Berlin,Etc/GMT+3orLocal(default"UTC")-loggerWarnsPerSecondLimitintPer-secondlimitonthenumberofWARNmessages.Ifmorethanthegivennumberofwarnsareemittedpersecond,thentheremainingwarnsaresuppressed.Zerovaluesdisabletheratelimit-maxConcurrentInsertsintThemaximumnumberofconcurrentinsertrequests.Sethighervaluewhenclientssenddataoverslownetworks.DefaultvaluedependsonthenumberofavailableCPUcores.Itshouldworkfineinmostcasessinceitminimizesresourceusage.Seealso-insert.maxQueueDuration-maxIngestionRateintThemaximumnumberofsamplesvmagentcanreceivepersecond.Dataingestionispausedwhenthelimitisexceeded.Bydefaulttherearenolimitsonsamplesingestionrate.Seealso-remoteWrite.rateLimit-maxInsertRequestSizesizeThemaximumsizeinbytesofasinglePrometheusremote_writeAPIrequestSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default33554432)-maxLabelsPerTimeseriesintThemaximumnumberoflabelspertimeseriestobeaccepted.Serieswithsuperfluouslabelsareignored.Inthiscasethevm_rows_ignored_total{reason="too_many_labels"}metricat/metricspageisincremented(default0)-maxLabelNameLenintThemaximumlengthoflabelnamesintheacceptedtimeseries.Serieswithlongerlabelnameareignored.Inthiscasethevm_rows_ignored_total{reason="too_long_label_name"}metricat/metricspageisincremented(default0)-maxLabelValueLenintThemaximumlengthoflabelvaluesintheacceptedtimeseries.Serieswithlongerlabelvalueareignored.Inthiscasethevm_rows_ignored_total{reason="too_long_label_value"}metricat/metricspageisincremented(default0)-memory.allowedBytessizeAllowedsizeofsystemmemoryVictoriaMetricscachesmayoccupy.Thisoptionoverrides-memory.allowedPercentifsettoanon-zerovalue.ToolowavaluemayincreasethecachemissrateusuallyresultinginhigherCPUanddiskIOusage.ToohighavaluemayevicttoomuchdatafromtheOSpagecacheresultinginhigherdiskIOusageSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-memory.allowedPercentfloatAllowedpercentofsystemmemoryVictoriaMetricscachesmayoccupy.Seealso-memory.allowedBytes.ToolowavaluemayincreasecachemissrateusuallyresultinginhigherCPUanddiskIOusage.ToohighavaluemayevicttoomuchdatafromtheOSpagecachewhichwillresultinhigherdiskIOusage(default60)-metrics.exposeMetadataWhethertoexposeTYPEandHELPmetadataatthe/metricspage,whichisexposedat-httpListenAddr.Themetadatamaybeneededwhenthe/metricspageisconsumedbysystems,whichrequirethisinformation.Forexample,ManagedPrometheusinGoogleCloud-https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting#missing-metric-type-metricsAuthKeyvalueAuthkeyfor/metricsendpoint.ItmustbepassedviaauthKeyqueryarg.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-metricsAuthKey=file:///abs/path/to/fileor-metricsAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-metricsAuthKey=http://host/pathor-metricsAuthKey=https://host/path-mtlsarrayWhethertorequirevalidclientcertificateforhttpsrequeststothecorresponding-httpListenAddr.Thisflagworksonlyif-tlsflagisset.Seealso-mtlsCAFile.ThisflagisavailableonlyinEnterprisebinaries.Seehttps://docs.victoriametrics.com/enterprise/Supportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-mtlsCAFilearrayOptionalpathtoTLSRootCAforverifyingclientcertificatesatthecorresponding-httpListenAddrwhen-mtlsisenabled.BydefaultthehostsystemTLSRootCAisusedforclientcertificateverification.ThisflagisavailableonlyinEnterprisebinaries.Seehttps://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-newrelic.maxInsertRequestSizesizeThemaximumsizeinbytesofasingleNewRelicrequestto/newrelic/infra/v2/metrics/events/bulkSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default67108864)-opentelemetry.usePrometheusNamingWhethertoconvertmetricnamesandlabelsintoPrometheus-compatibleformatforthemetricsingestedviaOpenTelemetryprotocol;seehttps://docs.victoriametrics.com/#sending-data-via-opentelemetry-opentsdbHTTPListenAddrstringTCPaddresstolistenforOpenTSDBHTTPputrequests.Usually:4242mustbeset.Doesn't work if empty. See also -opentsdbHTTPListenAddr.useProxyProtocol-opentsdbHTTPListenAddr.useProxyProtocolWhethertouseproxyprotocolforconnectionsacceptedat-opentsdbHTTPListenAddr.Seehttps://www.haproxy.org/download/1.8/doc/proxy-protocol.txt-opentsdbListenAddrstringTCPandUDPaddresstolistenforOpenTSDBmetrics.TelnetputmessagesandHTTP/api/putmessagesaresimultaneouslyservedonTCPport.Usually:4242mustbeset.Doesn't work if empty. See also -opentsdbListenAddr.useProxyProtocol-opentsdbListenAddr.useProxyProtocolWhethertouseproxyprotocolforconnectionsacceptedat-opentsdbListenAddr.Seehttps://www.haproxy.org/download/1.8/doc/proxy-protocol.txt-opentsdbTrimTimestampdurationTrimtimestampsforOpenTSDB'telnet put'datatothisduration.Minimumpracticaldurationis1s.Higherduration(i.e.1m)maybeusedforreducingdiskspaceusagefortimestampdata(default1s)-opentsdbhttp.maxInsertRequestSizesizeThemaximumsizeofOpenTSDBHTTPputrequestSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default33554432)-opentsdbhttpTrimTimestampdurationTrimtimestampsforOpenTSDBHTTPdatatothisduration.Minimumpracticaldurationis1ms.Higherduration(i.e.1s)maybeusedforreducingdiskspaceusagefortimestampdata(default1ms)-pprofAuthKeyvalueAuthkeyfor/debug/pprof/*endpoints.ItmustbepassedviaauthKeyqueryarg.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-pprofAuthKey=file:///abs/path/to/fileor-pprofAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-pprofAuthKey=http://host/pathor-pprofAuthKey=https://host/path-prevCacheRemovalPercentfloatItemsinthepreviouscachesareremovedwhenthepercentofrequestsitservesbecomeslowerthanthisvalue.HighervaluesreducememoryusageatthecostofhigherCPUusage.Seealso-cacheExpireDuration(default0.1)-promscrape.azureSDCheckIntervaldurationIntervalforcheckingforchangesinAzure.Thisworksonlyifazure_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/sd_configs/#azure_sd_configs for details (default 1m0s)-promscrape.cluster.memberLabelstringIfnon-empty,thenthelabelwiththisnameandthe-promscrape.cluster.memberNumvalueisaddedtoallthescrapedmetrics.Seehttps://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets for more info-promscrape.cluster.memberNumstringThenumberofvmagentinstanceintheclusterofscrapers.Itmustbeauniquevalueintherange0...promscrape.cluster.membersCount-1acrossscrapersinthecluster.CanbespecifiedaspodnameofKubernetesStatefulSet-pod-name-Num,whereNumisanumericpartofpodname.Seealso-promscrape.cluster.memberLabel.Seehttps://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets for more info (default "0")-promscrape.cluster.memberURLTemplatestringAnoptionaltemplateforURLtoaccessvmagentinstancewiththegiven-promscrape.cluster.memberNumvalue.Every%doccurrenceinthetemplateissubstitutedwith-promscrape.cluster.memberNumaturlstovmagentinstancesresponsibleforscrapingthegiventargetat/service-discoverypage.Forexample-promscrape.cluster.memberURLTemplate='http://vmagent-%d:8429/targets'.Seehttps://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets for more details-promscrape.cluster.membersCountintThenumberofmembersinaclusterofscrapers.Eachmembermusthaveaunique-promscrape.cluster.memberNumintherange0...promscrape.cluster.membersCount-1.Eachmemberthenscrapesroughly1/Nofallthetargets.Bydefault,clusterscrapingisdisabled,i.e.asinglescraperscrapesallthetargets.Seehttps://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets for more info (default 1)-promscrape.cluster.namestringOptionalnameofthecluster.Ifmultiplevmagentclustersscrapethesametargets,theneachclustermusthaveuniquenameinordertoproperlyde-duplicatesamplesreceivedfromtheseclusters.Seehttps://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets for more info-promscrape.cluster.replicationFactorintThenumberofmembersinthecluster,whichscrapethesametargets.Ifthereplicationfactorisgreaterthan1,thenthededuplicationmustbeenabledatremotestorageside.Seehttps://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets for more info (default 1)-promscrape.configstringOptionalpathtoPrometheusconfigfilewith'scrape_configs'sectioncontainingtargetstoscrape.Thepathcanpointtolocalfileandtohttpurl.Seehttps://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter for details-promscrape.config.dryRunChecks-promscrape.configfileforerrorsandunsupportedfieldsandthenexits.Returnsnon-zeroexitcodeonparsingerrorsandemitstheseerrorstostderr.Seealso-promscrape.config.strictParsecommand-lineflag.Pass-loggerLevel=ERRORifyoudon't need to see info messages in the output.-promscrape.config.strictParseWhethertodenyunsupportedfieldsin-promscrape.config.Settofalseinordertosilentlyskipunsupportedfields(defaulttrue)-promscrape.configCheckIntervaldurationIntervalforcheckingforchangesin-promscrape.configfile.Bydefault,thecheckingisdisabled.Seehowtoreload-promscrape.configfileathttps://docs.victoriametrics.com/vmagent/#configuration-update-promscrape.consul.waitTimedurationWaittimeusedbyConsulservicediscovery.Defaultvalueisusedifnotset-promscrape.consulSDCheckIntervaldurationIntervalforcheckingforchangesinConsul.Thisworksonlyifconsul_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/sd_configs/#consul_sd_configs for details (default 30s)-promscrape.consulagentSDCheckIntervaldurationIntervalforcheckingforchangesinConsulAgent.Thisworksonlyifconsulagent_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/sd_configs/#consulagent_sd_configs for details (default 30s)-promscrape.digitaloceanSDCheckIntervaldurationIntervalforcheckingforchangesindigitalocean.Thisworksonlyifdigitalocean_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/sd_configs/#digitalocean_sd_configs for details (default 1m0s)-promscrape.disableCompressionWhethertodisablesending'Accept-Encoding: gzip'requestheaderstoallthescrapetargets.ThismayreduceCPUusageonscrapetargetsatthecostofhighernetworkbandwidthutilization.Itispossibletoset'disable_compression: true'individuallypereach'scrape_config'sectionin'-promscrape.config'forfine-grainedcontrol-promscrape.disableKeepAliveWhethertodisableHTTPkeep-aliveconnectionswhenscrapingallthetargets.ThismaybeusefulwhentargetshasnosupportforHTTPkeep-aliveconnection.Itispossibletoset'disable_keepalive: true'individuallypereach'scrape_config'sectionin'-promscrape.config'forfine-grainedcontrol.NotethatdisablingHTTPkeep-alivemayincreaseloadonbothvmagentandscrapetargets-promscrape.discovery.concurrencyintThemaximumnumberofconcurrentrequeststoPrometheusautodiscoveryAPI(Consul,Kubernetes,etc.)(default100)-promscrape.discovery.concurrentWaitTimedurationThemaximumdurationforwaitingtoperformAPIrequestsifmorethan-promscrape.discovery.concurrencyrequestsaresimultaneouslyperformed(default1m0s)-promscrape.dnsSDCheckIntervaldurationIntervalforcheckingforchangesindns.Thisworksonlyifdns_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/sd_configs/#dns_sd_configs for details (default 30s)-promscrape.dockerSDCheckIntervaldurationIntervalforcheckingforchangesindocker.Thisworksonlyifdocker_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/sd_configs/#docker_sd_configs for details (default 30s)-promscrape.dockerswarmSDCheckIntervaldurationIntervalforcheckingforchangesindockerswarm.Thisworksonlyifdockerswarm_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/sd_configs/#dockerswarm_sd_configs for details (default 30s)-promscrape.dropOriginalLabelsWhethertodroporiginallabelsforscrapetargetsat/targetsand/api/v1/targetspages.Thismaybeneededforreducingmemoryusagewhenoriginallabelsforbignumberofscrapetargetsoccupybigamountsofmemory.Notethatthisreducesdebuggabilityforimproperper-targetrelabelingconfigs-promscrape.ec2SDCheckIntervaldurationIntervalforcheckingforchangesinec2.Thisworksonlyifec2_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/sd_configs/#ec2_sd_configs for details (default 1m0s)-promscrape.eurekaSDCheckIntervaldurationIntervalforcheckingforchangesineureka.Thisworksonlyifeureka_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/sd_configs/#eureka_sd_configs for details (default 30s)-promscrape.fileSDCheckIntervaldurationIntervalforcheckingforchangesin'file_sd_config'.Seehttps://docs.victoriametrics.com/sd_configs/#file_sd_configs for details (default 1m0s)-promscrape.gceSDCheckIntervaldurationIntervalforcheckingforchangesingce.Thisworksonlyifgce_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/sd_configs/#gce_sd_configs for details (default 1m0s)-promscrape.hetznerSDCheckIntervaldurationIntervalforcheckingforchangesinHetznerAPI.Thisworksonlyifhetzner_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/sd_configs/#hetzner_sd_configs for details (default 1m0s)-promscrape.httpSDCheckIntervaldurationIntervalforcheckingforchangesinhttpendpointservicediscovery.Thisworksonlyifhttp_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/sd_configs/#http_sd_configs for details (default 1m0s)-promscrape.kubernetes.apiServerTimeoutdurationHowfrequentlytoreloadthefullstatefromKubernetesAPIserver(default30m0s)-promscrape.kubernetes.attachNodeMetadataAllWhethertosetattach_metadata.node=trueforallthekubernetes_sd_configsat-promscrape.config.Itispossibletosetattach_metadata.node=falseindividuallypereachkubernetes_sd_configs.Seehttps://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs-promscrape.kubernetes.useHTTP2ClientWhethertouseHTTP/2clientforconnectiontoKubernetesAPIserver.ThismayreduceamountofconcurrentconnectionstoAPIserverwhenwatchingforabignumberofKubernetesobjects.-promscrape.kubernetesSDCheckIntervaldurationIntervalforcheckingforchangesinKubernetesAPIserver.Thisworksonlyifkubernetes_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs for details (default 30s)-promscrape.kumaSDCheckIntervaldurationIntervalforcheckingforchangesinkumaservicediscovery.Thisworksonlyifkuma_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/sd_configs/#kuma_sd_configs for details (default 30s)-promscrape.marathonSDCheckIntervaldurationIntervalforcheckingforchangesinMarathonservicediscovery.Thisworksonlyifmarathon_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/sd_configs.html#marathon_sd_configs for details (default 30s)-promscrape.maxDroppedTargetsintThemaximumnumberofdroppedTargetstoshowat/api/v1/targetspage.Increasethisvalueifyoursetupdropsmorescrapetargetsduringrelabelingandyouneedinvestigatinglabelsforallthedroppedtargets.Notethattheincreasednumberoftrackeddroppedtargetsmayresultinincreasedmemoryusage(default10000)-promscrape.maxResponseHeadersSizesizeThemaximumsizeofhttpresponseheadersfromPrometheusscrapetargetsSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default4096)-promscrape.maxScrapeSizesizeThemaximumsizeofscraperesponseinbytestoprocessfromPrometheustargets.BiggerresponsesarerejectedSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default16777216)-promscrape.minResponseSizeForStreamParsesizeTheminimumtargetresponsesizeforautomaticswitchingtostreamparsingmode,whichcanreducememoryusage.Seehttps://docs.victoriametrics.com/vmagent/#stream-parsing-modeSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default1000000)-promscrape.noStaleMarkersWhethertodisablesendingPrometheusstalemarkersformetricswhenscrapetargetdisappears.Thisoptionmayreducememoryusageifstalemarkersaren't needed for your setup. This option also disables populating the scrape_series_added metric. See https://prometheus.io/docs/concepts/jobs_instances/#automatically-generated-labels-and-time-series-promscrape.nomad.waitTimedurationWaittimeusedbyNomadservicediscovery.Defaultvalueisusedifnotset-promscrape.nomadSDCheckIntervaldurationIntervalforcheckingforchangesinNomad.Thisworksonlyifnomad_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/sd_configs/#nomad_sd_configs for details (default 30s)-promscrape.openstackSDCheckIntervaldurationIntervalforcheckingforchangesinopenstackAPIserver.Thisworksonlyifopenstack_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/sd_configs/#openstack_sd_configs for details (default 30s)-promscrape.ovhcloudSDCheckIntervaldurationIntervalforcheckingforchangesinOVHCloudVPSanddedicatedserver.Thisworksonlyifovhcloud_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/sd_configs/#ovhcloud_sd_configs for details (default 30s)-promscrape.puppetdbSDCheckIntervaldurationIntervalforcheckingforchangesinPuppetDBAPI.Thisworksonlyifpuppetdb_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/sd_configs/#puppetdb_sd_configs for details (default 30s)-promscrape.seriesLimitPerTargetintOptionallimitonthenumberofuniquetimeseriesasinglescrapetargetcanexpose.Seehttps://docs.victoriametrics.com/vmagent/#cardinality-limiter for more info-promscrape.streamParseWhethertoenablestreamparsingformetricsobtainedfromscrapetargets.Thismaybeusefulforreducingmemoryusagewhenmillionsofmetricsareexposedpereachscrapetarget.Itispossibletoset'stream_parse: true'individuallypereach'scrape_config'sectionin'-promscrape.config'forfine-grainedcontrol-promscrape.suppressDuplicateScrapeTargetErrorsWhethertosuppress'duplicate scrape target'errors;seehttps://docs.victoriametrics.com/vmagent/#troubleshooting for details-promscrape.suppressScrapeErrorsWhethertosuppressscrapeerrorslogging.Thelasterrorforeachtargetisalwaysavailableat'/targets'pageevenifscrapeerrorsloggingissuppressed.Seealso-promscrape.suppressScrapeErrorsDelay-promscrape.suppressScrapeErrorsDelaydurationThedelayforsuppressingrepeatedscrapeerrorsloggingpereachscrapetargets.Thismaybeusedforreducingthenumberofloglinesrelatedtoscrapeerrors.Seealso-promscrape.suppressScrapeErrors-promscrape.vultrSDCheckIntervaldurationIntervalforcheckingforchangesinVultr.Thisworksonlyifvultr_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/sd_configs.html#vultr_sd_configs for details (default 30s)-promscrape.yandexcloudSDCheckIntervaldurationIntervalforcheckingforchangesinYandexCloudAPI.Thisworksonlyifyandexcloud_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/sd_configs/#yandexcloud_sd_configs for details (default 30s)-pushmetrics.disableCompressionWhethertodisablerequestbodycompressionwhenpushingmetricstoevery-pushmetrics.url-pushmetrics.extraLabelarrayOptionallabelstoaddtometricspushedtoevery-pushmetrics.url.Forexample,-pushmetrics.extraLabel='instance="foo"'addsinstance="foo"labeltoallthemetricspushedtoevery-pushmetrics.urlSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-pushmetrics.headerarrayOptionalHTTPrequestheadertosendtoevery-pushmetrics.url.Forexample,-pushmetrics.header='Authorization: Basic foobar'adds'Authorization: Basic foobar'headertoeveryrequesttoevery-pushmetrics.urlSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-pushmetrics.intervaldurationIntervalforpushingmetricstoevery-pushmetrics.url(default10s)-pushmetrics.urlarrayOptionalURLtopushmetricsexposedat/metricspage.Seehttps://docs.victoriametrics.com/#push-metrics . By default, metrics exposed at /metrics page aren't pushed to any remote storageSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-reloadAuthKeyvalueAuthkeyfor/-/reloadhttpendpoint.ItmustbepassedviaauthKeyqueryarg.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-reloadAuthKey=file:///abs/path/to/fileor-reloadAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-reloadAuthKey=http://host/pathor-reloadAuthKey=https://host/path-remoteWrite.aws.accessKeyarrayOptionalAWSAccessKeytouseforthecorresponding-remoteWrite.urlif-remoteWrite.aws.useSigv4issetSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.aws.ec2EndpointarrayOptionalAWSEC2APIendpointtouseforthecorresponding-remoteWrite.urlif-remoteWrite.aws.useSigv4issetSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.aws.regionarrayOptionalAWSregiontouseforthecorresponding-remoteWrite.urlif-remoteWrite.aws.useSigv4issetSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.aws.roleARNarrayOptionalAWSroleARNtouseforthecorresponding-remoteWrite.urlif-remoteWrite.aws.useSigv4issetSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.aws.secretKeyarrayOptionalAWSSecretKeytouseforthecorresponding-remoteWrite.urlif-remoteWrite.aws.useSigv4issetSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.aws.servicearrayOptionalAWSServicetouseforthecorresponding-remoteWrite.urlif-remoteWrite.aws.useSigv4isset.Defaultsto"aps"Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.aws.stsEndpointarrayOptionalAWSSTSAPIendpointtouseforthecorresponding-remoteWrite.urlif-remoteWrite.aws.useSigv4issetSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.aws.useSigv4arrayEnablesSigV4requestsigningforthecorresponding-remoteWrite.url.Itisexpectedthatother-remoteWrite.aws.*command-lineflagsaresetifsigv4requestsigningisenabledSupportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-remoteWrite.basicAuth.passwordarrayOptionalbasicauthpasswordtouseforthecorresponding-remoteWrite.urlSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.basicAuth.passwordFilearrayOptionalpathtobasicauthpasswordtouseforthecorresponding-remoteWrite.url.Thefileisre-readeverysecondSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.basicAuth.usernamearrayOptionalbasicauthusernametouseforthecorresponding-remoteWrite.urlSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.bearerTokenarrayOptionalbearerauthtokentouseforthecorresponding-remoteWrite.urlSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.bearerTokenFilearrayOptionalpathtobearertokenfiletouseforthecorresponding-remoteWrite.url.Thetokenisre-readfromthefileeverysecondSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.disableOnDiskQueuearrayWhethertodisablestoringpendingdatato-remoteWrite.tmpDataPathwhentheremotestoragesystematthecorresponding-remoteWrite.urlcannotkeepupwiththedataingestionrate.Seehttps://docs.victoriametrics.com/vmagent#disabling-on-disk-persistence . See also -remoteWrite.dropSamplesOnOverloadSupportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-remoteWrite.dropSamplesOnOverloadWhethertodropsampleswhen-remoteWrite.disableOnDiskQueueissetandifthesamplescannotbepushedintotheconfigured-remoteWrite.urlsystemsinatimelymanner.Seehttps://docs.victoriametrics.com/vmagent#disabling-on-disk-persistence-remoteWrite.flushIntervaldurationIntervalforflushingthedatatoremotestorage.Thisoptiontakeseffectonlywhenlessthan10Kdatapointspersecondarepushedto-remoteWrite.url(default1s)-remoteWrite.forcePromProtoarrayWhethertoforcePrometheusremotewriteprotocolforsendingdatatothecorresponding-remoteWrite.url.Seehttps://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocolSupportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-remoteWrite.forceVMProtoarrayWhethertoforceVictoriaMetricsremotewriteprotocolforsendingdatatothecorresponding-remoteWrite.url.Seehttps://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocolSupportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-remoteWrite.headersarrayOptionalHTTPheaderstosendwitheachrequesttothecorresponding-remoteWrite.url.Forexample,-remoteWrite.headers='My-Auth:foobar'wouldsend'My-Auth: foobar'HTTPheaderwitheveryrequesttothecorresponding-remoteWrite.url.Multipleheadersmustbedelimitedby'^^':-remoteWrite.headers='header1:value1^^header2:value2'Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.keepDanglingQueuesKeeppersistentqueuescontentsat-remoteWrite.tmpDataPathincasetherearenomatching-remoteWrite.url.Usefulwhen-remoteWrite.urlischangedtemporarilyandpersistentqueuefileswillbeneededlateron.-remoteWrite.labelarrayOptionallabelintheform'name=value'toaddtoallthemetricsbeforesendingthemto-remoteWrite.url.Passmultiple-remoteWrite.labelflagsinordertoaddmultiplelabelstometricsbeforesendingthemtoremotestorageSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.maxBlockSizesizeThemaximumblocksizetosendtoremotestorage.Biggerblocksmayimproveperformanceatthecostoftheincreasedmemoryusage.Seealso-remoteWrite.maxRowsPerBlockSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default8388608)-remoteWrite.maxDailySeriesintThemaximumnumberofuniqueseriesvmagentcansendtoremotestoragesystemsduringthelast24hours.Excessseriesareloggedanddropped.Thiscanbeusefulforlimitingserieschurnrate.Seehttps://docs.victoriametrics.com/vmagent/#cardinality-limiter-remoteWrite.maxDiskUsagePerURLarrayThemaximumfile-basedbuffersizeinbytesat-remoteWrite.tmpDataPathforeach-remoteWrite.url.Whenbuffersizereachestheconfiguredmaximum,thenolddataisdroppedwhenaddingnewdatatothebuffer.Buffereddataisstoredin~500MBchunks.Itisrecommendedtosetthevalueforthisflagtoamultipleoftheblocksize500MB.Diskusageisunlimitedifthevalueissetto0Supportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB.(default0)Supportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettodefaultvalue.-remoteWrite.maxHourlySeriesintThemaximumnumberofuniqueseriesvmagentcansendtoremotestoragesystemsduringthelasthour.Excessseriesareloggedanddropped.Thiscanbeusefulforlimitingseriescardinality.Seehttps://docs.victoriametrics.com/vmagent/#cardinality-limiter-remoteWrite.maxRowsPerBlockintThemaximumnumberofsamplestosendineachblocktoremotestorage.Highernumbermayimproveperformanceatthecostoftheincreasedmemoryusage.Seealso-remoteWrite.maxBlockSize(default10000)-remoteWrite.oauth2.clientIDarrayOptionalOAuth2clientIDtouseforthecorresponding-remoteWrite.urlSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.oauth2.clientSecretarrayOptionalOAuth2clientSecrettouseforthecorresponding-remoteWrite.urlSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.oauth2.clientSecretFilearrayOptionalOAuth2clientSecretFiletouseforthecorresponding-remoteWrite.urlSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.oauth2.endpointParamsarrayOptionalOAuth2endpointparameterstouseforthecorresponding-remoteWrite.url.TheendpointparametersmustbesetinJSONformat:{"param1":"value1",...,"paramN":"valueN"}Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.oauth2.scopesarrayOptionalOAuth2scopestouseforthecorresponding-remoteWrite.url.Scopesmustbedelimitedby';'Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.oauth2.tokenUrlarrayOptionalOAuth2tokenURLtouseforthecorresponding-remoteWrite.urlSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.proxyURLarrayOptionalproxyURLforwritingdatatothecorresponding-remoteWrite.url.Supportedproxies:http,https,socks5.Example:-remoteWrite.proxyURL=socks5://proxy:1234Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.queuesintThenumberofconcurrentqueuestoeach-remoteWrite.url.Setmorequeuesifdefaultnumberofqueuesisn't enough for sending high volume of collected data to remote storage. Default value depends on the number of available CPU cores. It should work fine in most cases since it minimizes resource usage-remoteWrite.rateLimitarrayOptionalratelimitinbytespersecondfordatasenttothecorresponding-remoteWrite.url.Bydefault,theratelimitisdisabled.Itcanbeusefulforlimitingloadonremotestoragewhenbigamountsofbuffereddataissentaftertemporaryunavailabilityoftheremotestorage.Seealso-maxIngestionRate(default0)Supportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettodefaultvalue.-remoteWrite.relabelConfigstringOptionalpathtofilewithrelabelingconfigs,whichareappliedtoallthemetricsbeforesendingthemto-remoteWrite.url.Seealso-remoteWrite.urlRelabelConfig.Thepathcanpointeithertolocalfileortohttpurl.Seehttps://docs.victoriametrics.com/vmagent/#relabeling-remoteWrite.retryMaxTimearrayThemaxtimespentonretryattemptstosendablockofdatatothecorresponding-remoteWrite.url.Changethisvalueifitisexpectedfor-remoteWrite.urltobeunreachableformorethan-remoteWrite.retryMaxTime.Seealso-remoteWrite.retryMinInterval(default1m0s)Supportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettodefaultvalue.-remoteWrite.retryMinIntervalarrayTheminimumdelaybetweenretryattemptstosendablockofdatatothecorresponding-remoteWrite.url.Everynextretryattemptwilldoublethedelaytopreventhammeringofremotedatabase.Seealso-remoteWrite.retryMaxTime(default1s)Supportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettodefaultvalue.-remoteWrite.roundDigitsarrayRoundmetricvaluestothisnumberofdecimaldigitsafterthepointbeforewritingthemtoremotestorage.Examples:-remoteWrite.roundDigits=2wouldround1.236to1.24,while-remoteWrite.roundDigits=-1wouldround126.78to130.Bydefault,digitsroundingisdisabled.Setitto100fordisablingitforaparticularremotestorage.Thisoptionmaybeusedforimprovingdatacompressionforthestoredmetrics(default100)Supportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettodefaultvalue.-remoteWrite.sendTimeoutarrayTimeoutforsendingasingleblockofdatatothecorresponding-remoteWrite.url(default1m0s)Supportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettodefaultvalue.-remoteWrite.shardByURLWhethertoshardoutgoingseriesacrossalltheremotestoragesystemsenumeratedvia-remoteWrite.url.Bydefaultthedataisreplicatedacrossallthe-remoteWrite.url.Seehttps://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages . See also -remoteWrite.shardByURLReplicas-remoteWrite.shardByURL.ignoreLabelsarrayOptionallistoflabels,whichmustbeignoredwhenshardingoutgoingsamplesamongremotestoragesystemsif-remoteWrite.shardByURLcommand-lineflagisset.Bydefaultallthelabelsareusedforshardinginordertogainevendistributionofseriesoverthespecified-remoteWrite.urlsystems.Seealso-remoteWrite.shardByURL.labelsSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.shardByURL.labelsarrayOptionallistoflabels,whichmustbeusedforshardingoutgoingsamplesamongremotestoragesystemsif-remoteWrite.shardByURLcommand-lineflagisset.Bydefaultallthelabelsareusedforshardinginordertogainevendistributionofseriesoverthespecified-remoteWrite.urlsystems.Seealso-remoteWrite.shardByURL.ignoreLabelsSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.shardByURLReplicasintHowmanycopiesofdatatomakeamongremotestoragesystemsenumeratedvia-remoteWrite.urlwhen-remoteWrite.shardByURLisset.Seehttps://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages (default 1)-remoteWrite.showURLWhethertoshow-remoteWrite.urlintheexportedmetrics.Itishiddenbydefault,sinceitcancontainsensitiveinfosuchasauthkey-remoteWrite.significantFiguresarrayThenumberofsignificantfigurestoleaveinmetricvaluesbeforewritingthemtoremotestorage.Seehttps://en.wikipedia.org/wiki/Significant_figures.Zerovaluesavesallthesignificantfigures.Thisoptionmaybeusedforimprovingdatacompressionforthestoredmetrics.Seealso-remoteWrite.roundDigits(default0)Supportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettodefaultvalue.-remoteWrite.streamAggr.configarrayOptionalpathtofilewithstreamaggregationconfigforthecorresponding-remoteWrite.url.Seehttps://docs.victoriametrics.com/stream-aggregation/.Seealso-remoteWrite.streamAggr.keepInput,-remoteWrite.streamAggr.dropInputand-remoteWrite.streamAggr.dedupIntervalSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.streamAggr.dedupIntervalarrayInputsamplesarede-duplicatedwiththisintervalbeforeoptionalaggregationwith-remoteWrite.streamAggr.configatthecorresponding-remoteWrite.url.Seealso-dedup.minScrapeIntervalandhttps://docs.victoriametrics.com/stream-aggregation/#deduplication (default 0s)Supportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettodefaultvalue.-remoteWrite.streamAggr.dropInputarrayWhethertodropalltheinputsamplesaftertheaggregationwith-remoteWrite.streamAggr.configatthecorresponding-remoteWrite.url.Bydefault,onlyaggregatessamplesaredropped,whiletheremainingsamplesarewrittentothecorresponding-remoteWrite.url.Seealso-remoteWrite.streamAggr.keepInputandhttps://docs.victoriametrics.com/stream-aggregation/Supportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-remoteWrite.streamAggr.dropInputLabelsarrayAnoptionallistoflabelstodropfromsamplesbeforestreamde-duplicationandaggregationwith-remoteWrite.streamAggr.configand-remoteWrite.streamAggr.dedupIntervalatthecorresponding-remoteWrite.url.MultiplelabelsperremoteWrite.urlmustbedelimitedby'^^':-remoteWrite.streamAggr.dropInputLabels='replica^^az,replica'.Seehttps://docs.victoriametrics.com/stream-aggregation/#dropping-unneeded-labelsSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.streamAggr.enableWindowsarrayEnablesaggregationwithinfixedwindowsforallremotewrite's aggregators. This allows to get more precise results, but impacts resource usage as it requires twice more memory to store two states. See https://docs.victoriametrics.com/stream-aggregation/#aggregation-windows.Supportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse-remoteWrite.streamAggr.ignoreFirstIntervalsarrayNumberofaggregationintervalstoskipafterthestartforthecorresponding-remoteWrite.streamAggr.configatthecorresponding-remoteWrite.url.Increasethisvalueifyouobserveincorrectaggregationresultsaftervmagentrestarts.Itcouldbecausedbyreceivingbuffereddelayeddatafromclientspushingdataintothevmagent.Seehttps://docs.victoriametrics.com/stream-aggregation/#ignore-aggregation-intervals-on-start-remoteWrite.streamAggr.ignoreOldSamplesarrayWhethertoignoreinputsampleswitholdtimestampsoutsidethecurrentaggregationintervalforthecorresponding-remoteWrite.streamAggr.configatthecorresponding-remoteWrite.url.Seehttps://docs.victoriametrics.com/stream-aggregation/#ignoring-old-samplesSupportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-remoteWrite.streamAggr.keepInputarrayWhethertokeepalltheinputsamplesaftertheaggregationwith-remoteWrite.streamAggr.configatthecorresponding-remoteWrite.url.Bydefault,onlyaggregatessamplesaredropped,whiletheremainingsamplesarewrittentothecorresponding-remoteWrite.url.Seealso-remoteWrite.streamAggr.dropInputandhttps://docs.victoriametrics.com/stream-aggregation/Supportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-remoteWrite.tlsCAFilearrayOptionalpathtoTLSCAfiletouseforverifyingconnectionstothecorresponding-remoteWrite.url.Bydefault,systemCAisusedSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.tlsCertFilearrayOptionalpathtoclient-sideTLScertificatefiletousewhenconnectingtothecorresponding-remoteWrite.urlSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.tlsHandshakeTimeoutarrayThetimeoutforestablishingtlsconnectionstothecorresponding-remoteWrite.url(default20s)Supportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettodefaultvalue.-remoteWrite.tlsInsecureSkipVerifyarrayWhethertoskiptlsverificationwhenconnectingtothecorresponding-remoteWrite.urlSupportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-remoteWrite.tlsKeyFilearrayOptionalpathtoclient-sideTLScertificatekeytousewhenconnectingtothecorresponding-remoteWrite.urlSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.tlsServerNamearrayOptionalTLSservernametouseforconnectionstothecorresponding-remoteWrite.url.Bydefault,theservernamefrom-remoteWrite.urlisusedSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.tmpDataPathstringPathtodirectoryforstoringpendingdata,whichisn't sent to the configured -remoteWrite.url . See also -remoteWrite.maxDiskUsagePerURL and -remoteWrite.disableOnDiskQueue (default "vmagent-remotewrite-data")-remoteWrite.urlarrayRemotestorageURLtowritedatato.ItmustsupporteitherVictoriaMetricsremotewriteprotocolorPrometheusremote_writeprotocol.Exampleurl:http://<victoriametrics-host>:8428/api/v1/write.Passmultiple-remoteWrite.urloptionsinordertoreplicatethecollecteddatatomultipleremotestoragesystems.Thedatacanbeshardedamongtheconfiguredremotestoragesystemsif-remoteWrite.shardByURLflagissetSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.urlRelabelConfigarrayOptionalpathtorelabelconfigsforthecorresponding-remoteWrite.url.Seealso-remoteWrite.relabelConfig.Thepathcanpointeithertolocalfileortohttpurl.Seehttps://docs.victoriametrics.com/vmagent/#relabelingSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-remoteWrite.vmProtoCompressLevelintThecompressionlevelforVictoriaMetricsremotewriteprotocol.HighervaluesreducenetworktrafficatthecostofhigherCPUusage.NegativevaluesreduceCPUusageatthecostofincreasednetworktraffic.Seehttps://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocol-sortLabelsWhethertosortlabelsforincomingsamplesbeforewritingthemtoalltheconfiguredremotestoragesystems.Thismaybeneededforreducingmemoryusageatremotestoragewhentheorderoflabelsinincomingsamplesisrandom.Forexample,ifm{k1="v1",k2="v2"}maybesentasm{k2="v2",k1="v1"}Enabledsortingforlabelscanslowdowningestionperformanceabit-streamAggr.configstringOptionalpathtofilewithstreamaggregationconfig.Seehttps://docs.victoriametrics.com/stream-aggregation/.Seealso-streamAggr.keepInput,-streamAggr.dropInputand-streamAggr.dedupInterval-streamAggr.dedupIntervalvalueInputsamplesarede-duplicatedwiththisintervalonaggregatorbeforeoptionalaggregationwith-streamAggr.config.Seealso-dedup.minScrapeIntervalandhttps://docs.victoriametrics.com/stream-aggregation/#deduplicationThefollowingoptionalsuffixesaresupported:s(second),h(hour),d(day),w(week),y(year).Ifsuffixisn't set, then the duration is counted in months (default 0s)-streamAggr.dropInputWhethertodropalltheinputsamplesaftertheaggregationwith-remoteWrite.streamAggr.config.Bydefault,onlyaggregatessamplesaredropped,whiletheremainingsamplesarewrittentoremotestorageswrite.Seealso-streamAggr.keepInputandhttps://docs.victoriametrics.com/stream-aggregation/-streamAggr.dropInputLabelsarrayAnoptionallistoflabelstodropfromsamplesforaggregatorbeforestreamde-duplicationandaggregation.Seehttps://docs.victoriametrics.com/stream-aggregation/#dropping-unneeded-labelsSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-streamAggr.enableWindowsEnablesaggregationwithinfixedwindowsforallglobalaggregators.Thisallowstogetmorepreciseresults,butimpactsresourceusageasitrequirestwicemorememorytostoretwostates.Seehttps://docs.victoriametrics.com/stream-aggregation/#aggregation-windows.-streamAggr.ignoreFirstIntervalsintNumberofaggregationintervalstoskipafterthestartforaggregator.Increasethisvalueifyouobserveincorrectaggregationresultsaftervmagentrestarts.Itcouldbecausedbyreceivingunordereddelayeddatafromclientspushingdataintothevmagent.Seehttps://docs.victoriametrics.com/stream-aggregation/#ignore-aggregation-intervals-on-start-streamAggr.ignoreOldSamplesWhethertoignoreinputsampleswitholdtimestampsoutsidethecurrentaggregationintervalforaggregator.Seehttps://docs.victoriametrics.com/stream-aggregation/#ignoring-old-samples-streamAggr.keepInputWhethertokeepalltheinputsamplesaftertheaggregationwith-streamAggr.config.Bydefault,onlyaggregatessamplesaredropped,whiletheremainingsamplesarewrittentoremotestorageswrite.Seealso-streamAggr.dropInputandhttps://docs.victoriametrics.com/stream-aggregation/-tlsarrayWhethertoenableTLSforincomingHTTPrequestsatthegiven-httpListenAddr(akahttps).-tlsCertFileand-tlsKeyFilemustbesetif-tlsisset.Seealso-mtlsSupportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-tlsAutocertCacheDirstringDirectorytostoreTLScertificatesissuedviaLet's Encrypt. Certificates are lost on restarts if this flag isn'tset.ThisflagisavailableonlyinEnterprisebinaries.Seehttps://docs.victoriametrics.com/enterprise/-tlsAutocertEmailstringContactemailfortheissuedLet's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/-tlsAutocertHostsarrayOptionalhostnamesforautomaticissuingofLet's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-tlsCertFilearrayPathtofilewithTLScertificateforthecorresponding-httpListenAddrif-tlsisset.PreferECDSAcertsinsteadofRSAcertsasRSAcertsareslower.Theprovidedcertificatefileisautomaticallyre-readeverysecond,soitcanbedynamicallyupdated.Seealso-tlsAutocertHostsSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-tlsCipherSuitesarrayOptionallistofTLSciphersuitesforincomingrequestsoverHTTPSif-tlsisset.Seethelistofsupportedciphersuitesathttps://pkg.go.dev/crypto/tls#pkg-constantsSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-tlsKeyFilearrayPathtofilewithTLSkeyforthecorresponding-httpListenAddrif-tlsisset.Theprovidedkeyfileisautomaticallyre-readeverysecond,soitcanbedynamicallyupdated.Seealso-tlsAutocertHostsSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-tlsMinVersionarrayOptionalminimumTLSversiontouseforthecorresponding-httpListenAddrif-tlsisset.Supportedvalues:TLS10,TLS11,TLS12,TLS13Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Valuecancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-usePromCompatibleNamingWhethertoreplacecharactersunsupportedbyPrometheuswithunderscoresintheingestedmetricnamesandlabelnames.Forexample,foo.bar{a.b='c'}istransformedintofoo_bar{a_b='c'}duringdataingestionifthisflagisset.Seehttps://prometheus.io/docs/concepts/data_model/#metric-names-and-labels-versionShowVictoriaMetricsversion
