SECURITY: upgrade Go builder from Go1.20.5 to Go1.20.6. See the list of issues addressed in Go1.20.6.
indexdb
rotation (aka inverted index), which is performed once per -retentionPeriod
. The new algorithm gradually pre-populates newly created indexdb
during the last hour before the rotation. The number of pre-populated series in the newly created indexdb
can be monitored via vm_timeseries_precreated_total
metric. This should resolve this issue.or
filters. For example, {env="prod",job="a" or env="dev",job="b"}
selects series with either {env="prod",job="a"}
or {env="dev",job="b"}
labels. This functionality allows passing the selected series to rollup functions without the need to use subqueries. See these docs.keep_metric_names
modifier. For example, ({__name__=~"foo|bar"} / 10) keep_metric_names
leaves foo
and bar
metric names in division results. See these docs. This helps to address issues like this one.one
side of many-to-one operations by specifying *
inside group_left()
or group_right()
. Also allow adding a prefix for copied label names via group_left(*) prefix "..."
syntax. For example, the following query copies Kubernetes namespace labels to kube_pod_info
series and adds ns_
prefix for the copied label names: kube_pod_info * on(namespace) group_left(*) prefix "ns_" kube_namespace_labels
. The labels from on()
list aren't prefixed. This feature resolves this and that questions at StackOverflow.WITH
templates. Examples: WITH (w = 5m) m[w]
is automatically transformed to m[5m]
WITH (f(window, step, off) = m[window:step] offset off) f(5m, 10s, 1h)
is automatically transformed to m[5m:10s] offset 1h
Thanks to @lujiajing1126 for the initial idea and implementation. See this feature request.vmagent
instance at the second level. See these docs and this feature request for details.if
option of relabeling rules. The corresponding relabeling rule is executed when at least a single series selector matches. See these docs.match
option of stream aggregation configs. The input sample is aggregated when at least a single series selector matches. See this feature request.match
options from the configured aggregations. Previously all the input samples were dropped by default, so only the aggregated samples are written to the output storage. The previous behavior can be restored by passing -streamAggr.dropInput
command-line flag to single-node VictoriaMetrics or by passing -remoteWrite.streamAggr.dropInput
command-line flag to vmagent
.source
or destination
databases. See this issue.process_io_(read|write)_syscalls_total
metrics.start
, end
and time
query args in Prometheus querying API. See these docs and this feature request.-remoteWrite.url
. By default, vmalert will make multiple retry attempts with exponential delay. The total time spent during retry attempts shouldn't exceed -remoteWrite.retryMaxTime
(default is 30s). When retry time is exceeded vmalert drops the data dedicated for -remoteWrite.url
. Before, vmalert dropped data after 5 retry attempts with 1s delay between attempts (not configurable). See -remoteWrite.retryMinInterval
and -remoteWrite.retryMaxTime
cmd-line flags.vmalert_remotewrite_send_duration_seconds_total
counter, which can be used for determining high saturation of every connection to remote storage with an alerting query sum(rate(vmalert_remotewrite_send_duration_seconds_total[5m])) by(job, instance) > 0.9 * max(vmalert_remotewrite_concurrency) by(job, instance)
. This query triggers when a connection is saturated by more than 90%. This usually means that -remoteWrite.concurrency
command-line flag must be increased in order to increase the number of concurrent writings into remote endpoint. See this feature request.step
param attached to instant queries. This might be useful for using vmalert with datasources that to not support this param, unlike VictoriaMetrics. See this issue for details.-notifier.blackhole
cmd-line flag is set. Enable this flag if you want vmalert to evaluate alerting rules without sending any notifications to external receivers (eg. alertmanager). See this issue for details. Thanks to @venkatbvc for the pull request.graphite
datasource type, in the same way as it happens for prometheus
type. See this issue.keep_firing_for
field for alerting rules. See docs updated here and this issue. Thanks to @Haleygo for the pull request.vmauth_user_request_duration_seconds
and vmauth_unauthorized_user_request_duration_seconds
summary metrics for measuring requests latency per user.FEATURE: add ability to fine-tune Graphite API limits via the following command-line flags: -search.maxGraphiteTagKeys
for limiting the number of tag keys returned from Graphite API for tags -search.maxGraphiteTagValues
for limiting the number of tag values returned from Graphite API for tag values -search.maxGraphiteSeries
for limiting the number of series (aka paths) returned from Graphite API for series See this issue.
limit
series (limit
is an optional query arg passed to this API). Previously the limit exceeded error
error was returned in this case. See this issue.--remote-read-filter-time-start
flag is not set for remote-read mode. This flag is now required to use remote-read mode. See this issue.s
characters at the end of samples/s
output during data migration. For example, it could write samples/ssssss
. See this issue./targets
page when vmagent
runs with -promscrape.dropOriginalLabels
command-line flag, since it has no the original labels needed for relabel debug. See this issue.foo + (on{bar="baz"})
. Previously such queries could lead to panic. See this issue.Cache usage % by type
for cluster dashboard. Before, only vmstorage caches were shown.Released at 2023-06-30
SECURITY: upgrade Go builder from Go1.20.4 to Go1.20.5. See the list of issues addressed in Go1.20.5.
vmbackupmanager
not being able to restore data from a backup stored in GCS. See this issue for details.vmselect
when the -search.maxSamplesPerQuery
or -search.maxSamplesPerSeries
limit is exceeded. Previously incomplete response could be returned without the error if vmselect
runs with -replicationFactor
greater than 1. See this pull request.v1.90.0
to newer versions. See this issue for details.-replicationFactor
flag is set bigger than > 1. Before, vmselect could have skip waiting for the slowest replicas to respond. This could have resulted in issues illustrated here. Now, this optimization is disabled by default and could be re-enabled by passing -search.skipSlowReplicas
cmd-line flag to vmselect. See more details here.Released at 2023-06-02
Released at 2023-06-01
follow_redirects
at service discovery level of scrape configuration. See this issue. Thanks to @Haleygo for the pull request.FEATURE: vmselect: Decreases startup time for vmselect with a big number of vmstorage nodes. See this issue. Thanks to @Haleygo for the pull request.
http.pathPrefix
set. See this issue./api/v1/rules
for groups with present name but absent rules
. See this issue.Released at 2023-05-18
SECURITY: serve /robots.txt
content to disallow indexing of the exposed instances by search engines. See this issue for details.
-bigMergeConcurrency
command-line flag, since improper configuration for this flag frequently led to uncontrolled growth of unmerged parts, which, in turn, could lead to queries slowdown and increased CPU usage. The concurrency for background merges can be controlled via -smallMergeConcurrency
command-line flag, though it isn't recommended to change this flag in general case.2024-01-02+02:00
means January 2, 2024
at +02:00
time zone. See these docs.process_*
metrics at /metrics
page of all the VictoriaMetrics components under Windows OS. See this pull request.INFO
logging during VictoriaMetrics startup / shutdown. This should improve visibility for potentially important logs.cannot read hello: cannot read message with size 11: EOF
messages at vmstorage
during TCP health checks performed by Consul or other services. See this issue.filter
option. See this feature request.-remoteWrite.maxDiskUsagePerURL
command-line flag. See this issue.-rule
and -rule.templates
command-line flags by using **
in the glob pattern. See this issue./groups
. See this issue.POST
requests on the remaining backends if the currently selected backend isn't reachable. See this issue.WITH template
playground. See this feature request.-s3StorageClass
command-line flag for setting the storage class for AWS S3 backups. See this issue. Thanks to @justcompile for the pull request.backup_complete.ignore
file of backup contents. This allows determining the exact timestamp when the backup was created and completed.created_at
field to the output of /api/v1/backups
API and vmbackupmanager backup list
command. See this doc for data format details.--vm-native-filter-time-start
and --vm-native-filter-time-end
command-line flags. See this issue.FEATURE: vmctl: set default value for --vm-native-step-interval
command-line flag to month
. This enables time-based chunking of data based on monthly step value when using native migration mode. See this issue.
sum(rate(vm_slow_per_day_index_inserts_total))
. See this pull request.indexdb
rotation when -retentionTimezoneOffset
command-line flag is set and the local timezone is not UTC. See this issue. Thanks to @faceair for the fix.-memory.allowedPercent
from 100 to 200. See this issue.-maxConcurrentInserts
command-line flag. See this issue. Thanks to @zouxiang1993 for the fix."
chars when parsing Prometheus label names, since they aren't allowed by Prometheus text exposition format. Previously this could result in silent incorrect parsing of incorrect Prometheus labels such as foo{"bar"="baz"}
or {foo:"bar",baz="aaa"}
. See this issue.M
suffix. Such a suffix isn't allowed to use in durations, since it clashes with a million
suffix, e.g. it isn't clear whether rate(metric[5M])
means rate over 5 minutes, 5 months or 5 million seconds. See this and this issues.vm_promscrape_config_last_reload_successful
metric after config reload. See this issue.__meta_kubernetes_endpoints_name
label for all ports discovered from endpoint. Previously, ports not matched by Service
did not have this label. See this issue for details. Thanks to @thunderbird86 for discovering and fixing the issue.query
function for templating value of -external.alert.source
flag. See this issue./api/v1/rules
and /api/v1/alerts
API handlers. See this issue.TooHighMemoryUsage
alert from 90% to 80%, since 90% is too high for production environments.--vm-native-disable-retries
command-line flag. See this issue.Released at 2023-04-06
Update note: this release contains backwards-incompatible change in storage data format, so the previous versions of VictoriaMetrics will exit with the unexpected number of substrings in the part name
error when trying to run them on the data created by v1.90.0 or newer versions. The solution is to upgrade to v1.90.0 or newer releases
SECURITY: upgrade Go builder from Go1.20.2 to Go1.20.3. See the list of issues addressed in Go1.20.3.
-maxLabelValueLen
. This should simplify debugging for this case.vmagent
to Kafka
located in another datacenter or availability zone. See this feature request.-kafka.consumer.topic.concurrency
command-line flag. It controls the number of Kafka consumer workers to use by vmagent
. It should eliminate the need to start multiple vmagent
instances to improve data transfer rate. See this feature request.arm64
machines. See this issue.-remoteWrite.tmpDataPath
directory when there is no matching -remoteWrite.url
to send this data to. See this feature request.-relabelConfig
and -streamAggr.config
files additionally to -promscrape.config
when single-node VictoriaMetrics runs with -dryRun
command-line flag. This aligns the behaviour of single-node VictoriaMetrics with vmagent behaviour for -dryRun
command-line flag.show more
button. See this pull request.apply
and cancel
buttons to settings popup. See this issue.FEATURE: vmauth: add -configCheckInterval
command-line flag, which can be used for automatic re-reading the -auth.config
file. See this feature request.
EOF
. Usually, the error is caused by health checks and is not a sign of an actual error.series after dedup
error message in logs when -remoteWrite.streamAggr.dedupInterval
command-line flag is set at vmagent or when -streamAggr.dedupInterval
command-line flag is set at single-node VictoriaMetrics.%{ENV-VAR.SYNTAX}
. See these docs and this issue.-storage.minFreeDiskSpaceBytes
is set. See the issue.""
instead of "<no value>"
during templating, as Prometheus does. See this issue.--vm-native-filter-match
command-line flag to the data source. Previously filters from --vm-native-filter-match
were only used to discover the metric names, and the metric names like __name__="metric_name"
has been taken into account, while the remaining filters were ignored. For example --vm-native-src-addr={foo="bar",baz="abc"}
may found metric_name{foo="bar",baz="abc"}
and filter was treated as --vm-native-src-addr={__name__="metrics_name"}
, e.g. foo="bar",baz="abc"
filter was ignored. See this issue.Released at 2023-03-12
cannot unmarshal timeseries from rollupResultCache
panic after the upgrade to v1.89.0.Released at 2023-03-12
Update note: this release can crash with cannot unmarshal timeseries from rollupResultCache
panic after the upgrade from the previous releases. This issue can be fixed by removing caches stored on disk according to these docs. Another option is to upgrade to v1.89.1.
SECURITY: upgrade Go builder from Go1.20.1 to Go1.20.2. See the list of issues addressed in Go1.20.2.
--remote-read-http-timeout
command-line option from 30s (30 seconds) to 5m (5 minutes). This reduces the probability of timeout errors when migrating big number of time series. See this pull request.--vm-native-src-headers
and --vm-native-dst-headers
command-line flags, which can be used for setting custom HTTP headers during vm-native migration mode. Thanks to @baconmania for the pull request.--vm-native-src-bearer-token
and --vm-native-dst-bearer-token
command-line flags, which can be used for setting Bearer token headers for the source and the destination storage during vm-native migration mode. See this feature request.--vm-native-disable-http-keep-alive
command-line flag to allow vmctl
to use non-persistent HTTP connections in vm-native migration mode. Thanks to @baconmania for the pull request.-rule
command-line flag.FEATURE: vmalert enterprise: concurrently read config files from S3, GCS or S3-compatible object storage. This significantly improves config load speed for cases when there are thousands of files to read from the object storage.
invalid memory address or nil pointer dereference
panic during background merge. The issue has been introduced at v1.85.0. See this issue.SIGBUS
crash on ARM architectures (Raspberry Pi), which deny unaligned access to 8-byte words. Thanks to @oliverpool for narrowing down the issue and for the initial attempt to fix it.is_partial: true
in partial responses. Previously partial responses could be returned as non-partial in some cases.-rpc.disableCompression
command-line flag at vmstorage
. It was ignored since v1.78.0. See this pull request.Please enter a valid Query and execute it
error message on the first load of vmui.Run in VMUI
button click in VictoriaMetrics datasource plugin for Grafana.Explore
page.step
param for instant queries. See this issue.cannot serve http
panic when plain HTTP request is sent to vmauth
configured to accept requests over proxy protocol-encoded request (e.g. when vmauth
runs with -httpListenAddr.useProxyProtocol
command-line flag). The issue has been introduced at v1.87.0 when implementing this feature.Released at 2023-02-27
-snapshotCreateTimeout
flag to allow configuring timeout for snapshot process. See this issue.vm_http_requests_total
and vm_http_request_errors_total
metrics for snapshot/*
paths at VictoriaMetrics cluster vmstorage
and VictoriaMetrics Single. See this issue.-internStringDisableCache
command-line flag for disabling the cache for interned strings. This flag may be useful in some cases for reducing memory usage at the cost of higher CPU usage.FEATURE: add -internStringCacheExpireDuration
command-line flag for controlling the lifetime of cached interned strings.
aggr_func(rollup*(some_value))
. The panic has been introduced in v1.88.0.-remoteWrite.*
auth options when determining whether the remote storage supports VictoriaMetrics remote write protocol. Previously the auth options were ignored. This was preventing from automatic switch to VictoriaMetrics remote write protocol.vm_promscrape_config_*
metrics if -promscrape.config
flag is not used. Previously those metrics were registered and never updated, which was confusing and could trigger false-positive alerts.Released at 2023-02-24
SECURITY: upgrade Go builder from Go1.20.0 to Go1.20.1. See the list of issues addressed in Go1.20.1.
vmagent
to VictoriaMetrics located in another datacenter or availability zone. This also allows reducing disk IO under high load when vmagent
starts queuing the collected data to disk when the remote storage is temporarily unavailable or cannot keep up with the data ingestion rate. See this feature request.-maxConcurrentPerUserRequests
command-line flag and via max_concurrent_requests
config option. See this feature request and these docs.GET
requests on all the configured backends. Previously the backend error has been immediately returned to the client without retrying the request on the remaining backends.vmctl
was stopping after the first error. See this feature request.min
, max
or avg
for rollup, rollup_delta, rollup_deriv, rollup_increase, rollup_rate and rollup_scrape_interval function. If the second argument is passed, then the function returns only the selected aggregation type. This change can be useful for situations where only one type of rollup calculation is needed. For example, rollup_rate(requests_total[1i], "max")
would return only the max increase rates for requests_total
metric per each interval between adjacent points on the graph. See this article for details.open
, low
, high
, close
for rollup_candlestick function. If the second argument is passed, then the function returns only the selected aggregation type.mad_over_time(m[d])
function for calculating the median absolute deviation over raw samples on the lookbehind window d
. See this feature request.range_mad(q)
function for calculating the median absolute deviation over points per each time series returned by q
.range_zscore(q)
function for calculating z-score over points per each time series returned from q
.range_trim_outliers(k, q)
function for dropping outliers located farther than k*range_mad(q)
from the range_median(q)
. This should help removing outliers during query time at this issue.range_trim_zscore(z, q)
function for dropping outliers located farther than z*range_stddev(q)
from range_avg(q)
. This should help removing outliers during query time at this issue.median
instead of avg
in graph tooltip and line legend, since median
is more tolerant against spikes. See this issue.-search.maxSeriesPerAggrFunc
command-line flag, which can be used for limiting the number of time series MetricsQL aggregate functions can return in a single query. This flag can be useful for preventing OOMs when count_values function is improperly used.-search.logQueryMemoryUsage
command-line flag for logging queries, which need more memory than specified by this command-line flag. See this feature request. Thanks to @michal-kralik for the idea and the intial implementation.FEATURE: allow setting zero value for -search.latencyOffset
command-line flag. This may be needed in some cases. Previously the minimum supported value for -search.latencyOffset
command-line flag was 1s
.
vmagent
could wait for long time until all the in-flight requests are completed before reloading the configuration. This could significantly slow down configuration reload. See this issue.cannot create hard links from ...must-remove...
errors when making snapshots / backups. See this issue.Released at 2023-11-14
v1.87.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release
SECURITY: upgrade Go builder from Go1.21.3 to Go1.21.4. the list of issues addressed in Go1.21.4.
regex
, which start and end with .+
or .*
and which contain alternate sub-regexps. For example, .+;|;.+
or .*foo|bar|baz.*
. Previously such regexps were improperly parsed, which could result in undexpected relabeling results. See this issue.Released at 2023-10-16
v1.87.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release
SECURITY: upgrade Go builder from Go1.21.1 to Go1.21.3. See the list of issues addressed in Go1.21.2 and the list of issues addressed in Go1.21.3.
null
instead of Inf
in JSON query responses. See this issue.Released at 2023-09-10
v1.87.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release
SECURITY: upgrade Go builder from Go1.21.0 to Go1.21.1. See the list of issues addressed in Go1.20.6.
/insert/multitenant/*
urls, which have been broken since v1.93.2. See this issue.GOARCH=arm
. This has been broken after the upgrading of Go builder to Go1.21.0. See this issue.503 Service Unavailable
status code when partial responses are denied and some of vmstorage
nodes are temporarily unavailable. Previously 422 Unprocessable Entiry
status code was mistakenly returned in this case, which could prevent from automatic recovery by re-sending the request to healthy cluster replica in another availability zone.params
fields with multiple values were overriding each other instead of adding up. The bug was introduced in this commit starting from v1.87.7. See this issue.Released at 2023-09-01
v1.87.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release
/select/multitenant
prefix. Such requests must be rejected since vmselect does not support multitenancy endpoint. Previously, such requests were causing panic. See this issue.vmstorage
. Previously it wasn't properly checked, which could lead to increased resource usage and data ingestion slowdown when some of vmstorage
nodes are in read-only mode. See this issue.vmstorage
. Previously it wasn't properly closed, which prevents restoring vmstorage
node from read-only mode. See this issue.vmselect
-> vmstorage
RPC communication when vmstorage
returns an empty label name at /api/v1/labels
request. See this issue.-boolFlagName value
, since this leads to silent incomplete flags' parsing. This form should be replaced with -boolFlagName=value
. See this issue.:
chars in label names with _
when -usePromCompatibleNaming
command-line flag is passed to vmagent
, vminsert
or single-node VictoriaMetrics. This addresses this comment.-dst
belongs to specified -storageDataPath
. See this issue.Released at 2023-08-12
v1.87.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release
SECURITY: upgrade base docker image (Alpine) from 3.18.2 to 3.18.3. See alpine 3.18.3 release notes.
proxy_url
field at the scrape config. See this issue for details.if
filters during relabeling. Previously the if
filter could improperly work. See this issue and this pull request.honor_timestamps: true
option is explicitly set at scrape_config. This fixes gaps for metrics collected from cadvisor or similar exporters, which export metrics with invalid timestamps. See this issue and this comment for details.vmselect
when the -search.maxSamplesPerQuery
or -search.maxSamplesPerSeries
limit is exceeded. Previously incomplete response could be returned without the error if vmselect
runs with -replicationFactor
greater than 1. See this pull request.http.pathPrefix
set. See this issue./api/v1/rules
for groups with present name but absent rules
. See this issue.--remote-read-filter-time-start
flag is not set for remote-read mode. This flag is now required to use remote-read mode. See this issue.Released at 2023-05-18
v1.87.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release
SECURITY: serve /robots.txt
content to disallow indexing of the exposed instances by search engines. See this issue for details.
sum(rate(vm_slow_per_day_index_inserts_total))
. See this pull request."
chars when parsing Prometheus label names, since they aren't allowed by Prometheus text exposition format. Previously this could result in silent incorrect parsing of incorrect Prometheus labels such as foo{"bar"="baz"}
or {foo:"bar",baz="aaa"}
. See this issue.M
suffix. Such a suffix isn't allowed to use in durations, since it clashes with a million
suffix, e.g. it isn't clear whether rate(metric[5M])
means rate over 5 minutes, 5 months or 5 million seconds. See this and this issues.-maxConcurrentInserts
command-line flag. See this issue. Thanks to @zouxiang1993 for the fix./api/v1/rules
and /api/v1/alerts
API handlers. See this issue.__meta_kubernetes_endpoints_name
label for all ports discovered from endpoint. Previously, ports not matched by Service
did not have this label. See this issue for details. Thanks to @thunderbird86 for discovering and fixing the issue.indexdb
rotation when -retentionTimezoneOffset
command-line flag is set and the local timezone is not UTC. See this issue. Thanks to @faceair for the fix.TooHighMemoryUsage
alert from 90% to 80%, since 90% is too high for production environments.vm_promscrape_config_last_reload_successful
metric after config reload. See this issue.series after dedup
error message in logs when -remoteWrite.streamAggr.dedupInterval
command-line flag is set at vmagent or when -streamAggr.dedupInterval
command-line flag is set at single-node VictoriaMetrics.Released at 2023-04-06
v1.87.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release
SECURITY: upgrade Go builder from Go1.20.2 to Go1.20.3. See the list of issues addressed in Go1.20.3.
-storage.minFreeDiskSpaceBytes
is set. See the issue.Released at 2023-03-25
v1.87.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release
EOF
. Usually, the error is caused by health checks and is not a sign of an actual error.%{ENV-VAR.SYNTAX}
. See these docs and this issue.Released at 2023-03-12
v1.87.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release
SECURITY: upgrade Go builder from Go1.20.1 to Go1.20.2. See the list of issues addressed in Go1.20.2.
invalid memory address or nil pointer dereference
panic during background merge. The issue has been introduced at v1.85.0. See this issue.SIGBUS
crash on ARM architectures (Raspberry Pi), which deny unaligned access to 8-byte words. Thanks to @oliverpool for narrowing down the issue and for the initial attempt to fix it.is_partial: true
in partial responses. Previously partial responses could be returned as non-partial in some cases.-rpc.disableCompression
command-line flag at vmstorage
. It was ignored since v1.78.0. See this pull request.vm_promscrape_config_*
metrics if -promscrape.config
flag is not used. Previously those metrics were registered and never updated, which was confusing and could trigger false-positive alerts.cannot serve http
panic when plain HTTP request is sent to vmauth
configured to accept requests over proxy protocol-encoded request (e.g. when vmauth
runs with -httpListenAddr.useProxyProtocol
command-line flag). The issue has been introduced at v1.87.0 when implementing this feature.Released at 2023-02-24
v1.87.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release
SECURITY: upgrade Go builder from Go1.20.0 to Go1.20.1. See the list of issues addressed in Go1.20.1.
vmagent
could wait for long time until all the in-flight requests are completed before reloading the configuration. This could significantly slow down configuration reload. See this issue.cannot create hard links from ...must-remove...
errors when making snapshots / backups. See this issue.Released at 2023-02-09
v1.87.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release
-remoteRead.ignoreRestoreErrors
command-line flag becomes deprecated now and will have no effect if configured. While previously state restore attempt was made for all the loaded alerting rules, now it is called only for alerts which became active after the first evaluation. See this issue.FEATURE: vmui: hide the default per-job resource usage
dashboard if there is a custom dashboard exists at the directory specified via -vmui.customDashboardsPath
command-line flag. See this feature request.
debug
and update_entries_limit
rule's params on config's hot-reload.vm_concurrent_insert_current
metric before exposing it. Previously this metric could be left uninitialized in some cases, e.g. its value was zero. This could lead to false alerts for the query avg_over_time(vm_concurrent_insert_current[1m]) >= vm_concurrent_insert_capacity
. See this issue.vmagent
could wait for long time until all the in-flight requests are completed before reloading the configuration. This could significantly slow down configuration reload. See this issue.error setting up jwt verification
error.Released at 2023-02-01
v1.87.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release
-streamAggr.dedupInterval
and -remoteWrite.streamAggr.dedupInterval
command-line options.settings
menu in the top right corner. See this pull request.vmui
in full from isolated networks without access to Internet. Thanks to @ScottKevill for the pull request.__meta_kubernetes_pod_container_id
meta-label to the targets discovered via kubernetes_sd_configs. This label has been added in Prometheus starting from v2.42.0
. See this feature request.__meta_azure_machine_size
meta-label to the targets discovered via azure_sd_configs. This label has been added in Prometheus starting from v2.42.0
. See this pull request.vmauth
via -maxConcurrentRequests
command-line flag. This allows controlling memory usage of vmauth
and the resource usage of backends behind vmauth
. See this feature request. Thanks to @dmitryk-dk for the initial implementation.-httpListenAddr.useProxyProtocol
command-line flag.-internStringMaxLen
command-line flag, which can be used for fine-tuning RAM vs CPU usage in certain workloads. For example, if the stored time series contain long labels, then it may be useful reducing the -internStringMaxLen
in order to reduce memory usage at the cost of increased CPU usage. See this issue.FEATURE: provide GOARCH=386 binaries for single-node VictoriaMetrics, vmagent, vmalert, vmauth, vmbackup and vmrestore components at releases page. See this feature request. Thanks to @denisgolius for the pull request.
vmstorage
to vmselect
. Previously some timeout errors weren't returned from vmselect
to vmstorage
. Instead, vmstorage
could log the error and close the connection to vmselect
, so vmselect
was logging cryptic errors such as cannot execute funcName="..." on vmstorage "...": EOF
.__meta_ec2_availability_zone_id
attribute.200 OK
HTTP status code when importing data via Pushgateway protocol. See this issue.exported_
prefix to scraped metric names, which clash with the automatically generated metric names if honor_labels: true
option is set in the scrape_config. See the this and this issues.vmauth
was returning 400 Bad Request
instead of 401 Unauthorized
http response code.-replicationFactor
flag is set and request contains trace
query parameter. See this issue.Released at 2023-01-18
SECURITY: vmbackup: do not expose basic auth passwords from -snapshot.createURL
and -snapshot.deleteURL
command-line flags in logs. Thanks to @toanju for the pull request.
-vmui.customDashboardsPath
command-line flag. See this feature request and these docs.step
globally to all the displayed graphs. See this feature request.FEATURE: vmui: improve the appearance of graph lines by using more visually distinct colors. See this feature request.
vmselect
to v1.85.3 level when processing heavy queries. See this issue.FATAL: cannot locate metric name for metricID=...: EOF
panic, which could occur when retention filters are enabled.filters
only to objects of the specified role
. Previously filters were applied to all the objects, which could cause errors when different types of objects were used with filters that were not compatible with them. See this issue.-promscrape.suppressScrapeErrors
is enabled. Previously some scrape errors were logged even if -promscrape.suppressScrapeErrors
flag was set.200 OK
HTTP response code. See this issue.M
and Mi
suffixes as 1e6
multipliers in 1M
and 1Mi
numeric constants. See this issue. The issue has been introduced in v1.86.0.Table
view. For example, up[5m]
query now shows all the raw samples for the last 5 minutes for the up
metric at the Table
view. See this issue.Released at 2023-01-10
vmselect
to vmstorage
during query execution. This issue could result in the following error logs at vmstorage
under load: cannot process vmselect request: cannot execute "search_v7": couldn't start executing the request in 0.000 seconds, since -search.maxConcurrentRequests=... concurrent requests are already executed
. The issue has been introduced in v1.86.0.Released at 2023-01-10
It is recommended upgrading to VictoriaMetrics v1.86.1 because v1.86.0 contains a bug, which could lead to incorrect query results over time series with gaps.
Update note 1: This release changes the logic behind -maxConcurrentInserts
command-line flag. Previously this flag was limiting the number of concurrent connections established from clients, which send data to VictoriaMetrics. Some of these connections could be temporarily idle. Such connections do not take significant CPU and memory resources, so there is no need in limiting their count. The new logic takes into account only those connections, which actively ingest new data to VictoriaMetrics and to vmagent. This means that the default -maxConcurrentInserts
value should handle cases, which could require increasing the value in the previous releases. So it is recommended trying to remove the explicitly set -maxConcurrentInserts
command-line flag after upgrading to this release and verifying whether this reduces CPU and memory usage.
Update note 2: The vm_concurrent_addrows_current
and vm_concurrent_addrows_capacity
metrics exported by vmstorage
are replaced with vm_concurrent_insert_current
and vm_concurrent_insert_capacity
metrics in order to be consistent with the corresponding metrics exported by vminsert
. Please update queries in dahsboards and alerting rules with new metric names if old metric names are used there.
metric_relabel_configs
and the target labels when clicking on the debug metrics relabeling
link at the http://vmagent:8429/targets
page at the particular target. See these docs.job
/ instance
. See these docs and this feature request.RFC3339
date/time to time
, start
and end
query args at querying APIs and export APIs. For example, 2022
is equivalent to 2022-01-01T00:00:00Z
, while 2022-01-30T14
is equivalent to 2022-01-30T14:00:00Z
. See these docs.температура{город="Киев"}
is a valid MetricsQL expression now. Previously every non-ascii letters should be escaped with \
char when used inside MetricsQL expression: \т\е\м\п\е\р\а\т\у\р\а{\г\о\р\о\д="Киев"}
. Now both expressions are equivalent. Thanks to @hzwwww for the pull request.keepequal
and dropequal
relabeling actions, which are supported by Prometheus starting from v2.41.0. These relabeling actions are almost identical to keep_if_equal
and drop_if_equal
relabeling actions supported by VictoriaMetrics since v1.38.0
- see these docs - so it is recommended sticking to keep_if_equal
and drop_if_equal
actions instead of switching to keepequal
and dropequal
.-rule.updateEntriesLimit
command-line flag or per-rule via rule's update_entries_limit
configuration param. See these docs and this pull request.-maxConcurrentInserts
command-line flag. Previously this flag was limiting the number of concurrent connections from clients, which write data to VictoriaMetrics or vmagent. Some of these connections could be idle for some time. These connections do not need significant amounts of CPU and memory, so there is no sense in limiting their count. The updated logic behind -maxConcurrentInserts
limits the number of active insert requests, not counting idle connections.-httpAuth.*
command-line flag. Previously endpoints protected by -*AuthKey
command-line flags weren't protected by -httpAuth.*
. This could complicate the proper security setup. See this issue.-maxConcurrentInserts
and -insert.maxQueueDuration
command-line flags to vmstorage
, so they could be tuned if needed in the same way as at vminsert
nodes.vmstorage
proportionally to the number of available CPU cores, since every request can saturate a single CPU core at vmstorage
. Previously a single vmstorage
could accept and start processing arbitrary number of concurrent requests received from big number of vmselect
nodes. This could result in increased RAM, CPU and disk IO usage or event to out of memory crash at vmstorage
side under high load. The limit can be fine-tuned if needed via -search.maxConcurrentRequests
command-line flag at vmstorage
according to these docs. vmstorage
now exposes the following additional metrics at http://vmstorage:8482/metrics
page: vm_vmselect_concurrent_requests_capacity
- the maximum number of requests allowed to execute concurrentlyvm_vmselect_concurrent_requests_current
- the current number of concurrently executed requestsvm_vmselect_concurrent_requests_limit_reached_total
- the total number of requests, which were put in the wait queue when -search.maxConcurrentRequests
concurrent requests are being executedvm_vmselect_concurrent_requests_limit_timeout_total
- the total number of canceled requests because they were sitting in the wait queue for more than -search.maxQueueDuration
step
value in url after the step
input field has been manually changed. This allows preserving the proper step
when copy-n-pasting the url to another instance of web browser. See this issue..123
and 20.
during data import. See this issue.10S
, 5MS
, 1W
, etc. See this issue.vmagent
runs with -promscrape.dropOriginalLabels
command-line flag. See this issue. The bug has been introduced in v1.85.0.filters
field. See this issue.CamelCaseLabel
from /api/v1/labels. See this issue.opentsdb
HTTP endpoint not respecting -httpAuth.*
flags. See this issueSee changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See changes here
See releases page.