Year 2025

The following tip changes can be tested by building VictoriaMetrics components from the latest commits according to the following docs:

Metrics of the latest version of VictoriaMetrics cluster are available for viewing at our sandbox. The sandbox cluster installation runs under the constant load generated by prometheus-benchmark and is used for testing latest releases.

tip #

SECURITY: upgrade Go builder from Go1.23.5 to Go1.23.6. See the list of issues addressed in Go1.23.6.
SECURITY: upgrade base docker image (Alpine) from 3.21.2 to 3.21.3. See Alpine 3.21.3 release notes.
FEATURE: Single-node VictoriaMetrics and vmstorage: allow disabling per-day indexes for workloads with low or zero churn rates via -disablePerDayIndex cmd-line flag. This option works the best for IoT cases and significantly loads resource and disk usage. See this PR.
FEATURE: vmalert: add command-line flag -notifier.sendTimeout(default 10s) to allow configuring request timeout when sending alerts to the corresponding -notifier.url. See this issue for details. Thanks to @gjkim42 for the pull request.
FEATURE: vmagent and vmsingle: add support of aggregation windows to improve accuracy of stream aggregation. This feature is especially important for aggregation of histrograms, but enabling this feature requires additional memory. See this issue.
FEATURE: vmsingle, vminsert in VictoriaMetrics cluster and vmagent: add -influx.forceStreamMode cmd-line flag to force stream processing for data ingested via InfluxDB protocol. See this issue.
FEATURE: vmselect: support overriding vmstorage -maxUniqueTimeseries with specific resource limits: -search.maxLabelsAPISeries, -search. maxSeries, -search.maxTSDBStatusSeries and -search.maxDeleteSeries. See this issue.
FEATURE: vmui: add legend customization options - table view, hiding common values, grouping by label, and custom label format. See this issue and this pull request.
FEATURE: vmui: look back for recent data for longer time intervals when switching to Table or JSON view. The look back window depends on the step setting, which is now set as end - start (30m) by default. See this issue.
FEATURE: vmui: preserve user-defined step setting when changing the time interval of displayed query. See this comment.
FEATURE: vmui: improve numbers formatting for better readability on the Explore Cardinality page. See this issue.
FEATURE: vmui: print full error messages for failed queries on the Explore Cardinality page. Before, only response status code was printed.
FEATURE: vmui: move values representing changes relative to the previous day to a separate column for easier sorting on the Explore Cardinality page.
FEATURE: MetricsQL: support auto-format (prettify) for expressions that use quoted metric or label names. See this issue for details.
BUGFIX: all the VictoriaMetrics components: properly override basic authorization for API endpoints protected with authKey. See this issue for details.
BUGFIX: vmalert: fix polluted alert messages when multiple Alertmanager instances are configured with alert_relabel_configs. See this issue, and thanks to @evkuzin for the pull request.
BUGFIX: vminsert in VictoriaMetrics cluster and vmstorage in VictoriaMetrics cluster: properly return parsing errors during data ingestion in multi-level cluster setup. Before, some of the errors could have been silently ignored.
BUGFIX: Single-node VictoriaMetrics and vmselect: properly enforce presence of default -retentionPeriod configuration for retention filters debug interface. See this issue.

v1.111.0 #

Released at 2025-02-10

Update note 1: Single-node VictoriaMetrics and vmstorage stop exposing vm_index_search_duration_seconds histogram metric. This metric records time spent on search operations in the index. It was introduced in v1.56.0. However, this metric was used neither in dashboards nor in alerting rules. It also has high cardinality because index search operations latency can differ by 3 orders of magnitude. Hence, dropping it as unused.

FEATURE: Single-node VictoriaMetrics and vmstorage: improve startup times when opening a storage with the retention exceeding a few months.
FEATURE: vmui: add the ability to switch the heatmap to a line chart. Now, vmui would suggest to switch to line graph display if heatmap can’t be properly rendered. See this issue.
FEATURE: vmauth: add -httpInternalListenAddr cmd-line flag to serve internal HTTP routes /metrics, /flags, etc. It allows properly route requests to backends with the same service routes as vmauth. See this issue and this issue for details.
FEATURE: expose /-/healthy and /-/ready endpoints as Prometheus does on vmselect endpoints, thus achieving full Prometheus compatibility for external dependencies.
BUGFIX: vmagent: properly perform graceful shutdown for kafka client.
BUGFIX: vmagent: properly add message metadata headers for kafka remoteWrite target.
BUGFIX: vmui: improve clipboard error handling in tables and code snippets by showing detailed messages with possible reasons. See this pull request.
BUGFIX: vmui for VictoriaMetrics enterprise components: properly display enterprise features when the enterprise version is used.
BUGFIX: Single-node VictoriaMetrics and vmselect: fix discrepancies when using or binary operator. See this and this issues for details.
BUGFIX: vmsingle and vmstorage in VictoriaMetrics cluster: properly update number of unique series for cardinality limiter on ingestion. Previously, limit could undercount the real number of the ingested unique series.

v1.110.1 #

Released at 2025-02-10

v1.110.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.110.x line will be supported for at least 12 months since v1.110.0 release

BUGFIX: vmagent: properly perform graceful shutdown for kafka client.
BUGFIX: vmagent: properly add message metadata headers for kafka remoteWrite target.
BUGFIX: vmui: improve clipboard error handling in tables and code snippets by showing detailed messages with possible reasons. See this pull request.
BUGFIX: vmsingle and vmstorage in VictoriaMetrics cluster: properly update number of unique series for cardinality limiter on ingestion. Previously, limit could undercount the real number of the ingested unique series.
BUGFIX: Single-node VictoriaMetrics and vmselect: fix discrepancies when using or binary operator. See this and this issues for details.
BUGFIX: vmui for VictoriaMetrics enterprise components: properly display enterprise features when the enterprise version is used.

v1.102.13 #

Released at 2025-02-10

v1.102.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.102.x line will be supported for at least 12 months since v1.102.0 release

BUGFIX: export API: cancel export process on client connection close. Previously client connection close was ignored and VictoriaMetrics started to hog CPU by exporting metrics to nowhere until it export all of them.
BUGFIX: Single-node VictoriaMetrics and vmselect: fix discrepancies when using or binary operator. See this and this issues for details.
BUGFIX: vmsingle and vmstorage in VictoriaMetrics cluster: properly update number of unique series for cardinality limiter on ingestion. Previously, limit could undercount the real number of the ingested unique series.

v1.102.12 #

Released at 2025-01-28

BUGFIX: vmsingle, vminsert in VictoriaMetrics cluster and vmagent: log metric names for signals with unsupported delta temporality on ingestion via OpenTelemetry protocol for metrics. Thanks to @chenlujjj for the pull request.
BUGFIX: Single-node VictoriaMetrics and vmselect: respect staleness detection in increase, increase_pure and delta functions when time series has gaps and -search.maxStalenessInterval is set. See this issue for details.

v1.97.17 #

Released at 2025-01-28

v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release

BUGFIX: vmsingle, vminsert in VictoriaMetrics cluster and vmagent: log metric names for signals with unsupported delta temporality on ingestion via OpenTelemetry protocol for metrics. Thanks to @chenlujjj for the pull request.
BUGFIX: Single-node VictoriaMetrics and vmselect: respect staleness detection in increase, increase_pure and delta functions when time series has gaps and -search.maxStalenessInterval is set. See this issue for details.

v1.110.0 #

Released at 2025-01-24

SECURITY: upgrade Go builder from Go1.23.4 to Go1.23.5. See the list of issues addressed in Go1.23.5.
FEATURE: MetricsQL: allow executing queries with $__interval and $__rate_interval - these placeholders are automatically replaced with 1i (e.g. step arg value at /api/v1/query_range) during query execution. This simplifies copying queries from Grafana dashboards.
FEATURE: MetricsQL: allow specifying metric names in quotes inside {} braces: {"metric_name", job="foo"} is equal to metric_name{job="foo"}. See this issue for details.
FEATURE: vmsingle and vmselect in VictoriaMetrics cluster: add command-line flag -search.maxDeleteDuration(default 5m) to limit the duration of the /api/v1/admin/tsdb/delete_series call. Previously, the call is limited by -search.maxQueryDuration.
FEATURE: dashboards: all dashboards that use VictoriaMetrics Grafana datasource were updated to use a new datasource ID.
FEATURE: vmui: reflect column settings for the table view in URL, so the table view can be shared via link. See this issue.
BUGFIX: vmauth: fix possible runtime panic during requests processing under heavy load. See this issue for details.
BUGFIX: vmselect: fix panic when trying to delete series by using multitenant read endpoint. See this issue for the details.
BUGFIX: vmselect: prevent panic when vmselect receives an error response from vmstorage during the query execution and request processing for other vmstorage nodes is still in progress. See this issue for the details.
BUGFIX: all VictoriaMetrics enterprise components: properly trim whitespaces at the end of license provided via -license and -licenseFile command-line flags. Previously, the trailing whitespaces could cause the license verification to fail.
BUGFIX: Single-node VictoriaMetrics and vmselect: respect staleness detection in increase, increase_pure and delta functions when time series has gaps and -search.maxStalenessInterval is set. See this issue for details.
BUGFIX: vmsingle, vminsert in VictoriaMetrics cluster and vmagent: allow ingesting histograms with missing _sum metric via OpenTelemetry ingestion protocol in the same way as Prometheus does.
BUGFIX: vmui: fix an issue where pressing the “Enter” key in the query editor did not execute the query. See this issue.
BUGFIX: export API: cancel export process on client connection close. Previously client connection close was ignored and VictoriaMetrics started to hog CPU by exporting metrics to nowhere until it export all of them.

v1.102.11 #

Released at 2025-01-24

It is recommended upgrading to v1.102.12 because v1.102.11 contains a bug with staleness detection.

SECURITY: upgrade Go builder from Go1.23.4 to Go1.23.5. See the list of issues addressed in Go1.23.5.
BUGFIX: vmauth: fix possible runtime panic during requests processing under heavy load. See this issue for details.
BUGFIX: all VictoriaMetrics enterprise components: properly trim whitespaces at the end of license provided via -license and -licenseFile command-line flags. Previously, the trailing whitespaces could cause the license verification to fail.
BUGFIX: all VictoriaMetrics enterprise components: remove unnecessary delay before failing if all online verification attempts have failed. This should reduce the time required for the component to proceed if all online verification attempts have failed.
BUGFIX: Single-node VictoriaMetrics and vmselect: don’t take into account the last raw sample before the lookbehind window is sample exceeds the staleness interval. This affects correctness of increase, increase_pure, delta functions when preforming calculations on time series with gaps. See this pull request for details.
BUGFIX: vmsingle, vminsert in VictoriaMetrics cluster and vmagent: allow ingesting histograms with missing _sum metric via OpenTelemetry ingestion protocol in the same way as Prometheus does.

v1.97.16 #

Released at 2025-01-24

It is recommended upgrading to v1.97.17 because v1.97.16 contains a bug with staleness detection.

BUGFIX: all VictoriaMetrics enterprise components: remove unnecessary delay before failing if all online verification attempts have failed. This should reduce the time required for the component to proceed if all online verification attempts have failed.
BUGFIX: Single-node VictoriaMetrics and vmselect: don’t take into account the last raw sample before the lookbehind window is sample exceeds the staleness interval. This affects correctness of increase, increase_pure, delta functions when preforming calculations on time series with gaps. See this pull request for details.
BUGFIX: vmsingle, vminsert in VictoriaMetrics cluster and vmagent: allow ingesting histograms with missing _sum metric via OpenTelemetry ingestion protocol in the same way as Prometheus does.

v1.109.1 #

Released at 2025-01-17

SECURITY: upgrade base docker image (Alpine) from 3.21.0 to 3.21.2. See Alpine 3.21.1 release notes and Alpine 3.21.2 release notes.
BUGFIX: vmsingle, vminsert in VictoriaMetrics cluster and vmagent: log metric names for signals with unsupported delta temporality on ingestion via OpenTelemetry protocol for metrics. Thanks to @chenlujjj for the pull request.
BUGFIX: Single-node VictoriaMetrics and vmselect: fix incorrect behavior of increase, increase_pure, delta caused by this pull request. This fix reverts to the previous behavior before v1.109.0. But allows controlling staleness detection for these functions explicitly via -search.maxStalenessInterval.
BUGFIX: all VictoriaMetrics enterprise components: remove unnecessary delay before failing if all online verification attempts have failed. This should reduce the time required for the component to proceed if all online verification attempts have failed.
BUGFIX: vmselect: prevent panic when sending multitenant read request to /api/v1/series/count. See this issue for the details.

v1.109.0 #

Released at 2025-01-14

Update note: This release contains issue that might affect correctness of increase, increase_pure and delta functions. It affects only vmselect or VictoriaMetrics single-node on query time. Please, rollback vmselects to the closest previous version or update to v1.109.1.

FEATURE: all the VictoriaMetrics components: increase the default value for GOGC from 30 to 100. This should reduce CPU usage at the cost of slightly higher memory usage. Single-node VictoriaMetrics, vmagent and vmstorage components continue using GOGC=30, since they are optimized for low memory allocations and low memory usage, so they do not benefit from the increased GOGC value too much. It is possible to override the default GOGC value in any VictoriaMetrics component by setting GOGC environment variable to the desired value. For example, GOGC=200 ./path/to/vmagent starts vmagent with GOGC=200. See these docs about GOGC tuning. See this issue.
FEATURE: vmui: add export data functionality for the Raw Query page and the ability to import exported data into the Query Analyzer page. See this issue.
FEATURE: vmui: add markdown support for comments during data export. this pull request.
FEATURE: vmagent and Single-node VictoriaMetrics: added min and max metrics for Datadog Sketches API metrics, changed _ metric name separator to . if metrics are not sanitized for consistency.
FEATURE: vmagent and Single-node VictoriaMetrics: add service discovery support for Marathon. See these docs and this issue.
FEATURE: Single-node VictoriaMetrics: support -maxIngestionRate cmd-line flag to ratelimit samples/sec ingested. See this issue for details.
FEATURE: vmsingle and vmselect in VictoriaMetrics cluster: improve query performance on systems with high number of CPU cores. See this PR for details.
FEATURE: vmsingle and vmselect in VictoriaMetrics cluster: add command-line flag -search.maxBinaryOpPushdownLabelValues to allow using labels with more candidate values as push down filter in binary operation. See this pull request. Thanks to @tydhot for implementation.
FEATURE: vmsingle and vmstorage in VictoriaMetrics cluster: add command-line flag storage.finalDedupScheduleCheckInterval to control the final deduplication process interval. See this issue for details.
FEATURE: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: add a separate cache type for storing sparse entries when performing large index scans. This significantly reduces memory usage when applying downsampling filters and retention filters during background merge. See this issue for the details.
BUGFIX: dashboards: consistently use vmagent_remotewrite_pending_data_bytes on vmagent dashboard to represent persistent queue size.
BUGFIX: vmalert: fix the auto-generated metrics ALERTS and ALERTS_FOR_STATE for alerting rules. Previously, metrics might have incorrect labels and affect the restore process. See this issue.
BUGFIX: vmalert: do not append tenant info to VictoriaLogs datasource request path in clusterMode. See this doc for how to use multitenancy in VictoriaLogs.
BUGFIX: vmauth: properly set host field at debug information formatted with dump_request_on_errors: true setting.
BUGFIX: vmauth: properly handle discovery for ipv6 addresses. Thanks to @badie for the pull request.
BUGFIX: vmctl: fix support for migrating influx series without any tag. See this issue. Thanks to @bitbidu for reporting.
BUGFIX: vminsert in VictoriaMetrics cluster: storage nodes defined in -storageNode are now sorted, ensuring that varying node orders across different vminsert instances do not result in inconsistent replication.
BUGFIX: vmsingle and vminsert in VictoriaMetrics cluster: properly ingest influx line protocol metrics with empty tags. See this issue for details.
BUGFIX: vmselect: allow to override the default unique time series limit in vmstorage with command-line flags like -search.maxUniqueTimeseries, -search.maxLabelsAPISeries. See this issue.
BUGFIX: vmselect: properly set tenancy information when evaluating numbers in a query. Previously, the tenancy information could lead to mismatch between query result and a number. See this issue for the details.
BUGFIX: vmalert: fix marshaling of request body for Alertmanager notification. See this issue for the details.
BUGFIX: Single-node VictoriaMetrics and vmselect: don’t take into account the last raw sample before the lookbehind window is sample exceeds the staleness interval. This affects correctness of increase, increase_pure, delta functions when preforming calculations on time series with gaps. See this pull request for details.

v1.102.10 #

Released at 2025-01-14

FEATURE: all VictoriaMetrics enterprise components: add support of hot-reload for license key supplied by -licenseFile command-line flag.
BUGFIX: vmsingle, vmselect in VictoriaMetrics cluster: make instant query results consistent. VictoriaMetrics detects and adjusts scrape interval and while this is very useful for range queries (i.e. this eliminates gaps on the graph), it may cause instant queries to return a non-empty result when no result is expected. The fix is to disable scrape interval detection and always use the step as the scrape interval in instant queries. This will guarantee that the samples are searched within the (time-step, time] interval. See this issue for details.
BUGFIX: vmui: fix cursor reset in query input field. See this issue.
BUGFIX: vmauth: properly handle discovery for ipv6 addresses. Thanks to @badie for the pull request.
BUGFIX: vmctl: fix support for migrating influx series without any tag. See this issue. Thanks to @bitbidu for reporting.
BUGFIX: vminsert in VictoriaMetrics cluster: storage nodes defined in -storageNode are now sorted, ensuring that varying node orders across different vminsert instances do not result in inconsistent replication.

v1.97.15 #

Released at 2025-01-14

FEATURE: all VictoriaMetrics enterprise components: add support of hot-reload for license key supplied by -licenseFile command-line flag.
BUGFIX: vmsingle, vmselect in VictoriaMetrics cluster: make instant query results consistent. VictoriaMetrics detects and adjusts scrape interval and while this is very useful for range queries (i.e. this eliminates gaps on the graph), it may cause instant queries to return a non-empty result when no result is expected. The fix is to disable scrape interval detection and always use the step as the scrape interval in instant queries. This will guarantee that the samples are searched within the (time-step, time] interval. See this issue for details.
BUGFIX: vmctl: fix support for migrating influx series without any tag. See this issue. Thanks to @bitbidu for reporting.
BUGFIX: vminsert in VictoriaMetrics cluster: storage nodes defined in -storageNode are now sorted, ensuring that varying node orders across different vminsert instances do not result in inconsistent replication.

Previous releases #

See releases page.

Year 2025 #

tip #