victoriametrics.com

The following tip changes can be tested by building VictoriaMetrics components from the latest commits according to the following docs:

Metrics of the latest version of VictoriaMetrics cluster are available for viewing at our sandbox. The sandbox cluster installation runs under the constant load generated by prometheus-benchmark and is used for testing latest releases.

See also LTS releases.

tip #

  • FEATURE: all the VictoriaMetrics components: increase the default value for GOGC from 30 to 100. This should reduce CPU usage at the cost of slightly higher memory usage. Single-node VictoriaMetrics, vmagent and vmstorage components continue using GOGC=30, since they are optimized for low memory allocations and low memory usage, so they do not benefit from the increased GOGC value too much. It is possible to override the default GOGC value in any VictoriaMetrics component by setting GOGC environment variable to the desired value. For example, GOGC=200 ./path/to/vmagent starts vmagent with GOGC=200. See these docs about GOGC tuning. See this issue.

  • FEATURE: vmui: add export data functionality for the Raw Query page and the ability to import exported data into the Query Analyzer page. See this issue.

  • FEATURE: vmui: add markdown support for comments during data export. this pull request.

  • FEATURE: vmagent and Single-node VictoriaMetrics: added min and max metrics for Datadog Sketches API metrics, changed _ metric name separator to . if metrics are not sanitized for consistency.

  • FEATURE: Single-node VictoriaMetrics: support -maxIngestionRate cmd-line flag to ratelimit samples/sec ingested. See this issue for details.

  • FEATURE: vminsert: Storage nodes defined in -storageNode are now sorted, ensuring that varying node orders across different vminsert instances do not result in inconsistent replication.

  • BUGFIX: vmauth: properly set host field at debug information formatted with dump_request_on_errors: true setting.

v1.108.1 #

Released at 2024-12-18

  • BUGFIX: vmsingle properly apply -relabelConfig rules for data ingestion protocols. Issue was introduced at v1.108.0 and only affects vmsingle. release. See this issue for details.
  • BUGFIX: vmsingle, vmselect in VictoriaMetrics cluster: make instant query results consistent. VictoriaMetrics detects and adjusts scrape interval and while this is very useful for range queries (i.e. this eliminates gaps on the graph), it may cause instant queries to return a non-empty result when no result is expected. The fix is to disable scrape interval detection and always use the step as the scrape interval in instant queries. This will guarantee that the samples are searched within the (time-step, time] interval. See this issue for details.
  • BUGFIX: vmui: fix cursor reset in query input field. See this issue.

v1.108.0 #

Released at 2024-12-13

It is recommended upgrading to v1.108.1 because v1.108.0 contains a bug introduced at v1.108.0, which incorrectly applies -relabelConfig rules at vmsingle components. See this issue for details.

Update note 1: meaning of -maxLabelsPerTimeseries and -maxLabelValueLen has been changed. Previously, excessive labels, label names and values were truncated and could result in silent data collision. With the new change time series that are hitting the limits will be dropped instead. These events are reflected in logs and in vm_rows_ignored_total metric. The previous metrics vm_too_long_label_values_total, vm_too_long_label_names_total, vm_metrics_with_dropped_labels_total are deprecated.

Update note 2: docker_sd_configs now by default matches only first network if the container has multiple networks defined. This change aligns with Prometheus SD behavior. Please see this issue if you need to match multiple networks.

  • SECURITY: upgrade Go builder from Go1.23.3 to Go1.23.4. See the list of issues addressed in Go1.23.4.

  • SECURITY: upgrade base docker image (Alpine) from 3.20.3 to 3.21.0. See alpine 3.21.0 release notes.

  • FEATURE: Single-node VictoriaMetrics and vminsert in VictoriaMetrics cluster: changes behaviour of meaning of -maxLabelsPerTimeseries and -maxLabelValueLen flag limits, time series that are hitting the limits will be dropped. And removes corresponding metrics vm_too_long_label_values_total, vm_too_long_label_names_total, vm_metrics_with_dropped_labels_total. See this issue.

  • FEATURE: vmagent: support -maxLabelsPerTimeseries, -maxLabelNameLen and -maxLabelValueLen flags to limit amount of labels, label name and value length for pushed or scraped series. See this issue.

  • FEATURE: Single-node VictoriaMetrics, vminsert in VictoriaMetrics cluster and vmagent: adds new values [too_many_labels,too_long_label_name,too_long_label_value] to reason label of the vm_rows_ignored_total exposed metric name. See this issue.

  • FEATURE: vmagent and Single-node VictoriaMetrics: add match_first_network support for docker service discovery. It uses the first network if the container has multiple networks defined. See this issue.

  • FEATURE: vmagent and Single-node VictoriaMetrics: Docker service discovery now supports containers discovery via linked networks. See this PR for details.

  • FEATURE: dashboards: add NodeBecomesReadonlyIn3Days alert for detecting storages that will switch to read-only mode soon.

  • FEATURE: vmauth: allow to start vmauth with empty configuration file. See this issue for details.

  • FEATURE: vmalert-tool: support debug mode for alerting rule. See this doc.

  • FEATURE: vmui: update error messages for Clipboard API issues with docs links. See this issue.

  • FEATURE: Single-node VictoriaMetrics: support -maxIngestionRate flag to ratelimit samples/sec ingested

  • BUGFIX: all VictoriaMetrics components: consistently deduplicate values with stale markers within deduplication interval. Previously, deduplication could randomly prefer stale marker or value on the deduplication interval. See this issue for details. Thanks to @tIGO for the pull request.

  • BUGFIX: vmagent and Single-node VictoriaMetrics: add missing common service labels for docker swarm service discovery when role is set to tasks. See this issue.

  • BUGFIX: vmalert: properly template rule annotations and labels with define and block template functions. See this PR for details.

  • BUGFIX: vmalert: properly reload external templates defined with -rule.templates. Previously externalURL function worked incorrectly. See this PR for details.

  • BUGFIX: vmauth: fix requests routing by host when using src_hosts. Previously, request header could be ignored.See this PR for details.

  • BUGFIX: vmbackupmanager: prevent backup scheduler from scheduling two backups immediately one after another.

  • BUGFIX: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: properly parse the query rollup window specified in milliseconds. Previous implementation could lead to precision issues resulting in the parsed window being smaller by 1ms. See this issue for details.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: properly schedule historical data de-duplication at enterprise version with -dedup.minScrapeInterval configured. See this issue for details. Issue was introduced at v1.106.1 release.

  • BUGFIX: vmui: prevent accordion from collapsing when selecting text in headers. See this issue.

  • BUGFIX: vmui: fix query interval display in the instant query info. See this issue.

  • BUGFIX: vmselect in VictoriaMetrics cluster: respect -search.skipSlowReplicas when -globalReplicationFactor > 1. See this issue.

v1.107.0 #

Released at 2024-11-29

Update note 1: cmd-line flags -datasource.lookback, datasource.queryTimeAlignment and remoteRead.ignoreRestoreErrors on vmalert have been removed. Those flags were deprecated before v1.101.0.

  • FEATURE: vmui: add Raw Query tab for displaying raw data. See this issue.

  • FEATURE: vmauth: add dump_request_on_errors bool setting to auth config for debugging HTTP requests that missed routing rules. This should improve debugability of vmauth settings.

  • FEATURE: vmauth: add real_ip_header setting to ip_filters and corresponding global flag httpRealIPHeader. It allows vmauth obtain client IP from HTTP headers for filtering. See this issue for details.

  • FEATURE: vmauth: expose new metrics vmauth_user_ip_denies_total, vmauth_global_ip_denies_total and vmauth_unauthorized_user_ip_denies_total for ip_filters. See this issue for details.

  • FEATURE: vmauth: add dryRun flag to validate configuration. See this issue for details.

  • FEATURE: vmauth: add removeXFFHTTPHeaderValue flag to remove content of X-Forwarded-For HTTP Header before proxy it to the backend. See this issue for details.

  • FEATURE: vmalert: revert the default value of -remoteWrite.maxQueueSize from 1_000_000 to 100_000. It was bumped in v1.104.0, which increases memory usage and is not needed for most setups. See this issue.

  • FEATURE: stream aggregation: add ignore_first_sample_interval param to aggregation config. It allows users to control the time interval when aggregation skips sending aggregated samples to avoid unexpected spikes in values. By default, this interval is set to x2 of staleness_interval. The new setting is applicable only to total, total_prometheus, increase, increase_prometheus and histogram_bucket outputs. See this issue for details. Thanks to @iyuroch for the pull request.

  • FEATURE: vmui: add support for template alias in predefined panels. This allows creating more readable metric names in the legend using constructions like {{label_name}}, where label_name is the name of the label. See this commit

  • BUGFIX: vmagent: properly parse multitenant token value for multitenant endpoints. Before, pushing data into vmagent using multitenant URL resulted in unsupported path. See this issue.

  • BUGFIX: vmagent: return 200 OK HTTP status code when importing data via Pushgateway protocol using multitenant URL format. See this issue and this pull request.

  • BUGFIX: vmagent: properly set TCP connection timeout for Kubernetes API server connection for metric scrapping with kubernetes_sd_configs. See this issue.

  • BUGFIX: vmagent: fix the resource_group filter for Azure service discovery on virtual machine scale sets. Previously, this filter did not apply to virtual machine scale sets, causing all virtual machines to be discovered. See this issue.

  • BUGFIX: vmselect in VictoriaMetrics cluster: fix possible runtime panic introduced at v1.104.0 release. See this issue for details.

  • BUGFIX: vmselect in VictoriaMetrics cluster: properly handle multitenant query request errors and correctly perform search for available tenants. See this issue for details. This is follow-up after v1.106.1 release changes.

  • BUGFIX: vmselect in VictoriaMetrics cluster: properly set multitenant cache expiration duration with search.tenantCacheExpireDuration. Previously flag always used default value.

  • BUGFIX: vmauth: properly init ip_filters for unauthorized_user config section. Previously it was ignored and vmauth didn’t apply ip_filter for unauthorized access.

  • BUGFIX: vmsingle, vmselect in VictoriaMetrics cluster: properly return result for binary operation ^ aka pow at query requests for NaN values. See this issue for details.

  • BUGFIX: vmui: fix rendering of isolated data points on the graph that are not connected to other points.

  • BUGFIX: vmui: fix for showLegend and alias flags in predefined panels. See this issue

  • BUGFIX: vmalert: improve the correctness of alert state restoration. Previously, it could result in false-positive alerts if alert was resolved shortly before vmalert restart.

  • BUGFIX: vmalert-tool: exit immediately with error message if no test file is found under specified -files.

  • BUGFIX: vmalert-tool: print an error message if no rule group is found in rule_files.

v1.106.1 #

Released at 2024-11-15

It is recommended upgrading to v1.107.0 because v1.106.1 contains a bug introduced at v1.104.0, which can lead to runtime panic at vmselect component. See this issue for details.

v1.106.0 #

Released at 2024-11-04

It is recommended upgrading to v1.107.0 because v1.106.0 contains a bug introduced at v1.104.0, which can lead to runtime panic at vmselect component. See this issue for details.

  • FEATURE: vmalert: support VictoriaLogs as a datasource. See this doc for details.

  • FEATURE: vmalert: -rule cmd-line flag now supports multi-document YAML files. This could be useful when rules are retrieved via HTTP URL where multiple rule files were merged together in one response. See this issue. Thanks to @Irene-123 for the pull request.

  • FEATURE: vmagent: add service discovery support for PuppetDB resources. See these docs and this issue.

  • FEATURE: vmagent: support scraping from Kubernetes Native Sidecars. See this issue.

  • FEATURE: vmagent and Single-node VictoriaMetrics: add metric_relabel_configs and relabel_configs to the global section of scrape configuration. Relabeling configuration specified in global section will be pre-pended to relabeling configs of all jobs. See this issue and these docs for details.

  • FEATURE: dashboards for VM single-node, cluster, vmalert, vmagent, VictoriaLogs: add Restarts panel to show the events of process restarts. This panel should help correlate events of restart with unexpected behavior of processes.

  • FEATURE: alerts: add alerting rule RemoteWriteDroppingData to track number of dropped samples that weren’t sent to remote write URL.

  • FEATURE: vmalert-tool: use evaluation_interval as default interval value in test_group. This change aligns with promtool behavior.

  • BUGFIX: dashboards for Single-node VictoriaMetrics, cluster: The free disk space calculation now will subtract the size of the -storage.minFreeDiskSpaceBytes flag to correctly display the remaining available space of Single-node VictoriaMetrics/vmstorage rather than the actual available disk space, as well as the full ETA. See this issue for the details.

  • BUGFIX: vmalert: properly set group_name and file fields for recording rules in /api/v1/rules.

  • BUGFIX: vmgateway: properly forward multitenant requests with -clusterMode. See this issue for details.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: prevent panic when ingesting samples which are outside of configured retention filters. This could happen when backfilling data with retention filters which exclude samples from the backfill range.

  • BUGFIX: vmctl: fix issue with series matching for vmctl vm-native with --vm-native-disable-per-metric-migration flag enabled. See this issue.

  • BUGFIX: vmui: fix the display of the link to vmalert. See this issue.

v1.105.0 #

Released at 2024-10-21

It is recommended upgrading to v1.107.0 because v1.105.0 contains a bug introduced at v1.104.0, which can lead to runtime panic at vmselect component. See this issue for details.

Update note 1: -search.maxUniqueTimeseries limit on vmselect can no longer exceed -search.maxUniqueTimeseries limit on vmstorage. If you don’t set this flag at vmstorage, then it will be automatically calculated based on available resources. This can result into rejecting expensive read queries if they exceed auto-calculated limit. The limit can be overridden by manually setting -search.maxUniqueTimeseries at vmstorage, but for better reliability we recommend sticking to default values. Refer to the CHANGELOG below and this issue.

  • FEATURE: vmsingle, vminsert in VictoriaMetrics cluster and vmagent: add support of exponential histograms ingested via OpenTelemetry protocol for metrics. Such histograms will be automatically converted to VictoriaMetrics histogram format. See this pull request.

  • FEATURE: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: automatically set -search.maxUniqueTimeseries limit based on available memory and -search.maxConcurrentRequests. The more memory is available to the process and the lower is -search.maxConcurrentRequests, the higher will be -search.maxUniqueTimeseries limit. This should protect vmstorage from expensive queries without the need to manually set -search.maxUniqueTimeseries. The calculated limit will be printed during process start-up logs and exposed as vm_search_max_unique_timeseries metric. Set -search.maxUniqueTimeseries manually to override auto calculation. Please note, -search.maxUniqueTimeseries on vmselect can’t exceed the same name limit on vmstorage, it can only be set to lower values. See this issue.

  • FEATURE: vmsingle, vminsert in VictoriaMetrics cluster and vmagent: disable stream processing mode for data ingested via InfluxDB HTTP endpoints by default. With this change, the data is processed in batches (see -influx.maxRequestSize) and user will get parsing errors immediately as they happen. This also improves users’ experience and resiliency against thundering herd problems caused by clients without backoff policies like telegraf. To enable stream mode back, pass HTTP header Stream-Mode: "1" with each request. For data sent via TCP and UDP (see -influxListenAddr) protocols streaming processing remains enabled.

  • FEATURE: vmselect in VictoriaMetrics cluster: set default value for -search.maxUniqueTimeseries to 0. See this issue.

  • FEATURE: vmagent: allow using HTTP/2 client for Kubernetes service discovery if -promscrape.kubernetes.useHTTP2Client cmd-line flag is set. This could help to reduce the amount of opened connections to the Kubernetes API server. See this issue for the details.

  • FEATURE: add Darwin binaries for VictoriaMetrics cluster to the release flow. The binaries will be available in the new release.

  • FEATURE: vmalert: -rule cmd-line flag now supports multi-document YAML files. This could be useful when rules are retrieved via HTTP URL where multiple rule files were merged together in one response. See this issue. Thanks to @Irene-123 for the pull request.

  • FEATURE: vmgateway: allow parsing scope claim parsing in array format. This is useful for cases when identity provider does encode claims in array format.

  • FEATURE: vmui: add the ability to cancel running queries. See this issue.

  • FEATURE: vmui: add playgrounds for retention filters and downsampling configurations. These new pages are available in vmui in the Tools section. See this issue.

  • BUGFIX: vmgateway: fix possible panic during parsing of a token without vm_access claim. This issue was introduced in v1.104.0.

  • BUGFIX: vmui: fix error messages rendering from overflowing the screen with long messages. See this issue.

  • BUGFIX: vmctl: properly add metrics tags for opentsdb migration source. Previously it could have empty values. See this PR.

  • BUGFIX: vmalert-tool: reduce the initial health check interval for datasource. This reduces the time spent on evaluating rules by vmalert-tool. See this issue.

  • BUGFIX: vmalert-tool: allow specifying empty labels list labels: '{}' in the same way as promtool does. This improves compatibility between these tools.

  • BUGFIX: vmagent: support m unit for minutes duration in command-line flag -streamAggr.dedupInterval. Previously unit m wasn’t supported correctly.

  • BUGFIX: VictoriaMetrics cluster: properly apply replication factor when storage node groups are used and replication factor is configured via global value such as -replicationFactor=2. Previously, global replication factor was ignored for storage node groups. See these docs for more information about storage groups configuration.

  • BUGFIX: vmselect in VictoriaMetrics cluster: properly process response in multi-level cluster setup. Before, vmselect could return no data in multi-level setup. See this issue for details. The issue was introduced in v1.104.0.

  • BUGFIX: vmalert: properly apply configuration changes during hot-reload to rule groups that haven’t started yet. Previously, configuration updates to such groups could have resulted into blocking all evaluations within the group, until vmalert restart.

  • BUGFIX: Single-node VictoriaMetrics and vmagent: properly update max_scrape_size param change during hot-reload. See this issue.

v1.104.0 #

Released at 2024-10-02

It is recommended upgrading to v1.107.0 because v1.104.0 contains a bug, which can lead to runtime panic at vmselect component. See this issue for details.

Update note 1: *.passwordFile and similar flags are trimming trailing whitespaces at the end of content. If authorization check performed with *.passwordFile content, make sure to update authorization settings to not include trailing whitespaces before the upgrade. In case of operator managed installations, make sure to update operator version to v0.48.*. See this issue for the details. This change reverts behavior introduced at v1.102.0-rc2 release

  • SECURITY: upgrade Go builder from Go1.23.0 to Go1.23.1. See the list of issues addressed in Go1.23.1.

  • SECURITY: upgrade base docker image (Alpine) from 3.20.2 to 3.20.3. See alpine 3.20.3 release notes.

  • FEATURE: vmselect in VictoriaMetrics cluster: add support for querying over multiple tenants. See these docs and this issue for details.

  • FEATURE: vmagent and Single-node VictoriaMetrics: hide jobs that contain only healthy targets when show_only_unhealthy filter is enabled. Before, jobs without unhealthy targets were still displayed on the page. See this issue.

  • FEATURE: vmagent: add service discovery support for OVH Cloud VPS and OVH Cloud dedicated server. See these docs and this issue.

  • FEATURE: vmalert: bump default values for sending data to remoteWrite.url: remoteWrite.maxQueueSize from 100_000 to 1_000_000, remoteWrite.maxBatchSize from 1_000 to 10_000, remoteWrite.concurrency from 1 to 4. The new settings should improve remote write performance of vmalert with default settings.

  • FEATURE: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: introduce the -search.maxDeleteSeries command line flag to limit the number of series that can be deleted by a single /api/v1/admin/tsdb/delete_series call. Previously, one could delete any number of series and if this number was big (millions) the deletion could result in OOM. See this issue for details.

  • FEATURE: vmbackupmanager: add API for creating/scheduling backups. See documentation

  • FEATURE: all VictoriaMetrics enterprise components: add support of hot-reload for license key supplied by -licenseFile command-line flag.

  • FEATURE: vmgateway: allow disabling Bearer prefix enforcement for authentication header. This is useful for cases when identity token is used instead of access token.

  • FEATURE: vmgateway: support parsing vm_access claims in string format. This is useful for cases when identity provider does not support mapping claims to JSON format.

  • FEATURE: vmagent: respect Retry-After header when making retries for pushing the data to remote destination via remote write protocol. See this issue.

  • FEATURE: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: add new metrics for data ingestion: vm_rows_received_by_storage_total, vm_rows_ignored_total{reason="invalid_raw_metric_name"}, vm_rows_ignored_total{reason="hourly_limit_exceeded"}, vm_rows_ignored_total{reason="daily_limit_exceeded"}. See this PR for details.

  • FEATURE: vmui: change request method for /query_range and /query calls from GET to POST. See this issue.

  • FEATURE: vmui: add link to vmalert when proxy is enabled. See this issue.

  • FEATURE: vmui: keep selected columns in table view on page reloads. Before, selected columns were reset on each update. See this issue.

  • FEATURE: dashboards for VM single-node, cluster, vmalert, vmagent, VictoriaLogs: add Go scheduling latency panel to show the 99th quantile of Go goroutines scheduling. This panel should help identifying insufficient CPU resources for the service. It is especially useful if CPU gets throttled, which now should be visible on this panel.

  • FEATURE: dashboards: update dashboard for VictoriaMetrics k8s operator: replace deprecated TimeSeries panels, add latency panels for rest client and Golang scheduler, add overview panels. The dashboard is now also available for VictoriaMetrics Grafana datasource.

  • FEATURE: alerts: add alerting rule to track the Go scheduling latency for goroutines. It should notify users if VM component doesn’t have enough CPU to run or gets throttled.

  • BUGFIX: vmagent fix service discovery of Azure Virtual Machines for response contains nextLink in Host:Port format. See this issue.

  • BUGFIX: vmagent: properly consume messages from kafka. Previously vmagent could skip some messages during start-up.

  • BUGFIX: stream aggregation: perform deduplication for all received data when specifying -streamAggr.dedupInterval or -remoteWrite.streamAggr.dedupInterval command-line flags are set. Previously, if the -remoteWrite.streamAggr.config or -streamAggr.config is set, only series that matched aggregation config were deduplicated. See this issue for details.

  • BUGFIX: stream aggregation: fix -remoteWrite.streamAggr.dropInputLabels labels parsing. Now, this flag allows specifying a list of labels to drop (by using ‘^^’ separator, i.e. dropInputLabels='replica^^az,replica') per each corresponding remoteWrite.url. Before, -remoteWrite.streamAggr.dropInputLabels labels were incorrectly applied to all configured remoteWrite.urls. See this issue for the details.

  • BUGFIX: stream aggregation: fix duplicated aggregation results if there are multiple concurrent writing samples with the same label. See this pull request.

  • BUGFIX: vmagent dashboard: fix legend captions for stream aggregation related panels. Before they were displaying wrong label names.

  • BUGFIX: vmgateway: add missing datadog, newrelic, opentelemetry and pushgateway routes to the JWT authorization routes. Allows prefixed (prometheus/graphite) routes for query requests.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: properly cache empty list of matching time series for the given labels filter. This type of caching was broken since v1.97.0, which could result in the increased CPU usage when performing queries, which match zero time series. See this issue.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: Fixes start-up crash on Windows OS. See this issue for details.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: fix metric vm_object_references{type="indexdb"}. Previously, it was overcounted.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: properly ingest stale NaN samples. Previously it could be dropped if series didn’t exist at storage node. See this issue for details.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: properly track vm_missing_tsids_for_metric_id_total metric. See this issue for details.

  • BUGFIX: vmalert: do not send notifications without labels to Alertmanager. Such notifications are rejected by Alertmanager anyway. Before, vmalert could send alert notifications even if no label-value pairs left after applying alert_relabel_configs from notifier config.

  • BUGFIX: vmalert: properly update value of variable $activeAt in rules annotation during replay mode. Before, $activeAt could have provided incorrect values during replay.

  • BUGFIX: MetricsQL: properly handle c1 AND c2 and c1 OR c1 queries for constants c1 and c2. Previously such queries could return unexpected results. See this issue.

  • BUGFIX: MetricsQL: consistently return the first non-NaN value from range_first function across all the returned data points. Previously NaN data points weren’t replaced with the first non-NaN value.

  • BUGFIX: MetricsQL: consistently return the last non-NaN value from range_last function across all the returned data points. Previously NaN data points weren’t replaced with the last non-NaN value.

  • BUGFIX: all VictoriaMetrics components: increase default value of -loggerMaxArgLen cmd-line flag from 1000 to 5000. This should improve visibility on errors produced by very long queries.

  • BUGFIX: all VictoriaMetrics components: trim trailing spaces when reading content from *.passwordFile and similar flags. It reverts changes introduced at v1.102.0-rc2 release. See this issue for details.

v1.103.0 #

Released at 2024-08-28

Update note: The external_labels field in vmalert-tool test file will be deprecated soon. Please use -external.label command-line flag instead, in the same way as vmalert uses it. This change is done for the sake of consistency between vmalert and vmalert-tool configuration. See this issue.

  • FEATURE: add /influx/health health-check handler for Influx endpoints. This is needed as some clients use the health endpoint to determine if the server is healthy and ready for data ingestion. See this issue for the details.

  • FEATURE: vmctl: add --vm-backoff-retries, --vm-backoff-factor, --vm-backoff-min-duration and --vm-native-backoff-retries, --vm-native-backoff-factor, --vm-native-backoff-min-duration command-line flags. These flags allow to change backoff policy config for import requests to VictoriaMetrics. See this issue.

  • FEATURE: vmagent: allow overriding the sample_limit option at scrape_configs when a label __sample_limit__ is specified for target. See this issue. Thanks to @zoglam for the pull request.

  • FEATURE: vmagent: reduce memory usage when scraping targets with big response body. See this issue.

  • FEATURE: vmagent and Single-node VictoriaMetrics: stop adding default port 80/443 for scrape URLs without a port. The value in instance label will still carry port for backward compatibility. See this issue.

  • FEATURE: vmagent: add flags -remoteWrite.retryMinInterval and -remoteWrite.retryMaxTime for adjusting remote-write requests retry policy. See this issue. Thanks to @yorik for the pull request.

  • FEATURE: vmalert: add command-line flag -notifier.headers to allow configuring additional headers for all requests sent to the corresponding -notifier.url.

  • FEATURE: vmalert-tool: add -external.label and -external.url command-line flags, in the same way as these flags are supported by vmalert. See this issue.

  • FEATURE: vmbackup, vmrestore, vmbackupmanager: use exponential backoff for retries when uploading or downloading data from S3. This should reduce the number of failed uploads and downloads when S3 is temporarily unavailable. See this issue.

  • FEATURE: stream aggregation: do not allow enabling -streamAggr.keepInput and keep_metric_names options together in stream aggregation config, as it may result in time series collision.

  • FEATURE: vmui: add search functionality to the column display settings in the table. See this issue.

  • FEATURE: vmui: add the ability to select all columns in the column display settings of the table. See this issue. Thanks to @yincongcyincong for the pull request.

  • FEATURE: vmselect in VictoriaMetrics cluster: add command-line flag -search.inmemoryBufSizeBytes for configuring size of in-memory buffers used by vmselect during processing of vmstorage responses. A new summary metric vm_tmp_blocks_inmemory_file_size_bytes is exposed to show the size of the buffer during requests processing. See this pull request for details. Thanks to @tydhot for implementation.

  • BUGFIX: vmagent: fixes proxy_url authorization for scrape targets. Previously proxy authorization configuration was ignored for https targets. See this issue for details.

  • BUGFIX: vmalert: respect HTTP headers defined in notifier configuration file for each request to notifiers. Previously, this param was ignored by mistake.

  • BUGFIX: stream aggregation: correctly apply -streamAggr.dropInputLabels when global stream deduplication is enabled without -streamAggr.config. Previously, -remoteWrite.streamAggr.dropInputLabels was used instead.

  • BUGFIX: stream aggregation: fix command-line flag -remoteWrite.streamAggr.ignoreFirstIntervals to accept multiple values and be applied per each corresponding -remoteWrite.url. Previously, this flag only could have been used globally for all URLs.

  • BUGFIX: graphite: respect -search.denyPartialResponse cmd-line flag and deny_partial_response query GET param when serving requests via Graphite API. Before, partial responses were always denied. Thanks to @penguinlav for the pull request.

  • BUGFIX: vmagent: account for -usePromCompatibleNaming cmd-line flag during when pushing data to remote storages. Thanks to @12345XXX for the pull request.

  • BUGFIX: vminsert in VictoriaMetrics cluster: reduce CPU usage by limiting the number of concurrently running inserts. The issue was introduced in this commit starting from v1.101.0. See this issue for details.

  • BUGFIX: MetricsQL: fix calculation histogram_quantile over Prometheus buckets with inconsistent values. It was producing incorrect results in case lower buckets. The issue was introduced in v1.102.0 release, see this issue for the details.

  • BUGFIX: vmalert, vmctl and snapshot API: verify correctness of URLs provided via cmd-line flags before executing HTTP requests. See this issue for details.

  • BUGFIX: vmalert: reduce memory usage when parsing responses with big number of metrics in response. The memory usage was increased in v1.102.0-rc1 after attempt to reduce CPU usage for heavy loaded vmalerts.

  • BUGFIX: all VictoriaMetrics components: forcefully set owner/group for release tars to 1000:1000. This helps to avoid unpacking issues on systems with limitations around UID:GID configuration. See this pull request.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: properly register index metrics for previous index part. See this issue for details.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: Removes the fallback to global index search when the search using per-day index fails due to too many time series found (the global index will fail anyway with the same error and so the fallback is not needed and only slows down the search). See this for details.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: fix metric names registering in the per-day index for new dates for existing time series when making calls to /tags/tagSeries and /tags/tagMultiSeries handlers of Graphite API. See this for details.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: properly ignore deleted metrics when applying retention filters and downsampling. See this issue for the details.

v1.102.9 #

Released at 2024-12-13

v1.102.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.102.x line will be supported for at least 12 months since v1.102.0 release

  • SECURITY: upgrade base docker image (Alpine) from 3.20.3 to 3.21.0. See alpine 3.21.0 release notes.

  • SECURITY: upgrade Go builder from Go1.23.3 to Go1.23.4. See the list of issues addressed in Go1.23.4.

  • BUGFIX: all VictoriaMetrics components: consistently deduplicate values with stale markers within deduplication interval. Previously, deduplication could randomly prefer stale marker or value on the deduplication interval. See this issue for details. Thanks to @tIGO for the pull request.

  • BUGFIX: vmsingle, vmselect in VictoriaMetrics cluster: properly parse the query rollup window specified in milliseconds. Previous implementation could lead to precision issues resulting in the parsed window being smaller by 1ms. See this issue for details.

  • BUGFIX: vmauth: fix requests routing by host when using src_hosts. Previously, request header could be ignored.

  • BUGFIX: vmbackupmanager: prevent backup scheduler from scheduling two backups immediately one after another.

  • BUGFIX: vmui: prevent accordion from collapsing when selecting text in headers. See this issue.

  • BUGFIX: vmui: fix query interval display in the instant query info. See this issue.

  • BUGFIX: vmagent and Single-node VictoriaMetrics: add missing common service labels for docker swarm service discovery when role is set to tasks. See this issue.

  • BUGFIX: vmalert: fix possible template collision between rule annotations.

  • BUGFIX: vmalert: fix externalURL function in -rule.templates after config reload.

  • BUGFIX: vmselect in VictoriaMetrics cluster: respect -search.skipSlowReplicas when -globalReplicationFactor > 1. See this issue.

v1.102.8 #

Released at 2024-11-29

v1.102.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.102.x line will be supported for at least 12 months since v1.102.0 release

  • BUGFIX: vmagent: properly parse multitenant token value for multitenant endpoints. Before, pushing data into vmagent using multitenant URL resulted in unsupported path. See this issue.
  • BUGFIX: vmagent: return 200 OK HTTP status code when importing data via Pushgateway protocol using multitenant URL format. See this issue and this pull request.
  • BUGFIX: vmagent: properly set TCP connection timeout for Kubernetes API server connection for metric scrapping with kubernetes_sd_configs. See this issue.
  • BUGFIX: vmagent: fix the resource_group filter for Azure service discovery on virtual machine scale sets. Previously, this filter did not apply to virtual machine scale sets, causing all virtual machines to be discovered. See this issue.
  • BUGFIX: vmauth: properly init ip_filters for unauthorized_user config section. Previously it was ignored and vmauth didn’t apply ip_filter for unauthorized access.
  • BUGFIX: vmsingle, vmselect in VictoriaMetrics cluster: properly return result for binary operation ^ aka pow at query requests for NaN values. See this issue for details.
  • BUGFIX: vmui: fix for showLegend and alias flags in predefined panels. See this issue

v1.102.7 #

Released at 2024-11-15

v1.102.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.102.x line will be supported for at least 12 months since v1.102.0 release

  • SECURITY: upgrade Go builder from Go1.23.1 to Go1.23.3. See the list of issues addressed in Go1.23.2 and Go1.23.3.

  • BUGFIX: vmctl: drop rows that do not belong to the current series during import. The dropped rows should belong to another series whose tags are a superset of the current series. See this issue and this pull request. Thanks to @dpedu for reporting and cooperating with the test.

  • BUGFIX: vmsingle, vmselect in VictoriaMetrics cluster: keep the order of resulting time series when limit_offset is applied. See this issue.

  • BUGFIX: graphite: properly handle xFilesFactor=0 for transformRemoveEmptySeries function. See this PR for details.

  • BUGFIX: vmauth: properly check availability of all the backends before giving up when proxying requests. Previously, vmauth could return an error even if there were healthy backends available. See this issue for details.

  • BUGFIX: vmauth: fixed unauthorized routing behavior inconsistency. See this issue for details.

  • BUGFIX: vmauth: properly inherit drop_src_path_prefix_parts, load_balancing_policy, retry_status_codes and discover_backend_ips options by url_map entries if url_prefix option isn’t set at the user config level. These options were inherited only when the url_prefix option was set. See this issue.

  • BUGFIX: dashboards: add file label filter to vmalert dashboard panels. Previously, metrics from groups with the same name but different rule files could be mixed in the results.

v1.102.6 #

Released at 2024-11-04

v1.102.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.102.x line will be supported for at least 12 months since v1.102.0 release

  • BUGFIX: vmalert: properly set group_name and file fields for recording rules in /api/v1/rules.
  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: prevent panic when ingesting samples which are outside of configured retention filters. This could happen when backfilling data with retention filters which exclude samples from the backfill range.
  • BUGFIX: vmctl: fix issue with series matching for vmctl vm-native with --vm-native-disable-per-metric-migration flag enabled. See this issue.

v1.102.5 #

Released at 2024-10-21

v1.102.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.102.x line will be supported for at least 12 months since v1.102.0 release

  • BUGFIX: vmui: fix error messages rendering from overflowing the screen with long messages. See this issue.
  • BUGFIX: vmctl: properly add metrics tags for opentsdb migration source. Previously it could have empty values. See this PR.
  • BUGFIX: vmagent: support m unit for minutes duration in command-line flag -streamAggr.dedupInterval. Previously unit m wasn’t supported correctly.
  • BUGFIX: VictoriaMetrics cluster: properly apply replication factor when storage node groups are used and replication factor is configured via global value such as -replicationFactor=2. Previously, global replication factor was ignored for storage node groups. See these docs for more information about storage groups configuration.
  • BUGFIX: vmalert: properly apply configuration changes during hot-reload to rule groups that haven’t started yet. Previously, configuration updates to such groups could have resulted into blocking all evaluations within the group, until vmalert restart.
  • BUGFIX: Single-node VictoriaMetrics and vmagent: properly update max_scrape_size param change during hot-reload. See this issue.

v1.102.4 #

Released at 2024-10-02

v1.102.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.102.x line will be supported for at least 12 months since v1.102.0 release

  • BUGFIX: stream aggregation: fix -remoteWrite.streamAggr.dropInputLabels labels parsing. Now, this flag allows specifying a list of labels to drop (by using ‘^^’ separator, i.e. dropInputLabels='replica^^az,replica') per each corresponding remoteWrite.url. Before, -remoteWrite.streamAggr.dropInputLabels labels were incorrectly applied to all configured remoteWrite.urls. See this issue for the details.
  • BUGFIX: stream aggregation: fix duplicated aggregation results if there are multiple concurrent writing samples with the same label. See this pull request.
  • BUGFIX: MetricsQL: consistently return the first non-NaN value from range_first function across all the returned data points. Previously NaN data points weren’t replaced with the first non-NaN value.

v1.102.3 #

Released at 2024-09-23

  • SECURITY: upgrade Go builder from Go1.22.5 to Go1.23.1. See the list of issues addressed in Go1.23.1.

  • SECURITY: upgrade base docker image (Alpine) from 3.20.2 to 3.20.3. See alpine 3.20.3 release notes.

  • BUGFIX: vmagent fix service discovery of Azure Virtual Machines for response contains nextLink in Host:Port format. See this issue.

  • BUGFIX: vmagent: properly consume messages from kafka. Previously vmagent could skip some messages during start-up.

  • BUGFIX: stream aggregation: perform deduplication for all received data when specifying -streamAggr.dedupInterval or -remoteWrite.streamAggr.dedupInterval command-line flags are set. Previously, if the -remoteWrite.streamAggr.config or -streamAggr.config is set, only series that matched aggregation config were deduplicated. See this issue for details.

  • BUGFIX: vmagent dashboard: fix legend captions for stream aggregation related panels. Before they were displaying wrong label names.

  • BUGFIX: vmgateway: add missing datadog, newrelic, opentelemetry and pushgateway routes to the JWT authorization routes. Allows prefixed (prometheus/graphite) routes for query requests.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: properly cache empty list of matching time series for the given labels filter. This type of caching was broken since v1.97.0, which could result in the increased CPU usage when performing queries, which match zero time series. See this issue.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: Fixes start-up crash on Windows OS. See this issue for details.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: fix metric vm_object_references{type="indexdb"}. Previously, it was overcounted.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: properly ingest stale NaN samples. Previously it could be dropped if series didn’t exist at storage node. See this issue [https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5069] for details.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: properly track vm_missing_tsids_for_metric_id_total metric. See this issue for details.

  • BUGFIX: vmalert: do not send notifications without labels to Alertmanager. Such notifications are rejected by Alertmanager anyway. Before, vmalert could send alert notifications even if no label-value pairs left after applying alert_relabel_configs from notifier config.

  • BUGFIX: vmalert: properly update value of variable $activeAt in rules annotation during replay mode. Before, $activeAt could have provided incorrect values during replay.

  • BUGFIX: MetricsQL: properly handle c1 AND c2 and c1 OR c1 queries for constants c1 and c2. Previously such queries could return unexpected results. See this issue.

  • BUGFIX: all VictoriaMetrics components: increase default value of -loggerMaxArgLen cmd-line flag from 1000 to 5000. This should improve visibility on errors produced by very long queries.

v1.102.2 #

Released at 2024-08-28

v1.102.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.102.x line will be supported for at least 12 months since v1.102.0 release

  • BUGFIX: vmagent: fixes proxy_url authorization for scrape targets. Previously proxy authorization configuration was ignored for https targets. See this issue for details.
  • BUGFIX: vmagent fix service discovery of Azure Virtual Machines for response contains nextLink. See this issue.
  • BUGFIX: vmalert: respect HTTP headers defined in notifier configuration file for each request to notifiers. Previously, this param was ignored by mistake.
  • BUGFIX: stream aggregation: correctly apply -streamAggr.dropInputLabels when global stream deduplication is enabled without -streamAggr.config. Previously, -remoteWrite.streamAggr.dropInputLabels was used instead.
  • BUGFIX: stream aggregation: fix command-line flag -remoteWrite.streamAggr.ignoreFirstIntervals to accept multiple values and be applied per each corresponding -remoteWrite.url. Previously, this flag only could have been used globally for all URLs.
  • BUGFIX: graphite: respect -search.denyPartialResponse cmd-line flag and deny_partial_response query GET param when serving requests via Graphite API. Before, partial responses were always denied. Thanks to @penguinlav for the pull request.
  • BUGFIX: vmagent: account for -usePromCompatibleNaming cmd-line flag during when pushing data to remote storages. Thanks to @12345XXX for the pull request.
  • BUGFIX: vminsert in VictoriaMetrics cluster: reduce CPU usage by limiting the number of concurrently running inserts. The issue was introduced in this commit starting from v1.101.0. See this issue for details.
  • BUGFIX: MetricsQL: fix calculation histogram_quantile over Prometheus buckets with inconsistent values. It was producing incorrect results in case lower buckets. The issue was introduced in v1.102.0 release, see this issue for the details.
  • BUGFIX: vmalert, vmctl and snapshot API: verify correctness of URLs provided via cmd-line flags before executing HTTP requests. See this issue for details.
  • BUGFIX: vmalert: reduce memory usage when parsing responses with big number of metrics in response. The memory usage was increased in v1.102.0-rc1 after attempt to reduce CPU usage for heavy loaded vmalerts.
  • BUGFIX: all VictoriaMetrics components: forcefully set owner/group for release tars to 1000:1000. This helps to avoid unpacking issues on systems with limitations around UID:GID configuration. See this pull request.
  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: properly register index metrics for previous index part. See this issue for details.
  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: Removes the fallback to global index search when the search using per-day index fails due to too many time series found (the global index will fail anyway with the same error and so the fallback is not needed and only slows down the search). See this for details.
  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: fix metric names registering in the per-day index for new dates for existing time series when making calls to /tags/tagSeries and /tags/tagMultiSeries handlers of Graphite API. See this for details.
  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: properly ignore deleted metrics when applying retention filters and downsampling. See this issue for the details.

v1.102.1 #

Released at 2024-08-01

v1.102.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.102.x line will be supported for at least 12 months since v1.102.0 release

Update note 1: vmauth HTTP response code has changed from 503 to 502 for a case when all upstream backends were not available. This was changed to align vmauth behaviour with other well-known reverse-proxies behaviour.

  • SECURITY: upgrade base docker image (Alpine) from 3.20.1 to 3.20.2. See alpine 3.20.2 release notes.

  • FEATURE: vmauth: add keep_original_host option, which can be used for proxying the original Host header from client request to the backend. By default, the backend host is used as Host header when proxying requests to the configured backends. See these docs.

  • FEATURE: vmauth: now returns HTTP 502 status code when all upstream backends are not available. Previously, it returned HTTP 503 status code. This change aligns vmauth behavior with other well-known reverse-proxies behavior.

  • FEATURE: vmagent dashboard: add Scrape duration 0.99 quantile panel to show the 99th quantile of scrape duration in seconds. This should help identifying vmagent instances that experiences too high scraping durations.

  • BUGFIX: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: Fixes panic if incorrect metricsql expression passed to the prettify-query API. See this issue for details.

  • BUGFIX: all VictoriaMetrics components: validate files specified via -tlsKeyFile and -tlsCertFile cmd-line flags on the process start-up. Previously, validation happened on the first connection accepted by HTTP server. See this issue for the details. Thanks to @yincongcyincong for the pull request.

  • BUGFIX: vmauth: properly proxy requests to backend urls ending with / if the original request path equals to /. Previously the trailing / at the backend path was incorrectly removed. For example, if the request to http://vmauth/ is configured to be proxied to url_prefix=http://backend/foo/, then it was proxied to http://backend/foo, while it should go to http://backend/foo/.

  • BUGFIX: vmauth: fix cannot read data after closing the reader error when proxying HTTP requests without body (aka GET requests). The issue has been introduced in v1.102.0 in this commit.

  • BUGFIX: vmui: fix auto-completion triggers for metric and label names, and disable it after specific characters. See this issue and these comments.

  • BUGFIX: vmui: fix automatic addition of quotes when inserting label values from autocomplete suggestions. See this issue.

v1.102.0 #

Released at 2024-07-17

v1.102.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.102.x line will be supported for at least 12 months since v1.102.0 release

Update note 1: support for snap packages was removed due to lack of interest from community. See this pull request for details. Please read about supported package types here.

Update note 2: stream aggregation config now prevents setting multiple identical outputs. For example, outputs: [total, total] will fail the validation phase. In addition, outputs: ["quantiles(0.5)", "quantiles(0.9)"] will fail the validation as well - use outputs: ["quantiles(0.5, 0.9)"] instead.

v1.102.0-rc2 #

Released at 2024-06-24

Update note 1: the --vm-disable-progress-bar command-line flag at vmctl was deprecated. Use --disable-progress-bar instead.

Update note 2: *.passwordFile and similar flags are no longer trimming trailing whitespaces at the end of content. Make sure to update the templating of password files or HTTP endpoints to not include trailing whitespaces before the upgrade. See this PR for the details.

  • FEATURE: alerts-vmagent: add new alerting rules StreamAggrFlushTimeout and StreamAggrDedupFlushTimeout to notify about issues during stream aggregation.

  • FEATURE: dashboards/vmagent: add row Streaming aggregation with panels related to streaming aggregation process.

  • FEATURE: vmauth: add -idleConnTimeout flag set to 50s by default. It should reduce the probability of broken pipe or connection reset by peer errors in vmauth logs.

  • FEATURE: vmauth: add automatic retry for requests to backend for trivial network errors, such as broken pipe and connection reset for requests to the configured backends.

  • FEATURE: vmagent: increase default value of -promscrape.maxDroppedTargets command-line flag to 10_000 from 1000. This makes it easier to track down large number of dropped targets. See this issue.

  • FEATURE: vmagent: add max_scrape_size option to scrape config for setting custom limit on the response size target can send. The new automatically generated metric scrape_response_size_bytes is added to reflect the response size of the target. See these issues: 1, 2, 3, 4.

  • FEATURE: vmsingle: check for ranged vector arguments in non-rollup expressions when -search.disableImplicitConversion or -search.logImplicitConversion are enabled. For example, sum(up[5m]) or absent(up[5m]) will fail to execute if these flags are set.

  • FEATURE: vmsingle: validate that rollup expressions has ranged vector arguments passed when -search.disableImplicitConversion or -search.logImplicitConversion are enabled. For example, rate(metric) or count_over_time(metric) will fail to execute if these flags are set.

  • FEATURE: vmalert-tool: support file path with hierarchical patterns and regexes, and http url in unittest cmd-line flag -files, e.g. -files="http://<some-server-addr>/path/to/rules" or -files="dir/**/*.yaml".

  • BUGFIX: all VictoriaMetrics components: properly calculate process_resident_memory_bytes metric for OS with non-default value of memory page size. See this issue for details.

  • BUGFIX: all VictoriaMetrics components: prioritize -configAuthKey and -reloadAuthKey over -httpAuth.* settings. This change aligns behavior of mentioned flags with other auth flags like -metricsAuthKey, -flagsAuthKey, -pprofAuthKey. Check this issue.

  • BUGFIX: all VictoriaMetrics components: do not trim trailing spaces when reading content from *.passwordFile and similar flags. Previously, trailing spaces were trimmed from the content of the password file, which could lead to unexpected authentication errors.

  • BUGFIX: vmctl: add --disable-progress-bar global command-line flag. It can be used for disabling dynamic progress bar for all migration modes. --vm-disable-progress-bar command-line flag is deprecated and will be removed in the future releases. See this issue.

  • BUGFIX: stream aggregation: prevent rate_sum and rate_avg producing NaN results for stale time series. Before, when series matched for aggregation became stale or weren’t updated during aggregation interval, the rate_sum or rate_avg could produce data point with NaN value. During visualization, such aggregation results would be displayed as gaps in time series.

  • BUGFIX: vmalert: fix path for system links printed on default vmalert’s UI page when -http.pathPrefix is set. See this issue.

  • BUGFIX: vmalert: exit replay mode with non-zero code if generated samples are not successfully written into remoteWrite url. See this issue.

  • BUGFIX: vmbackup: properly configure authentication with S3 when -configFilePath cmd-line flag is specified. See this issue.

  • BUGFIX: vmalert enterprise: properly configure authentication with S3 when -s3.configFilePath cmd-line flag is specified for reading rule configs.

  • BUGFIX: vmalert: properly specify oauth2 ClientSecret when configuring authentication for notifier.url. See this issue for details. Thanks to @yincongcyincong for the pull request.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: add validation for the max value specified for -retentionPeriod. See this issue for details.

  • BUGFIX: vmui: copy row button in Table view produces unexpected result. See this issue and pull request.

  • BUGFIX: vmalert-tool: prevent hanging when processing groups without rules. See this issue.

  • BUGFIX: vmauth: fix discovering backend IPs when url_prefix contains hostname with srv+ prefix. Thanks to @shichanglin5 for the pull request.

v1.102.0-rc1 #

Released at 2024-06-07

Update note 1: the -remoteWrite.multitenantURL command-line flag at vmagent was removed starting from this release. This flag was deprecated since v1.96.0. Use -enableMultitenantHandlers instead, as it is easier to use and combine with multitenant URL at vminsert. See these docs for details.

Update note 2: the -streamAggr.dropInputLabels command-line flag at vmagent was renamed to -remoteWrite.streamAggr.dropInputLabels. -streamAggr.dropInputLabels is now used for global streaming aggregation.

Update note 3: the -maxLabelValueLen command-line flag default value was changed from 16KiB to 4KiB. It may lead to truncating of labels with too long values.

  • SECURITY: upgrade Go builder from Go1.22.2 to Go1.22.4. See the list of issues addressed in Go1.22.3 and Go1.22.4.

  • SECURITY: upgrade base docker image (Alpine) from 3.19.1 to 3.20.0. See alpine 3.20.0 release notes.

  • SECURITY: add release images built from scratch image. Such images could be more preferable for using in environments with higher security standards. See this pull request.

  • FEATURE: dashboards/single: support selecting of multiple instances on the dashboard. See this issue for details.

  • FEATURE: dashboards/single: properly display version in the Stats row for the custom builds of VictoriaMetrics.

  • FEATURE: dashboards/single: add Network Usage panel to Resource Usage row.

  • FEATURE: dashboards/operator, dashboards/backupmanager and dashboard/tenant-statistic: update dashboard to be compatible with Grafana 10+ version.

  • FEATURE: dashboards/cluster: add new panel Concurrent selects to vmstorage row. The panel will show how many ongoing select queries are processed by vmstorage and should help to identify resource bottlenecks. See panel description for more details.

  • FEATURE: dashboards: use $__interval variable for offsets and look-behind windows in annotations. This should improve precision of restarts and version change annotations when zooming-in/zooming-out on the dashboards.

  • FEATURE: vmagent: support aggregation and deduplication configs before replicating data to configured -remoteWrite.url destinations. This saves CPU and memory resources when incoming data needs to be aggregated or deduplicated once and then replicated to multiple destinations. See this issue.

  • FEATURE: vmagent: add service discovery support for Vultr. See these docs and this issue.

  • FEATURE: vmagent: allow specifying -remoteWrite.disableOnDiskQueue command-line flag per each -remoteWrite.url. If multiple -remoteWrite.disableOnDiskQueue command-line flags are configured, then the -remoteWrite.dropSamplesOnOverload is automatically set to true, so samples are automatically dropped if they cannot be sent to the corresponding -remoteWrite.url in a timely manner. See this pull request. Thanks to @rbizos for implementation!

  • FEATURE: vmagent: add path and url labels to vmagent_remotewrite_push_failures_total and vmagent_remotewrite_samples_dropped_total metrics, so the number of failed pushes and dropped samples can be tracked per each -remoteWrite.url.

  • FEATURE: stream aggregation: add rate_sum and rate_avg aggregation outputs.

  • FEATURE: stream aggregation: reduce the number of allocated objects in heap during deduplication and aggregation. The change supposed to reduce pressure on Garbage Collector, as it will need to scan less objects. See this pull request.

  • FEATURE: vmalert: add -datasource.idleConnTimeout, -remoteWrite.idleConnTimeout and -remoteRead.idleConnTimeout flags. These flags are set to 50s by default and should reduce the probability of broken pipe or connection reset by peer errors in vmalert logs. See this issue for details.

  • FEATURE: vmalert: add auto request retry for trivial network errors, such as broken pipe and connection reset for requests to remoteRead, remoteWrite and datasource URLs. See this issue for details.

  • FEATURE: vmalert: reduce CPU usage when evaluating high number of alerting and recording rules.

  • FEATURE: vmalert: speed up retrieving rules files from object storages by skipping unchanged objects during reloading. See this issue.

  • FEATURE: vmalert: support reading DNS SRV records in -datasource.url, -remoteWrite.url and -remoteRead.url command-line option. For example, -remoteWrite.url=http://srv+victoria-metrics automatically resolves the victoria-metrics DNS SRV to a list of hostnames with TCP ports and then sends data to one of the addresses. See this feature request.

  • FEATURE: vmbackup, vmrestore, vmbackupmanager: add -s3TLSInsecureSkipVerify command-line flag for skipping TLS certificates verification when connecting to S3 endpoint.

  • FEATURE: expose metric vm_indexdb_items_dropped_total to track the number of IndexDB records that had to be dropped during ingestion. The reason of dropping the record will be annotated in reason label of the exposed metric. This change also comes with a new alerting rule to track changes of this metric.

  • FEATURE: alerts-health: add new alerting rules TooLongLabelValues and TooLongLabelNames to notify about truncation of label values or names respectively.

  • FEATURE: stream aggregation: expose vm_streamaggr_ignored_samples_total counters at /metrics page, which can be used for detecting amount of too old or NaN valued ignored samples. Expose also vm_streamaggr_samples_lag_seconds histogram to monitor aggregated samples lag.

  • FEATURE: stream aggregation: improve filtering speed of the received data samples if match field is matching only metric name.

  • FEATURE: VictoriaMetrics cluster: adds -relabelConfigCheckInterval flag and /-/reload endpoint for reloading relabeling rules. See this pull request for details.

  • BUGFIX: vmui: fix bug that prevents the first query trace from expanding on click event. See this issue. The issue was introduced in v1.100.0 release.

  • BUGFIX: vmui: fix calendar display when UTC+00:00 timezone is set. See this issue.

  • BUGFIX: vmui: remove redundant requests on the Explore Cardinality page. See this issue.

  • BUGFIX: vmui: fix handling of URL params for browser history navigation (back and forward buttons). See this issue and this comment.

  • BUGFIX: vmagent: prevent potential panic during stream aggregation when more than one -remoteWrite.url command-line flags are passed to vmagent together with non-zero -remoteWrite.streamAggr.dedupInterval command-line flag. See this issue.

  • BUGFIX: vmagent: skip empty data blocks before sending to the remote write destination. Thanks to @viperstars for the pull request.

  • BUGFIX: stream aggregation: set correct suffix <output>_prometheus for aggregation outputs increase_prometheus and total_prometheus. Before, outputs total and total_prometheus or increase and increase_prometheus had the same suffix.

  • BUGFIX: stream aggregation: prevent from excessive resource usage when stream aggregation config file is empty.

  • BUGFIX: properly estimate the needed memory for query execution if it has the format aggr_func(rollup_func[d] (for example, sum(rate(request_duration_seconds_bucket[5m]))). This should allow performing aggregations over bigger number of time series when VictoriaMetrics runs in environments with small amounts of available memory. The issue has been introduced in this commit in v1.83.0.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: correctly apply -inmemoryDataFlushInterval when it’s set to minimum supported value 1s.

  • BUGFIX: Single-node VictoriaMetrics and vminsert in VictoriaMetrics cluster: reduce the default value for -maxLabelValueLen command-line flag from 16KiB to 1KiB. This should prevent from issues like this one when time series with too long labels are ingested into VictoriaMetrics.

  • BUGFIX: vmauth: properly release memory used for metrics during config reload. See this issue.

  • BUGFIX: dashboards: fix AnnotationQueryRunner error in Grafana when executing annotations query against Prometheus backend. See this issue for details.

  • BUGFIX: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: filter deleted label names and values from /api/v1/labels and /api/v1/label/.../values responses when match[] filter matches small number of time series. The issue was introduced v1.81.0.

  • BUGFIX: vmalert-tool: fix float values template in input_series. See this issue.

  • BUGFIX: vmalert: fix misleading error logs in vmalert’s stdout when unsupported HTTP path is requested.

  • BUGFIX: retry files delete attempts on vXFS file system for EEXIST error type. See the related issue. Thanks to @pludov for the pull request.

  • DEPRECATION: vmagent: removed deprecated -remoteWrite.multitenantURL flag from vmagent. This flag was deprecated since v1.96.0. Use -enableMultitenantHandlers instead, as it is easier to use and combine with multitenant URL at vminsert. See these docs for details.

v1.101.0 #

Released at 2024-04-26

  • FEATURE: MetricsQL: support filtering by multiple numeric constants via q == (C1, ..., CN) and q != (C1, ..., CN) syntax. For example, status_code == (200, 201, 300) returns status_code metrics with any of 200, 201 or 300 values, while status_code != (400, 404, 500) returns status_code metrics with all the values except of 400, 404 and 500.

  • FEATURE: VictoriaMetrics cluster: add support for fault domain awareness to vmselect. It can be configured to return full responses if up to -globalReplicationFactor - 1 fault domains (aka vmstorage groups) are unavailable. See this feature request and these docs.

  • FEATURE: all VictoriaMetrics enterprise components: add support for automatic issuing of TLS certificates for HTTPS server at -httpListenAddr via Let’s Encrypt service. See these docs and this feature request.

  • FEATURE: vmsingle: support disable or log implicit conversions for subquery with cmd-line flags -search.disableImplicitConversion and -search.logImplicitConversion. See this feature request.

  • FEATURE: vmagent: support data replication additionally to sharding among remote storage systems if -remoteWrite.shardByURLReplicas=N command-line flag is set additionally to -remoteWrite.shardByURL command-line flag, where N is desired replication factor. This allows setting up data replication among failure domains when the replication factor is smaller than the number of failure domains. See these docs and this feature request.

  • FEATURE: vmagent: reduce CPU usage when sharding among remote storage systems is enabled.

  • FEATURE: stream aggregation: reduce memory usage during stream aggregation if multiple aggregation configs are used for the same set of data.

  • FEATURE: vmagent: support DNS SRV addresses in -remoteWrite.url command-line option and in scrape target urls. For example, -remoteWrite.url=http://srv+victoria-metrics/api/v1/write automatically resolves the victoria-metrics DNS SRV to a list of hostnames with TCP ports and then sends the collected metrics to these TCP addresses. See these docs and this feature request.

  • FEATURE: stream aggregation: allow skipping first N aggregation intervals via cmd-line flag -streamAggr.ignoreFirstIntervals for Single-node VictoriaMetrics or -remoteWrite.streamAggr.ignoreFirstIntervals for vmagent. See more details here.

  • FEATURE: vmauth: support automatic discovering and load balancing for TCP addresses behind DNS SRV addresses. These addresses can be put inside url_prefix urls in the form http://srv+addr/path, where the addr is the DNS SRV address, which is automatically resolved to hostnames with TCP ports. See these docs for details.

  • FEATURE: vmauth: support specifying client TLS certificates and TLS ServerName for requests to HTTPS backends. See these docs.

  • FEATURE: vmauth: support regex matching when routing incoming requests based on HTTP query args via src_query_args option at url_map. See these docs and this feature request.

  • FEATURE: vmui: optimize auto-suggestion performance for metric names when the database contains big number of unique time series.

  • FEATURE: vmui: in the Select component, user-entered values are now preserved on blur if they match options in the list.

  • FEATURE: vmui: auto-suggestion triggers at any cursor position in the query input. See this issue.

  • FEATURE: vmui: update error messages on the Query page for enhanced clarity. See this pull request.

  • BUGFIX: downsampling: skip unnecessary index lookups if downsampling wasn’t set for ENT versions of VictoriaMetrics. Before, users of VictoriaMetrics ENT could have experience elevated CPU usage even if no downsampling was configured. The issue was introduced in v1.100.0.

  • BUGFIX: downsampling: properly populate downsampling metadata for data parts created by VictoriaMetrics ENT versions lower than v1.100.0. The bug could trigger the downsampling actions for parts that were downsampled already. This bug doesn’t have any negative effect apart from spending extra CPU resources on the repeated downsampling. The issue was introduced in v1.100.0.

  • BUGFIX: vmalert: supported any status codes from the range 200-299 from alertmanager. Previously, only 200 status code considered a successful action. See this issue.

  • BUGFIX: vmalert: avoid blocking /api/v1/rules, /api/v1/alerts, /metrics APIs when alerting rule uses template functions query, which could takes a while to execute. See this issue.

  • BUGFIX: vmalert: fix links with anchors in vmalert’s UI. Starting from v1.99.0 vmalert could ignore anchors pointing to specific rule groups if search param was present in URL.

  • BUGFIX: vmauth: don’t treat concurrency limit hit as an error of the backend. Previously, hitting the concurrency limit would increment both vmauth_concurrent_requests_limit_reached_total and vmauth_user_request_backend_errors_total counters. Now, only concurrency limit counter is incremented. Updates this issue.

v1.100.1 #

Released at 2024-04-11

Update note 1: This release contains the issue which could lead to extra CPU usage of storage component in ENT distribution of VictoriaMetrics. The issue is caused by downsampling per distinct sets of time series feature introduced in 1.100.0 ENT version. Please, upgrade to v1.101.0 or rollback to v1.98.0 ENT version if you’re affected.

Update note 2: When upgrading to this release from v1.99.0 or v1.100.0 it is recommended to reset caches stored on disk according to these docs.

  • FEATURE: vmbackupmanager: allow specifying custom backup interval via -backupInterval command-line flag. See this feature request.

  • BUGFIX: properly register new entries in IndexDB when many new time series are ingested into VictoriaMetrics in a short period of time. See this and this issues. The bug has been introduced in v1.99.0.

  • BUGFIX: vmbackupmanager: fix panic when performing a restore from GCS. Previously, filters for GCS objects were not properly set which caused objects to be returned instead of prefixes.

v1.100.0 #

Released at 2024-04-04

This release contains the issue, which can prevent from storing data for new time series under high rate of search queries. Please rollback to v1.98.0 or upgrade to v1.101.0.

Update note 1: the -datasource.lookback command-line flag at vmalert is no-op starting from this release. This flag will be removed in the future, so please switch to eval_delay option. See this issue for more details.

Update note 2: -streamAggr.dedupInterval and -remoteWrite.streamAggr.dedupInterval cmd-line flags are now used for all incoming samples if -streamAggr.config and -remoteWrite.streamAggr.config aren’t specified. Before, deduplication wasn’t applied if aggregation config was omitted.

v1.99.0 #

Released at 2024-03-01

This release contains the issue, which can prevent from storing data for new time series under high rate of search queries. Please rollback to v1.98.0 or upgrade to v1.101.0.

v1.98.0 #

Released at 2024-02-14

  • SECURITY: upgrade Go builder from Go1.21.6 to Go1.22.0. See the list of issues addressed in Go1.21.7, plus the changelog for Go1.22.0.

  • FEATURE: all VictoriaMetrics components: add support for TLS client certificate verification at -httpListenAddr (aka mTLS). See these docs and this feature request.

  • FEATURE: all VictoriaMetrics components: add support for accepting http requests over multiple distinct TCP addresses. This can be done by starting VictoriaMetrics component with multiple -httpListenAddr command-line flags. For example, ./victoria-metrics -httpListenAddr=some-host:12345 -httpListenAddr=localhost:8428 starts VictoriaMetrics, which accepts incoming http requests at both some-host:12345 and localhost:8428. See this feature request.

  • FEATURE: all VictoriaMetrics components: add support for empty command-line flag values in short array notation. For example, -remoteWrite.sendTimeout=',20s,' specifies three -remoteWrite.sendTimeout values - the first one and the last one have default values (30s in this case), while the second one is set to 20s.

  • FEATURE: vmauth: add support for mTLS-based request routing to different backends depending on the subject of the TLS certificate provided by the client. See these docs and this feature request.

  • FEATURE: vmagent and single-node VictoriaMetrics: add support for data ingestion via DataDog lambda extension aka /api/beta/sketches endpoint. See these docs and this feature request. Thanks to @AndrewChubatiuk for the pull request.

  • FEATURE: vmagent: add -remoteWrite.tlsHandshakeTimeout command-line flag for tuning the timeout needed for establishing TLS connections to -remoteWrite.url. Bigger tls handshake timeouts should reduce the probability of http: TLS handshake error from ...: EOF errors at the remote storage side under high load. See this issue.

  • FEATURE: VictoriaMetrics cluster: add -disableReroutingOnUnavailable command-line flag to vminsert, which can be used for reducing resource usage spikes at vmstorage nodes during rolling restart. See these docs. Thanks to @Muxa1L for the pull request.

  • FEATURE: add -search.resetRollupResultCacheOnStartup command-line flag for resetting query cache on startup. See this feature request.

  • FEATURE: MetricsQL: propagate label filters across label_set and alias functions. For example, label_set(q1, "a", "b") + q2{c="d"} is automatically transformed to label_set(q1{c="d"}, "a", "b") + q2{a="b",c="d"} now. This should improve performance for such queries. See this issue.

  • FEATURE: MetricsQL: add sum_eq_over_time, sum_gt_over_time and sum_le_over_time functions. See this feature request.

  • FEATURE: dashboards/vmagent: add Targets scraped/s stat panel showing the number of targets scraped by the vmagent per-second.

  • FEATURE: dashboards/all: add new panel CPU spent on GC. It should help identifying cases when too much CPU is spent on garbage collection, and advice users on how this can be addressed.

  • FEATURE: vmalert: support filtering for /api/v1/rules API. See the pull request by @victoramsantos.

  • FEATURE: vmbackup: support client-side TLS configuration for creating and deleting snapshots via -snapshot.tls* cmd-line flags. See this feature request. Thanks to @khushijain21 for the pull request.

  • FEATURE: vmui: add a time picker to the “Logs Explorer” page. See this issue.

  • BUGFIX: all VictoriaMetrics components: do not close connections to -httpListenAddr every 2 minutes. This behavior didn’t help spreading load among multiple backend servers behind load-balancing TCP proxy. Instead, it could lead to hard-to-debug issues like this one. If you still need periodically closing client connections because of some reason, then pass the desired timeout to -http.connTimeout command-line flag.

  • BUGFIX: vmagent: reduce CPU usage when -promscrape.dropOriginalLabels command-line flag is set. This issue has been introduced in v1.96.0 when addressing this feature request.

  • BUGFIX: vmauth: properly release memory during config reload. See this issue.

  • BUGFIX: vmauth: properly expose vmauth_unauthorized_user_concurrent_requests_capacity, vmauth_unauthorized_user_concurrent_requests_current, vmauth_user_concurrent_requests_capacity and vmauth_user_concurrent_requests_current metrics after config reload. Previously these metrics didn’t work after config reload.

  • BUGFIX: MetricsQL: properly propagate label filters from multiple arguments passed to aggregate functions. For example, sum({job="foo"}, {job="bar"}) by (job) + a was improperly optimized to sum({job="foo"}, {job="bar"}) by (job) + a{job="foo"} before being executed. This could lead to unexpected results. See this issue.

  • BUGFIX: MetricsQL: properly handle precision errors when calculating changes, changes_prometheus, increases_over_time and resets functions. See this issue.

  • BUGFIX: all VictoriaMetrics components: consistently return 200 http status code from /-/reload endpoint. Previously single-node VictoriaMetrics was returning 204 http status code. See this feature request.

  • BUGFIX: properly store staleness markers for self-scraped metrics on single-node VictoriaMetrics shutdown. See this issue.

  • BUGFIX: prevent from possible too big indexBlockSize panic when samples with too long label values (~64Kb) are ingested into VictoriaMetrics.

  • BUGFIX: vmui: fix the graph dragging for Firefox and Safari. See this issue.

  • BUGFIX: vmui: fix handling invalid timezone. See this issue.

  • BUGFIX: vmui: fix the bug where the select does not open. See this issue.

  • BUGFIX: vmui: clear entered text in select after selecting a value. See this issue.

  • BUGFIX: vmui: improve the operation of the context for autocomplete. See this, this and this issues.

  • BUGFIX: dashboards: update Storage full ETA panels for Single-node and Cluster dashboards to prevent them from showing negative or blank results caused by increase of deduplicated samples. Deduplicated samples were part of the expression to provide a better estimate for disk usage, but due to sporadic nature of deduplication in VictoriaMetrics it rather produced skewed results. See this pull request.

  • BUGFIX: vmalert: reduce memory usage for ENT version of vmalert for configurations with high number of groups with enabled multitenancy.

v1.97.14 #

Released at 2024-12-13

v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release

  • SECURITY: upgrade base docker image (Alpine) from 3.20.3 to 3.21.0. See alpine 3.21.0 release notes.

  • SECURITY: upgrade Go builder from Go1.23.3 to Go1.23.4. See the list of issues addressed in Go1.23.4.

  • BUGFIX: all VictoriaMetrics components: consistently deduplicate values with stale markers within deduplication interval. Previously, deduplication could randomly prefer stale marker or value on the deduplication interval. See this issue for details. Thanks to @tIGO for the pull request.

  • BUGFIX: vmsingle, vmselect in VictoriaMetrics cluster: properly parse the query rolloup window specified in milliseconds. Previous implementation could lead to precision issues resulting in the parsed window being smaller by 1ms. See this issue for details.

  • BUGFIX: vmauth: fix requests routing by host when using src_hosts. Previously, request header could be ignored.

  • BUGFIX: vmbackupmanager: prevent backup scheduler from scheduling two backups immediately one after another.

  • BUGFIX: vmui: prevent accordion from collapsing when selecting text in headers. See this issue.

  • BUGFIX: vmui: fix query interval display in the instant query info. See this issue.

  • BUGFIX: vmagent and Single-node VictoriaMetrics: add missing common service labels for docker swarm service discovery when role is set to tasks. See this issue.

  • BUGFIX: vmalert: fix possible template collision between rule annotations.

v1.97.13 #

Released at 2024-11-29

v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release

  • BUGFIX: vmagent: return 200 OK HTTP status code when importing data via Pushgateway protocol using multitenant URL format. See this issue and this pull request.
  • BUGFIX: vmagent: properly set TCP connection timeout for Kubernetes API server connection for metric scrapping with kubernetes_sd_configs. See this issue.
  • BUGFIX: vmagent: fix the resource_group filter for Azure service discovery on virtual machine scale sets. Previously, this filter did not apply to virtual machine scale sets, causing all virtual machines to be discovered. See this issue.
  • BUGFIX: vmauth: properly init ip_filters for unauthorized_user config section. Previously it was ignored and vmauth didn’t apply ip_filter for unauthorized access.
  • BUGFIX: vmsingle, vmselect in VictoriaMetrics cluster: properly return result for binary operation ^ aka pow at query requests for NaN values. See this issue for details.

v1.97.12 #

Released at 2024-11-15

v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release

  • BUGFIX: vmctl: drop rows that do not belong to the current series during import. The dropped rows should belong to another series whose tags are a superset of the current series. See this issue and this pull request. Thanks to @dpedu for reporting and cooperating with the test.
  • BUGFIX: vmsingle, vmselect in VictoriaMetrics cluster: keep the order of resulting time series when limit_offset is applied. See this issue.
  • BUGFIX: graphite: properly handle xFilesFactor=0 for transformRemoveEmptySeries function. See this PR for details.
  • BUGFIX: vmauth: properly check availability of all the backends before giving up when proxying requests. Previously, vmauth could return an error even if there were healthy backends available. See this issue for details.
  • BUGFIX: vmauth: fixed unauthorized routing behavior inconsistency. See this issue for details.

v1.97.11 #

Released at 2024-11-04

v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release

v1.97.10 #

Released at 2024-10-21

v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release

  • BUGFIX: vmui: fix error messages rendering from overflowing the screen with long messages. See this issue.
  • BUGFIX: vmctl: properly add metrics tags for opentsdb migration source. Previously it could have empty values. See this PR.
  • BUGFIX: VictoriaMetrics cluster: properly apply replication factor when storage node groups are used and replication factor is configured via global value such as -replicationFactor=2. Previously, global replication factor was ignored for storage node groups. See these docs for more information about storage groups configuration.
  • BUGFIX: vmalert: properly apply configuration changes during hot-reload to rule groups that haven’t started yet. Previously, configuration updates to such groups could have resulted into blocking all evaluations within the group, until vmalert restart.

v1.97.9 #

Released at 2024-10-02

v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release

  • BUGFIX: MetricsQL: consistently return the first non-NaN value from range_first function across all the returned data points. Previously NaN data points weren’t replaced with the first non-NaN value.

v1.97.8 #

Released at 2024-09-23

  • SECURITY: upgrade Go builder from Go1.22.5 to Go1.23.1. See the list of issues addressed in Go1.23.1.

  • SECURITY: upgrade base docker image (Alpine) from 3.20.2 to 3.20.3. See alpine 3.20.3 release notes.

  • BUGFIX: vmagent fix service discovery of Azure Virtual Machines for response contains nextLink in Host:Port format. See this issue.

  • BUGFIX: vmagent: properly consume messages from kafka. Previously vmagent could skip some messages during start-up.

  • BUGFIX: vmgateway: add missing datadog, newrelic, opentelemetry and pushgateway routes to the JWT authorization routes. Allows prefixed (prometheus/graphite) routes for query requests.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: properly cache empty list of matching time series for the given labels filter. This type of caching was broken since v1.97.0, which could result in the increased CPU usage when performing queries, which match zero time series. See this issue.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: Fixes start-up crash on Windows OS. See this issue for details.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: fix metric vm_object_references{type="indexdb"}. Previously, it was overcounted.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: properly ingest stale NaN samples. Previously it could be dropped if series didn’t exist at storage node. See this issue [https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5069] for details.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: properly track vm_missing_tsids_for_metric_id_total metric. See this issue for details.

  • BUGFIX: vmalert: do not send notifications without labels to Alertmanager. Such notifications are rejected by Alertmanager anyway. Before, vmalert could send alert notifications even if no label-value pairs left after applying alert_relabel_configs from notifier config.

  • BUGFIX: vmalert: properly update value of variable $activeAt in rules annotation during replay mode. Before, $activeAt could have provided incorrect values during replay.

  • BUGFIX: MetricsQL: properly handle c1 AND c2 and c1 OR c1 queries for constants c1 and c2. Previously such queries could return unexpected results. See this issue.

  • BUGFIX: all VictoriaMetrics components: increase default value of -loggerMaxArgLen cmd-line flag from 1000 to 5000. This should improve visibility on errors produced by very long queries.

v1.97.7 #

Released at 2024-08-28

v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release

  • BUGFIX: vmagent: fixes proxy_url authorization for scrape targets. Previously proxy authorization configuration was ignored for https targets. See this issue for details.
  • BUGFIX: vmagent fix service discovery of Azure Virtual Machines for response contains nextLink. See this issue.
  • BUGFIX: vmalert: respect HTTP headers defined in notifier configuration file for each request to notifiers. Previously, this param was ignored by mistake.
  • BUGFIX: graphite: respect -search.denyPartialResponse cmd-line flag and deny_partial_response query GET param when serving requests via Graphite API. Before, partial responses were always denied. Thanks to @penguinlav for the pull request.
  • BUGFIX: vmagent: account for -usePromCompatibleNaming cmd-line flag during when pushing data to remote storages. Thanks to @12345XXX for the pull request.
  • BUGFIX: MetricsQL: fix calculation histogram_quantile over Prometheus buckets with inconsistent values. It was producing incorrect results in case lower buckets. The issue was introduced in v1.102.0 release, see this issue for the details.
  • BUGFIX: vmalert, vmctl and snapshot API: verify correctness of URLs provided via cmd-line flags before executing HTTP requests. See this issue for details.
  • BUGFIX: all VictoriaMetrics components: forcefully set owner/group for release tars to 1000:1000. This helps to avoid unpacking issues on systems with limitations around UID:GID configuration. See this pull request.
  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: properly register index metrics for previous index part. See this issue for details.
  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: fix metric names registering in the per-day index for new dates for existing time series when making calls to /tags/tagSeries and /tags/tagMultiSeries handlers of Graphite API. See this for details.

v1.97.6 #

Released at 2024-07-17

v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release

v1.97.5 #

Released at 2024-06-07

v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release

  • SECURITY: upgrade Go builder from Go1.22.2 to Go1.22.4. See the list of issues addressed in Go1.22.3 and Go1.22.4.

  • SECURITY: upgrade base docker image (Alpine) from 3.19.1 to 3.20.0. See alpine 3.20.0 release notes.

  • BUGFIX: vmui: fix calendar display when UTC+00:00 timezone is set. See this issue.

  • BUGFIX: vmui: remove redundant requests on the Explore Cardinality page. See this issue.

  • BUGFIX: vmui: fix handling of URL params for browser history navigation (back and forward buttons). See this issue and this comment.

  • BUGFIX: vmagent: skip empty data blocks before sending to the remote write destination. Thanks to @viperstars for the pull request.

  • BUGFIX: properly estimate the needed memory for query execution if it has the format aggr_func(rollup_func[d] (for example, sum(rate(request_duration_seconds_bucket[5m]))). This should allow performing aggregations over bigger number of time series when VictoriaMetrics runs in environments with small amounts of available memory. The issue has been introduced in this commit in v1.83.0.

  • BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: correctly apply -inmemoryDataFlushInterval when it’s set to minimum supported value 1s.

  • BUGFIX: vmauth: properly release memory used for metrics during config reload. See this issue.

  • BUGFIX: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: filter deleted label names and values from /api/v1/labels and /api/v1/label/.../values responses when match[] filter matches small number of time series. The issue was introduced v1.81.0.

  • BUGFIX: vmalert: fix misleading error logs in vmalert’s stdout when unsupported HTTP path is requested.

  • BUGFIX: retry files delete attempts on vXFS file system for EEXIST error type. See the related issue. Thanks to @pludov for the pull request.

v1.97.4 #

Released at 2024-04-19

v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release

  • SECURITY: upgrade Go builder from Go1.21.7 to Go1.22.2. See the list of issues addressed in Go1.22.1 and the list of issues addressed in Go1.22.2.

  • BUGFIX: prevent from automatic deletion of newly registered time series when it is queried immediately after the addition. See this and this issue.

  • BUGFIX: vmagent: properly set Host header in requests to scrape targets when server_name option at tls_config is set. Previously the Host header was set incorrectly to the target hostname in this case.

  • BUGFIX: vmagent: properly set Host header in requests to scrape targets if it is specified via headers option. Thanks to @fholzer for the bugreport and the fix.

  • BUGFIX: vmalert: set correct endsAt value in notifications sent to the Alertmanager. Previously, a rule with evaluation intervals lower than 10s could never be triggered. See this issue for details.

  • BUGFIX: vmalert: properly account for -rule.resendDelay for alerting rules that are constantly switching state from inactive to firing. Before, notifications for such rules could have been skipped if state change happened more often than -rule.resendDelay. See this pull request for details.

  • BUGFIX: vmalert: respect -remoteWrite.maxBatchSize at shutdown period. See this issue. Thanks to @jiekun for the pull request.

  • BUGFIX: vmalert: supported any status codes from the range 200-299 from alertmanager. Previously, only 200 status code considered a successful action. See this issue.

  • BUGFIX: vmalert: avoid blocking /api/v1/rules, /api/v1/alerts, /metrics APIs when alerting rule uses template functions query, which could takes a while to execute. See this issue.

  • BUGFIX: vmbackupmanager: fix panic when performing a restore from GCS. Previously, filters for GCS objects were not properly set which caused objects to be returned instead of prefixes.

v1.97.3 #

Released at 2024-03-01

v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release

  • BUGFIX: downgrade Go builder from 1.22.0 to 1.21.7, since 1.22.0 contains the bug, which can lead to deadlocked HTTP connections to remote storage systems, scrape targets and service discovery endpoints at vmagent. This may result in incorrect service discovery, target scraping and failed sending samples to remote storage.
  • BUGFIX: take into account -search.maxUniqueTimeseries limit at /api/v1/labels, /api/v1/label/…/values and /api/v1/series. This limit was ignored at these endpoints starting from v1.97.0 when trying to address this issue. It has been appeared that this can result in high CPU usage when heavyweight queries hit these endpoints. This can prevent from executing lightweight queries to these endpoints. It is better to return limit exceeded error to heavyweight queries instead of burning CPU.
  • BUGFIX: all VictoriaMetrics components: return back periodic closing of incoming connections to -httpListenAddr every 2 minutes, which was disabled in v1.97.2 when addressing this issue. See this comment for details on why the periodic closing of incoming connections has been returned back.
  • BUGFIX: vmagent: accept OpenTelemetry data at /opentelemetry/v1/metrics, since the /v1/metrics suffix is hardcoded at OpenTelemetry protocol specification.
  • BUGFIX: vmagent: fix possible deadlock when sharding among remote storages is enabled with -remoteWrite.shardByURL command-line flag. Thanks to @penguinlav for the fix for this issue.
  • BUGFIX: fix the misleading error 0ms is out of allowed range [0 ... when passing step=0 to /api/v1/query or /api/v1/query_range. See this issue.
  • BUGFIX: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: fixed floating-point error when parsing time in RFC3339 format. See this issue for details.
  • BUGFIX: vmalert: consistently sort groups by name and filename on /groups page in UI. This should prevent non-deterministic sorting for groups with identical names.
  • BUGFIX: vmalert: reduce memory usage for ENT version of vmalert for configurations with high number of groups with enabled multitenancy.
  • BUGFIX: MetricsQL: properly construct the destination label in label_join if it is used as source label. See this issue for details.

v1.97.2 #

Released at 2024-02-14

v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release

  • SECURITY: upgrade Go builder from Go1.21.6 to Go1.22.0. See the list of issues addressed in Go1.21.7, plus the changelog for Go1.22.0.

  • BUGFIX: all VictoriaMetrics components: do not close connections to -httpListenAddr every 2 minutes. This behavior didn’t help spreading load among multiple backend servers behind load-balancing TCP proxy. Instead, it could lead to hard-to-debug issues like this one. If you still need periodically closing client connections because of some reason, then pass the desired timeout to -http.connTimeout command-line flag.

  • BUGFIX: vmagent: reduce CPU usage when -promscrape.dropOriginalLabels command-line flag is set. This issue has been introduced in v1.96.0 when addressing this feature request.

  • BUGFIX: vmauth: properly release memory during config reload. See this issue.

  • BUGFIX: vmauth: properly expose vmauth_unauthorized_user_concurrent_requests_capacity, vmauth_unauthorized_user_concurrent_requests_current, vmauth_user_concurrent_requests_capacity and vmauth_user_concurrent_requests_current metrics after config reload. Previously these metrics didn’t work after config reload.

  • BUGFIX: MetricsQL: properly propagate label filters from multiple arguments passed to aggregate functions. For example, sum({job="foo"}, {job="bar"}) by (job) + a was improperly optimized to sum({job="foo"}, {job="bar"}) by (job) + a{job="foo"} before being executed. This could lead to unexpected results. See this issue.

  • BUGFIX: MetricsQL: properly handle precision errors when calculating changes, changes_prometheus, increases_over_time and resets functions. See this issue.

  • BUGFIX: properly store staleness markers for self-scraped metrics on single-node VictoriaMetrics shutdown. See this issue.

  • BUGFIX: prevent from possible too big indexBlockSize panic when samples with too long label values (~64Kb) are ingested into VictoriaMetrics.

v1.97.1 #

Released at 2024-02-01

v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release

  • FEATURE: vmagent: add support for username_file option at basic_auth section in scrape_configs. See this feature request. Thanks to @wasim-nihal for the initial implementation.

  • BUGFIX: vmagent: fix the increased CPU usage when sending the data to remote storage. The issue has been introduced in v1.97.0.

  • BUGFIX: fix runtime error: slice bounds out of range panic, which can occur during query execution. See this issue. The bug has been introduced in v1.97.0.

  • BUGFIX: MetricsQL: properly handle avg_over_time({some_filter}[d]) keep_metric_names queries, where some_filter matches multiple time series with multiple names, while d is bigger or equal to 3h. See this issue.

  • BUGFIX: dashboards/single: fix typo in query for version annotation which falsely produced many version change events.

v1.97.0 #

Released at 2024-01-30

v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release

  • SECURITY: upgrade base docker image (Alpine) from 3.19.0 to 3.19.1. See alpine 3.19.1 release notes.

  • SECURITY: upgrade Go builder from Go1.21.5 to Go1.21.6. See the list of issues addressed in Go1.21.6.

  • FEATURE: improve new time series registration speed on systems with high number of CPU cores. Thanks to @misutoth for the initial idea and implementation.

  • FEATURE: make background merge more responsive and scalable. This should help the following issues: 5190, 3425, 648.

  • FEATURE: graphite: add support for negative index in groupByNode and aliasByNode functions. Thanks to @rbizos for the pull request.

  • FEATURE: vmagent: add support for discovering Hetzner Cloud and Hetzner Robot scrape targets. See this feature request and these docs.

  • FEATURE: vmagent: add support for DataDog v2 data ingestion protocol. See these docs and this feature request.

  • FEATURE: vmagent: reduce initial memory usage when scraping targets, which return huge responses (for example, kube-state-metrics in large Kubernetes cluster may return 100MB+ responses), without the need to explicitly enable stream parsing mode. See this issue.

  • FEATURE: vmagent: expose ability to set OAuth2 endpoint parameters per each -remoteWrite.url via the command-line flag -remoteWrite.oauth2.endpointParams. See these docs. Thanks to @mhill-holoplot for the pull request.

  • FEATURE: vmagent: add ability to set attach_metadata.node=true option for all the kubernetes_sd_configs defined at -promscrape.config via -promscrape.kubernetes.attachNodeMetadataAll command-line flag. See this feature request. Thanks to @wasim-nihal for the initial implementation.

  • FEATURE: vmagent: do not send unfinished aggregation state on shutdown or hot config reload by default, as it tend to produce unexpected anomalies with lower values. The old behavior can be restored by specifying flush_on_shutdown: true setting in streaming aggregation config. See more details here.

  • FEATURE: streaming aggregation: expand %{ENV_VAR} placeholders in config files with the corresponding environment variable values.

  • FEATURE: vmalert: expose ability to set OAuth2 endpoint parameters via the following command-line flags:

    • -datasource.oauth2.endpointParams for -datasource.url
    • -notifier.oauth2.endpointParams for -notifier.url
    • -remoteRead.oauth2.endpointParams for -remoteRead.url
    • -remoteWrite.oauth2.endpointParams for -remoteWrite.url

  • FEATURE: vmauth: add ability to proxy incoming requests to different backends based on the requested host via src_hosts option at url_map. See these docs.

  • FEATURE: vmauth: expose vmauth_user_request_backend_errors_total and vmauth_unauthorized_user_request_backend_errors_total metrics, which track the number of failed requests because of backend errors. See this feature request.

  • FEATURE: vmauth: add an ability to specify additional labels for per-user metrics via metric_labels section. See this feature request.

  • FEATURE: all VictoriaMetrics components: break HTTP client connection if an error occurs after the server at -httpListenAddr already sent response status code. Previously such an error couldn’t be detected at client side. Now the client will get an error about invalid chunked response. The error message is simultaneously written to the server log and in the last line of the response. This should help detecting errors when migrating data between VictoriaMetrics instances by vmctl. See this issue.

  • FEATURE: all VictoriaMetrics components: add ability to specify arbitrary HTTP headers to send with every request to -pushmetrics.url. See push metrics docs.

  • FEATURE: all VictoriaMetrics components: add -metrics.exposeMetadata command-line flag, which allows displaying TYPE and HELP metadata at /metrics page exposed at -httpListenAddr. This may be needed when the /metrics page is scraped by collector, which requires the TYPE and HELP metadata such as Google Cloud Managed Prometheus.

  • FEATURE: all VictoriaMetrics components: add ability to dynamically re-read auth keys and passwords from files and urls when using file:///path/to/file or http://host/path syntax for the following command-line flags: -configAuthKey, -deleteAuthKey, -flagsAuthKey, -forceMergeAuthKey, -forceFlushAuthKey, -httpAuth.password, -metricsAuthKey, -pprofAuthKey, -reloadAuthKey, -search.resetCacheAuthKey, -snapshotAuthKey. For example, -httpAuth.password=file:///path/to/password. See these docs for details.

  • FEATURE: dashboards/cluster: add panels for detailed visualization of traffic usage between vmstorage, vminsert, vmselect components and their clients. New panels are available in the rows dedicated to specific components.

  • FEATURE: dashboards/cluster: update “Slow Queries” panel to show percentage of the slow queries to the total number of read queries served by vmselect. The percentage value should make it more clear for users whether there is a service degradation.

  • FEATURE: dashboards/single: change dashboard title from VictoriaMetrics to VictoriaMetrics - single-node. The new title should provide better understanding of this dashboard purpose.

  • FEATURE: vmctl: rename cmd-line flag vm-native-disable-retries to vm-native-disable-per-metric-migration to better reflect its meaning.

  • FEATURE: vmctl: add -vm-native-src-insecure-skip-verify and -vm-native-dst-insecure-skip-verify command-line flags for native protocol. It can be used for skipping TLS certificate verification when connecting to the source or destination addresses.

  • FEATURE: Alerting rules for VictoriaMetrics: add job label to DiskRunsOutOfSpace alerting rule, so it is easier to understand to which installation the triggered instance belongs.

  • FEATURE: vmstorage: add tenant identifier for log messages regarding dropping excessive labels due to limits defined by -maxLabelsPerTimeseries or -maxLabelValueLen command-line flags. Previously, it was hard to understand to which tenant the dropped labels belong.

  • FEATURE: vmui: add the ability to export and import query reports:

    • add a Query Analyzer page that allows you to build graphs from JSON data containing the results of executing a query request.
    • add an Export query button to the graph that saves the result of executing the query in JSON. See this pull request.

  • FEATURE: vmui: add -vmui.defaultTimezone flag to set a default timezone. See this issue and these docs.

  • FEATURE: vmui: include UTC in the timezone selection dropdown for standardized time referencing. See this issue.

  • FEATURE: vmui: add the ability to expand/collapse all tracing entries and download tracing data in .json format. See this issue.

  • FEATURE: add VictoriaMetrics datasource to docker compose environment. See this pull request.

  • BUGFIX: properly return the list of matching label names and label values from /api/v1/labels and /api/v1/label/.../values when the database contains more than -search.maxUniqueTimeseries unique time series on the selected time range. Previously VictoriaMetrics could return the number of matching timeseries exceeds ... error in this case. See this issue.

  • BUGFIX: properly return errors from export APIs. Previously these errors were silently suppressed. See this pull request.

  • BUGFIX: VictoriaMetrics cluster: properly return full results when -search.skipSlowReplicas command-line flag is passed to vmselect and when vmstorage groups are in use. Previously partial results could be returned in this case.

  • BUGFIX: vminsert: properly accept samples via OpenTelemetry data ingestion protocol when these samples have no resource attributes. Previously such samples were silently skipped.

  • BUGFIX: vmstorage: added missing -inmemoryDataFlushInterval command-line flag, which was missing in VictoriaMetrics cluster after implementing this feature in v1.85.0.

  • BUGFIX: vmstorage: properly expire storage/prefetchedMetricIDs cache. Previously this cache was never expired, so it could grow big under high churn rate. This could result in increasing CPU load over time.

  • BUGFIX: vmalert: check -external.url schema when starting vmalert, must be http or https. Before, alertmanager could reject alert notifications if -external.url contained no or wrong schema.

  • BUGFIX: vmalert: automatically add exported_ prefix for original evaluation result label if it’s conflicted with external or reserved one, previously it was overridden. See this issue.

  • BUGFIX: vmalert: autogenerate ALERTS_FOR_STATE time series for alerting rules with for: 0. Previously, ALERTS_FOR_STATE was generated only for alerts with for > 0. The change aligns with Prometheus behavior. See more details in this issue.

  • BUGFIX: vmalert: fix data race during hot-config reload. The result of the race could cause all rules from updated group to fail with context cancelled error.

  • BUGFIX: MetricsQL: consistently sort results for q1 or q2 query, so they do not change colors with each refresh in Grafana. See this issue.

  • BUGFIX: MetricsQL: properly return results from bottomk and bottomk_*() functions when some of these results contain NaN values. See this issue. Thanks to @xiaozongyang for the fix.

  • BUGFIX: MetricsQL: properly handle queries, which wrap rollup functions with multiple arguments without explicitly specified lookbehind window in square brackets into aggregate functions. For example, sum(quantile_over_time(0.5, process_resident_memory_bytes)) was resulting to expecting at least 2 args to ...; got 1 args error. Thanks to @atykhyy for the pull request.

  • BUGFIX: vmctl: retry on import errors in vm-native mode. Before, retries happened only on writes into a network connection between source and destination. But errors returned by server after all the data was transmitted were logged, but not retried.

  • BUGFIX: vmagent: properly assume role with AWS IRSA authorization. Previously role chaining was not supported. See this issue for details.

  • BUGFIX: vmagent: exit if there is config syntax error in scrape_config_files when -promscrape.config.strictParse=true. See this issue.

  • BUGFIX: vmagent: properly discover targets for role: endpoints and role: endpointslice in kubernetes_sd_configs. Previously some endpoints and endpointslice targets could be left undiscovered or some targets could have missing __meta_* labels when performing service discovery in busy Kubernetes clusters with large number of pods. See this pull request.

  • BUGFIX: vmagent: respect explicitly set series_limit: 0 in scrape_config. This allows removing series_limit restriction on a per-scrape_config basis when global limit is set via -promscrape.seriesLimitPerTarget. Previously, 0 value was ignored in favor of -promscrape.seriesLimitPerTarget.

  • BUGFIX: vmagent: do not discover scrape targets for already terminated pods and containers in kubernetes_sd_configs. Such pods and containers cannot be scraped and cannot resurrect, so there is no sense in generating scrape targets for them.

  • BUGFIX: vmagent: properly follow redirects to another hostnames when follow_redirects: true option is set in scrape_configs while stream parsing mode is disabled. See this issue and this pull request.

  • BUGFIX: vmui: fix a link for the statistic inaccuracy explanation in the cardinality explorer tool. See this issue.

  • BUGFIX: vmui: fix the display of autocomplete results and cache the results. See this issue and this issue.

  • BUGFIX: vmui: send step param for instant queries. The change reverts this issue due to reasons explained in this comment.

  • BUGFIX: vmui: fix the Enter key press issue in the Query field when using autocomplete. See this issue.

  • BUGFIX: all: fix potential panic during components shutdown when metrics push is configured. See this issue. Thanks to @zhdd99 for the pull request.

  • BUGFIX: MetricsQL: properly process queries with too big lookbehind window such as foo[100y]. Previously, such queries could return empty responses even if foo is present in database. See this issue.

  • BUGFIX: MetricsQL: properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase(). See this issue.

  • BUGFIX: vmselect: vmsingle/vmselect returns http status 429 (TooManyRequests) instead of 503 (ServiceUnavailable) when max concurrent requests limit is reached.

  • BUGFIX: vmselect: set proper timestamps in instant query results if the query contains rollup functions with lookbehind windows in square brackets exceeding 3 hours. For example, sum(avg_over_time(up[24h])). This bug has been introduced in v1.95.0. See this and this issues.

v1.96.0 #

See changes here

v1.95.1 #

See changes here

v1.95.0 #

See changes here

v1.94.0 #

See changes here

v1.93.16 #

Released at 2024-07-17

v1.93.x is a line of LTS releases. It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since v1.93.0 release

  • SECURITY: upgrade Go builder from Go1.22.4 to Go1.22.5. See the list of issues addressed in Go1.22.5.

  • SECURITY: upgrade base docker image (Alpine) from 3.20.0 to 3.20.1. See alpine 3.20.1 release notes.

  • BUGFIX: vmbackup: properly configure authentication with S3 when -configFilePath cmd-line flag is specified.

  • BUGFIX: vmalert: fix path for system links printed on default vmalert’s UI page when -http.pathPrefix is set.

  • BUGFIX: vmalert: fix incorrect redirection in WebUI of vmalert. See this issue and this pull request.

  • BUGFIX: vmalert enterprise: properly configure authentication with S3 when -s3.configFilePath cmd-line flag is specified for reading rule configs.

  • BUGFIX: vmalert: properly specify oauth2 ClientSecret when configuring authentication for notifier.url. See this issue for details. Thanks to @yincongcyincong for the pull request.

  • BUGFIX: vmui: copy row button in Table view produces unexpected result. See this issue and pull request.

  • BUGFIX: MetricsQL: properly calculate histogram_quantile over Prometheus buckets with inconsistent values. See this comment and this pull request. Updates this issue.

v1.93.15 #

Released at 2024-06-07

v1.93.x is a line of LTS releases. It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since v1.93.0 release

  • SECURITY: upgrade Go builder from Go1.22.2 to Go1.22.4. See the list of issues addressed in Go1.22.3 and Go1.22.4.

  • SECURITY: upgrade base docker image (Alpine) from 3.19.1 to 3.20.0. See alpine 3.20.0 release notes.

  • BUGFIX: vmui: fix calendar display when UTC+00:00 timezone is set. See this issue.

  • BUGFIX: vmagent: skip empty data blocks before sending to the remote write destination. Thanks to @viperstars for the pull request.

  • BUGFIX: properly estimate the needed memory for query execution if it has the format aggr_func(rollup_func[d] (for example, sum(rate(request_duration_seconds_bucket[5m]))). This should allow performing aggregations over bigger number of time series when VictoriaMetrics runs in environments with small amounts of available memory. The issue has been introduced in this commit in v1.83.0.

  • BUGFIX: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: filter deleted label names and values from /api/v1/labels and /api/v1/label/.../values responses when match[] filter matches small number of time series. The issue was introduced v1.81.0.

  • BUGFIX: vmalert: fix misleading error logs in vmalert’s stdout when unsupported HTTP path is requested.

  • BUGFIX: retry files delete attempts on vXFS file system for EEXIST error type. See the related issue. Thanks to @pludov for the pull request.

v1.93.14 #

Released at 2024-04-19

v1.93.x is a line of LTS releases. It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since v1.93.0 release

  • SECURITY: upgrade Go builder from Go1.21.7 to Go1.22.2. See the list of issues addressed in Go1.22.1 and the list of issues addressed in Go1.22.2.

  • BUGFIX: vmalert: set correct endsAt value in notifications sent to the Alertmanager. Previously, a rule with evaluation intervals lower than 10s could never be triggered. See this issue for details.

  • BUGFIX: vmalert: properly account for -rule.resendDelay for alerting rules that are constantly switching state from inactive to firing. Before, notifications for such rules could have been skipped if state change happened more often than -rule.resendDelay. See this pull request for details.

  • BUGFIX: vmalert: respect -remoteWrite.maxBatchSize at shutdown period. See this issue. Thanks to @jiekun for the pull request.

  • BUGFIX: vmalert: supported any status codes from the range 200-299 from alertmanager. Previously, only 200 status code considered a successful action. See this issue.

v1.93.13 #

Released at 2024-03-01

v1.93.x is a line of LTS releases. It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since v1.93.0 release

  • BUGFIX: downgrade Go builder from 1.22.0 to 1.21.7, since 1.22.0 contains the bug, which can lead to deadlocked HTTP connections to remote storage systems, scrape targets and service discovery endpoints at vmagent. This may result in incorrect service discovery, target scraping and failed sending samples to remote storage.

  • BUGFIX: vmagent: accept OpenTelemetry data at /opentelemetry/v1/metrics, since the /v1/metrics suffix is hardcoded at OpenTelemetry protocol specification.

  • BUGFIX: vmagent: fix possible deadlock when sharding among remote storages is enabled with -remoteWrite.shardByURL command-line flag. Thanks to @penguinlav for the fix for this issue.

  • BUGFIX: fix the misleading error 0ms is out of allowed range [0 ... when passing step=0 to /api/v1/query or /api/v1/query_range. See this issue.

  • BUGFIX: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: fixed floating-point error when parsing time in RFC3339 format. See this issue for details.

  • BUGFIX: vmalert: consistently sort groups by name and filename on /groups page in UI. This should prevent non-deterministic sorting for groups with identical names.

  • BUGFIX: MetricsQL: properly construct the destination label in label_join if it is used as source label. See this issue for details.

v1.93.12 #

Released at 2024-02-14

v1.93.x is a line of LTS releases. It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since v1.93.0 release

v1.93.11 #

Released at 2024-02-01

v1.93.x is a line of LTS releases. It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since v1.93.0 release

  • SECURITY: upgrade base docker image (Alpine) from 3.19.0 to 3.19.1. See alpine 3.19.1 release notes.

  • BUGFIX: properly return errors from export APIs. Previously these errors were silently suppressed. See this pull request.

  • BUGFIX: vmagent: properly discover targets for role: endpoints and role: endpointslice in kubernetes_sd_configs. Previously some endpoints and endpointslice targets could be left undiscovered or some targets could have missing __meta_* labels when performing service discovery in busy Kubernetes clusters with large number of pods. See this pull request.

  • BUGFIX: vmagent: respect explicitly set series_limit: 0 in scrape_config. This allows removing series_limit restriction on a per-scrape_config basis when global limit is set via -promscrape.seriesLimitPerTarget. Previously, 0 value was ignored in favor of -promscrape.seriesLimitPerTarget.

  • BUGFIX: MetricsQL: properly process queries with too big lookbehind window such as foo[100y]. Previously, such queries could return empty responses even if foo is present in database. See this issue.

  • BUGFIX: MetricsQL: properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase(). See this issue.

v1.93.10 #

Released at 2024-01-17

v1.93.x is a line of LTS releases. It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since v1.93.0 release

  • SECURITY: upgrade Go builder from Go1.21.5 to Go1.21.6. See the list of issues addressed in Go1.21.6.

  • BUGFIX: vminsert: properly accept samples via OpenTelemetry data ingestion protocol when these samples have no resource attributes. Previously such samples were silently skipped.

  • BUGFIX: vmstorage: added missing -inmemoryDataFlushInterval command-line flag, which was missing in VictoriaMetrics cluster after implementing this feature in v1.85.0.

  • BUGFIX: vmstorage: properly expire storage/prefetchedMetricIDs cache. Previously this cache was never expired, so it could grow big under high churn rate. This could result in increasing CPU load over time.

  • BUGFIX: vmalert: check -external.url schema when starting vmalert, must be http or https. Before, alertmanager could reject alert notifications if -external.url contained no or wrong schema.

  • BUGFIX: MetricsQL: properly return results from bottomk and bottomk_*() functions when some of these results contain NaN values. See this issue. Thanks to @xiaozongyang for the fix.

  • BUGFIX: MetricsQL: properly handle queries, which wrap rollup functions with multiple arguments without explicitly specified lookbehind window in square brackets into aggregate functions. For example, sum(quantile_over_time(0.5, process_resident_memory_bytes)) was resulting to expecting at least 2 args to ...; got 1 args error. Thanks to @atykhyy for the pull request.

  • BUGFIX: vmagent: properly assume role with AWS IRSA authorization. Previously role chaining was not supported. See this issue for details.

  • BUGFIX: all: fix potential panic during components shutdown when metrics push is configured. See this issue. Thanks to @zhdd99 for the pull request.

  • BUGFIX: vmctl: check for Error field in response from influx client during migration. Before, only network errors were checked. Thanks to @wozz for the pull request.

  • BUGFIX: vmctl: retry on import errors in vm-native mode. Before, retries happened only on writes into a network connection between source and destination. But errors returned by server after all the data was transmitted were logged, but not retried.

v1.93.9 #

See changes here

v1.93.8 #

See changes here

v1.93.7 #

See changes here

v1.93.6 #

See changes here

v1.93.5 #

See changes here

v1.93.4 #

See changes here

v1.93.3 #

See changes here

v1.93.2 #

See changes here

v1.93.1 #

See changes here

v1.93.0 #

See changes here

v1.92.1 #

See changes here

v1.92.0 #

See changes here

v1.91.3 #

See changes here

v1.91.2 #

See changes here

v1.91.1 #

See changes here

v1.91.0 #

See changes here

v1.90.0 #

See changes here

v1.89.1 #

See changes here

v1.89.0 #

See changes here

v1.88.1 #

See changes here

v1.88.0 #

See changes here

v1.87.14 #

Released at 2024-02-01

v1.87.x is a line of LTS releases. It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release

  • SECURITY: upgrade base docker image (Alpine) from 3.19.0 to 3.19.1. See alpine 3.19.1 release notes.

  • BUGFIX: properly return errors from export APIs. Previously these errors were silently suppressed. See this pull request.

  • BUGFIX: vmagent: respect explicitly set series_limit: 0 in scrape_config. This allows removing series_limit restriction on a per-scrape_config basis when global limit is set via -promscrape.seriesLimitPerTarget. Previously, 0 value was ignored in favor of -promscrape.seriesLimitPerTarget.

  • BUGFIX: MetricsQL: properly process queries with too big lookbehind window such as foo[100y]. Previously, such queries could return empty responses even if foo is present in database. See this issue.

  • BUGFIX: MetricsQL: properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase(). See this issue.

v1.87.13 #

Released at 2024-01-17

v1.87.x is a line of LTS releases. It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since v1.87.0 release

  • SECURITY: upgrade Go builder from Go1.21.5 to Go1.21.6. See the list of issues addressed in Go1.21.6.

  • BUGFIX: vmstorage: added missing -inmemoryDataFlushInterval command-line flag, which was missing in VictoriaMetrics cluster after implementing this feature in v1.85.0.

  • BUGFIX: MetricsQL: properly handle queries, which wrap rollup functions with multiple arguments without explicitly specified lookbehind window in square brackets into aggregate functions. For example, sum(quantile_over_time(0.5, process_resident_memory_bytes)) was resulting to expecting at least 2 args to ...; got 1 args error. Thanks to @atykhyy for the pull request.

  • BUGFIX: vmstorage: properly expire storage/prefetchedMetricIDs cache. Previously this cache was never expired, so it could grow big under high churn rate. This could result in increasing CPU load over time.

  • BUGFIX: MetricsQL: properly return results from bottomk and bottomk_... functions when some of these results contain NaN values. See this issue. Thanks to @xiaozongyang for the fix.

  • BUGFIX: all: fix potential panic during components shutdown when metrics push is configured. See this issue. Thanks to @zhdd99 for the pull request.

v1.87.12 #

See changes here

v1.87.11 #

See changes here

v1.87.10 #

See changes here

v1.87.9 #

See changes here

v1.87.8 #

See changes here

v1.87.7 #

See changes here

v1.87.6 #

See changes here

v1.87.5 #

See changes here

v1.87.4 #

See changes here

v1.87.3 #

See changes here

v1.87.2 #

See changes here

v1.87.1 #

See changes here

v1.87.0 #

See changes here

v1.86.2 #

See changes here

v1.86.1 #

See changes here

v1.86.0 #

See changes here

v1.85.3 #

See changes here

v1.85.2 #

See changes here

v1.85.1 #

See changes here

v1.85.0 #

See changes here

v1.84.0 #

See changes here

v1.83.1 #

See changes here

v1.83.0 #

See changes here

v1.82.1 #

See changes here

v1.82.0 #

See changes here

v1.81.2 #

See changes here

v1.81.1 #

See changes here

v1.81.0 #

See changes here

v1.80.0 #

See changes here

v1.79.14 #

See changes here

v1.79.13 #

See changes here

v1.79.12 #

See changes here

v1.79.11 #

See changes here

v1.79.10 #

See changes here

v1.79.9 #

See changes here

v1.79.8 #

See changes here

v1.79.7 #

See changes here

v1.79.6 #

See changes here

v1.79.5 #

See changes here

v1.79.4 #

See changes here

v1.79.3 #

See changes here

v1.79.2 #

See changes here

v1.79.1 #

See changes here

v1.79.0 #

See changes here

v1.78.1 #

See changes here

v1.78.0 #

See changes here

v1.77.2 #

See changes here

v1.77.1 #

See changes here

v1.77.0 #

See changes here

v1.76.1 #

See changes here

v1.76.0 #

See changes here

v1.75.1 #

See changes here

v1.75.0 #

See changes here

v1.74.0 #

See changes here

v1.73.1 #

See changes here

v1.73.0 #

See changes here

v1.72.0 #

See changes here

v1.71.0 #

See changes here

v1.70.0 #

See changes here

v1.69.0 #

See changes here

v1.68.0 #

See changes here

v1.67.0 #

See changes here

v1.66.2 #

See changes here

v1.66.1 #

See changes here

v1.66.0 #

See changes here

v1.65.0 #

See changes here

v1.64.1 #

See changes here

v1.64.0 #

See changes here

v1.63.0 #

See changes here

v1.62.0 #

See changes here

v1.61.1 #

See changes here

v1.61.0 #

See changes here

v1.60.0 #

See changes here

v1.59.0 #

See changes here

v1.58.0 #

See changes here

v1.57.1 #

See changes here

v1.57.0 #

See changes here

v1.56.0 #

See changes here

v1.55.1 #

See changes here

v1.55.0 #

See changes here

v1.54.1 #

See changes here

v1.54.0 #

See changes here

v1.53.1 #

See changes here

v1.53.0 #

See changes here

v1.52.0 #

See changes here

v1.51.0 #

See changes here

v1.50.2 #

See changes here

v1.50.1 #

See changes here

v1.50.0 #

See changes here

v1.49.0 #

See changes here

v1.48.0 #

See changes here

v1.47.0 #

See changes here

v1.46.0 #

See changes here

v1.45.0 #

See changes here

v1.44.0 #

See changes here

v1.43.0 #

See changes here

v1.42.0 #

See changes here

Previous releases #

See releases page.

Content

Back to Page