Please find the changelog for VictoriaMetrics Anomaly Detection below.
v1.16.1 #
Released: 2024-10-02
- FIX: This patch release prevents the service from crashing by rolling back the version of a third-party dependency. Affected releases: v1.16.0.
v1.16.0 #
Released: 2024-10-01
Note: A bug was discovered in this release that causes the service to crash. Please use the patch v1.16.1 to resolve this issue.
FEATURE: Introduced data dumps to a host filesystem for VmReader. Resource-intensive setups (multiple queries returning many metrics, bigger
fit_window
arg) will have RAM consumption reduced during fit calls.IMPROVEMENT: Added a
groupby
argument for logical grouping in multivariate models. When specified, a separate multivariate model is trained for each unique combination of label values in thegroupby
columns. For example, to perform multivariate anomaly detection on metrics at the machine level without cross-entity interference, you can usegroupby: [host]
orgroupby: [instance]
, ensuring one model per entity being trained (e.g., per host). Please find more details here.IMPROVEMENT: Improved performance of VmReader on multicore instances for reading and data processing.
IMPROVEMENT: Introduced new CLI argument aliases to enhance compatibility with Helm charts (i.e. using secrets) and better align with VictoriaMetrics flags:
--licenseFile
as an alias for--license-file
--license.forceOffline
as an alias for--license-verify-offline
--loggerLevel
as an alias for--log-level
- The previous argument format is retained for backward compatibility.
FIX: The
provide_series
common argument now correctly filters the written time series in the IsolationForestMultivariate model.
v1.15.9 #
Released: 2024-08-27
- IMPROVEMENT: Added support for bearer token authentication in
push
mode within the self-monitoring configuration section.
v1.15.8 #
Released: 2024-08-27
- FIX: Made minor adjustments to how the reader and writer handle bearer tokens across different modes.
v1.15.7 #
Released: 2024-08-27
- FIX: Made minor adjustments to how the reader and writer handle bearer tokens across different modes.
v1.15.6 #
Released: 2024-08-26
- IMPROVEMENT: Introduced the
bearer_token_file
argument to the reader and writer components to enhance secret management.
v1.15.5 #
Released: 2024-08-19
- FIX: following v1.15.2 online model enhancement, now
data_range
parameter is correctly initialized for online models, created (for new time series returned by particular query) duringinfer
calls.
v1.15.4 #
Released: 2024-08-15
- IMPROVEMENT: better config handling of writer and monitoring sections if using
vmanomaly
with helm charts.
v1.15.3 #
Released: 2024-08-14
- IMPROVEMENT: better config handling of
reader
section if usingvmanomaly
with helm charts.
v1.15.2 #
Released: 2024-08-13
- IMPROVEMENT: Enhanced online models (e.g.,
OnlineQuantileModel
) to automatically create model instances for unseen time series duringinfer
calls, eliminating the need to wait for the nextfit
call. This ensures no inferences are skipped when using online models. - FIX: Corrected an issue with the
OnlineMADModel
to ensure proper functionality when used in combination with on-disk model dump mode. - FIX: Addressed numerical instability in the
OnlineQuantileModel
whenuse_transform
is set toTrue
. - FIX: Resolved a logging issue that could cause a
RuntimeError: reentrant call inside <_io.BufferedWriter name='<stderr>'>
when a termination event was received.
v1.15.1 #
Released: 2024-08-10
FEATURE: Introduced backward-compatible
data_range
query-specific parameter to the VmReader. It enables the definition of valid data ranges for input per individual query inqueries
, resulting in:- High anomaly scores (>1) when the data falls outside the expected range, indicating a data constraint violation.
- Lowest anomaly scores (=0) when the model’s predictions (
yhat
) fall outside the expected range, signaling uncertain predictions. - For more details, please refer to the documentation.
IMPROVEMENT: Added
latency_offset
argument to the VmReader to override the default-search.latencyOffset
flag of VictoriaMetrics (30s). The default value is set to 1ms, which should help in cases wheresampling_frequency
is low (10-60s) andsampling_frequency
equalsinfer_every
in the PeriodicScheduler. This prevents users from receivingservice - WARNING - [Scheduler [scheduler_alias]] No data available for inference.
warnings in logs and allows for consecutiveinfer
calls without gaps. To restore the backward compatible behavior, set it equal to your-search.latencyOffset
value in VmReader config section.FIX: Ensure the
use_transform
argument of theOnlineQuantileModel
functions as intended.FIX: Add a docstring for
query_from_last_seen_timestamp
arg of VmReader.
v1.15.0 #
Released: 2024-08-06
FEATURE: Introduced models that support online learning for stream-like input. These models significantly reduce the amount of data required for the initial fit stage. For example, they enable reducing
fit_every
from weeks to hours and increasingfit_every
from hours to weeks in the PeriodicScheduler, significantly reducing the peak amount of data queried from VictoriaMetrics duringfit
stages. The next models were added:OnlineZscoreModel
- online version of existing Z-score implementation with the same exact behavior.OnlineMADModel
- online version of existing MADModel implementation with approximate behavior, based on t-digests for online quantile estimation.OnlineQuantileModel
- online quantile model, that supports custom ranges for seasonality estimation to cover more complex data patterns.- Find out more about online models specifics in correspondent section.
FEATURE: Introduced the
optimized_business_params
key (list of strings) to theAutoTuned
optimization_params
. This allows particular business-specific parameters such asdetection_direction
andmin_dev_from_expected
to remain unchanged during optimizations, retaining their default values.IMPROVEMENT: Optimized the
AutoTuned
model logic to minimize deviations from the expectedanomaly_percentage
specified in the configuration and the detected percentage in the data, while also reducing discrepancies between the actual values (y
) and the predictions (yhat
).IMPROVEMENT: Allow
ProphetModel
to fit with multiple seasonalities when used inAutoTuned
mode.
v1.14.2 #
Released: 2024-07-26
v1.14.1 #
Released: 2024-07-26
FEATURE: Allow to process larger data chunks in VmReader that exceed
-search.maxPointsPerTimeseries
constraint in VictoriaMetrics by splitting the range and sending multiple requests. A warning is printed in logs, suggesting reducing the range or step, or increasingsearch.maxPointsPerTimeseries
constraint in VictoriaMetrics, which is still a recommended option.FEATURE: Backward-compatible redesign of
queries
arg of VmReader. Old format of{q_alias1: q_expr1, q_alias2: q_expr2, ...}
will be implicitly converted to a new one with a warning raised in logs. New format allows to specify per-query parameters, likestep
to reduce amount of data read from VictoriaMetrics TSDB and to allow config flexibility. Find out more in Per-query parameters section of VmReader.IMPROVEMENT: Added multi-platform builds for
linux/amd64
andlinux/arm64
architectures.
v1.13.3 #
Released: 2024-07-17
- FIX: now validation of
args
argument forHoltWinters
model works properly.
v1.13.2 #
Released: 2024-07-15
- IMPROVEMENT: update
node-exporter
preset to reduce false positives - FIX: add
verify_tls
arg forpush
monitoring section. Also,verify_tls
is now correctly used in VmWriter. - FIX: now
AutoTuned
model wrapper works correctly in on-disk model storage mode. - FIX: now rolling models, like
RollingQuantile
are properly handled in One-off scheduler, when wrapped inAutoTuned
v1.13.0 #
Released: 2024-06-11
- FEATURE: Introduced
preset
mode to run vmanomaly service with minimal user input and on widely-known metrics, like those produced bynode_exporter
. - FEATURE: Introduced
min_dev_from_expected
model common arg, aimed at reducing false positives in scenarios where deviations between the real valuey
and the expected valueyhat
are relatively high and may cause models to generate high anomaly scores. However, these deviations are not significant enough in absolute values to be considered anomalies based on domain knowledge. - FEATURE: Introduced
detection_direction
model common arg, enabling domain-driven anomaly detection strategies. Configure models to identify anomalies occurring above, below, or in both directions relative to the expected values. - FEATURE: add
n_jobs
arg toBacktestingScheduler
to allow proportionally faster (yet more resource-intensive) evaluations of a config on historical data. Default value is 1, that implies sequential execution. - FEATURE: allow anomaly detection models to be dumped to a host filesystem after
fit
stage (instead of in-memory). Resource-intensive setups (many models, many metrics, biggerfit_window
arg) and/or 3rd-party models that store fit data (like ProphetModel or HoltWinters) will have RAM consumption greatly reduced at a cost of slightly slowerinfer
stage. Please find how to enable it here - IMPROVEMENT: Reduced the resource used for each fitted
ProphetModel
by up to 6 times. This includes both RAM for in-memory models and disk space for on-disk models storage. For more details, refer to this discussion on Facebook’s Prophet. - IMPROVEMENT: now config components class can be referenced by a short alias instead of a full class path - i.e.
model.zscore.ZscoreModel
becomeszscore
,reader.vm.VmReader
becomesvm
,scheduler.periodic.PeriodicScheduler
becomesperiodic
, etc. - FIX: if using multi-scheduler setup (introduced in v1.11.0), prevent schedulers (and correspondent services) that are not attached to any model (so neither found in ‘schedulers’ arg nor left blank in
model
section) from being spawn, causing resource overhead and slight interference with existing ones. - FIX: set random seed for ProphetModel to assure uncertainty estimates (like
yhat_lower
,yhat_upper
) and dependant series (likeanomaly_score
), produced during.infer()
calls are always deterministic given the same input. See initial issue for the details. - FIX: prevent orphan queries (that are not attached to any model or scheduler) found in
queries
arg of Reader config section to be fetched from VictoriaMetrics TSDB, avoiding redundant data processing. A warning will be logged, if such queries exist in a parsed config.
v1.12.0 #
Released: 2024-03-31
- FEATURE: Introduction of
AutoTunedModel
model class to optimize any built-in model on data duringfit
phase. Specify as little asanomaly_percentage
param from(0, 0.5)
interval andtuned_model_class
(i.e.model.zscore.ZscoreModel
) to get it working with best settings that match your data. See details here.
- IMPROVEMENT: Better logging of model lifecycle (fit/infer stages).
- IMPROVEMENT: Introduce
provide_series
arg to all the built-in models to define what output fields to generate for writing (i.e.provide_series: ['anomaly_score']
means only scores are being produced) - FIX: Self-monitoring metrics are now aggregated to
queries
aliases level (not to label sets of individual timeseries) and aligned with reader, writer and model sections description , so/metrics
endpoint holds only necessary information for scraping. - FIX: Self-monitoring metric
vmanomaly_models_active
now has additional labelsmodel_alias
,scheduler_alias
,preset
to align with model-centric self-monitoring. - IMPROVEMENT: Add possibility to use temporal information in IsolationForest models via cyclical encoding. This is particularly helpful to detect multivariate seasonality-dependant anomalies.
- BREAKING CHANGE: ARIMA model is removed from built-in models. For affected users, it is suggested to replace ARIMA by Prophet or Holt-Winters.
v1.11.0 #
Released: 2024-02-22
- FEATURE: Multi-scheduler support. Now users can use multiple model specs in a single config (via aliasing), each spec can be run with its own (even multiple) schedulers.
- Introduction of
schedulers
arg in model spec:- It allows each model to be managed by 1 (or more) schedulers, so overall resource usage is optimized and flexibility is preserved.
- Passing an empty list or not specifying this param implies that each model is run in all the schedulers, which is a backward-compatible behavior.
- Please find more details in docs on Model section
- Introduction of
- DEPRECATION: slight refactor of a scheduler config section
- DEPRECATION: The
--watch
CLI option for config file reloads is deprecated and will be ignored in the future.
v1.10.0 #
Released: 2024-02-15
FEATURE: Multi-model support. Now users can specify multiple model specs in a single config (via aliasing), as well as to reference what queries from VmReader it should be run on.
- Introduction of
queries
arg in model spec:- It allows the model to be executed only on a particular query subset from
reader
section. - Passing an empty list or not specifying this param implies that each model is run on results from all queries, which is a backward-compatible behavior.
- Please find more details in docs on Model section
- It allows the model to be executed only on a particular query subset from
- Introduction of
DEPRECATION: slight refactor of a model config section
- Now models are passed as a mapping of
model_alias: model_spec
under model sections. Using old format (<= 1.9.2) will produce warnings for now and will be removed in future versions. - Please find more details in docs on Model section
- Now models are passed as a mapping of
IMPROVEMENT: now logs from
monitoring.pull
GET requests to/metrics
endpoint are shown only in DEBUG modeIMPROVEMENT: labelset for multivariate models is deduplicated and cleaned, resulting in better UX
Note: These updates support more flexible setup and effective resource management in service, as now it’s not longer needed to spawn several instances of
vmanomaly
to split queries/models context across.
v1.9.2 #
Released: 2024-01-29
- BUGFIX: now multivariate models (like
IsolationForestMultivariateModel
) are properly handled throughout fit/infer phases.
v1.9.1 #
Released: 2024-01-27
- IMPROVEMENT: Updated the offline license verification backbone to mitigate a critical vulnerability identified in the
ecdsa
library, ensuring enhanced security despite initial non-impact. - IMPROVEMENT: bump 3rd-party dependencies for Python 3.12.1
v1.9.0 #
Released: 2024-01-26
- BUGFIX: The
query_from_last_seen_timestamp
internal logic in VmReader, first introduced in v1.5.1, now functions correctly. This fix ensures that the input data shape remains consistent for subsequentfit
-based model calls in the service. - BREAKING CHANGE: The
sampling_period
parameter is now mandatory in VmReader. This change aims to clarify and standardize the frequency of input/output invmanomaly
, thereby reducing uncertainty and aligning with user expectations.
Note: The majority of users, who have been proactively specifying the
sampling_period
parameter in their configurations, will experience no disruption from this update. This transition formalizes a practice that was already prevalent and expected among our user base.
v1.8.0 #
Released: 2024-01-15
- FEATURE: Added Univariate MAD (median absolute deviation) model support.
- IMPROVEMENT: Update Python to 3.12.1 and all the dependencies.
- IMPROVEMENT: Don’t check /health endpoint, check the real /query_range or /import endpoints directly. Users kept getting problems with /health.
- DEPRECATION: “health_path” param is deprecated and doesn’t do anything in config (reader, writer, monitoring.push).
v1.7.2 #
Released: 2023-12-21
- FIX: fit/infer calls are now skipped if we have insufficient valid data to run on.
- FIX: proper handling of
inf
andNaN
in fit/infer calls. - FEATURE: add counter of skipped model runs
vmanomaly_model_runs_skipped
to healthcheck metrics. - FEATURE: add exponential retries wrapper to VmReader’s
read_metrics()
. - FEATURE: add
BacktestingScheduler
for consecutive retrospective fit/infer calls. - FEATURE: add improved & numerically stable anomaly scores.
- IMPROVEMENT: add full config validation. The probability of getting errors in later stages (say, model fit) is greatly reduced now. All the config validation errors that needs to be fixed are now a part of logging.
note: this is an backward-incompatible change, as
model
config section now expects key-value args for internal model defined in nestedargs
. - IMPROVEMENT: add explicit support of
gzip
-ed responses from vmselect in VmReader.
v1.6.0 #
Released: 2023-10-30
- IMPROVEMENT:
- now all the produced healthcheck metrics have
vmanomaly_
prefix for easier accessing. - updated docs for monitoring.
note: this is an backward-incompatible change, as metric names will be changed, resulting in new metrics creation, i.e.
model_datapoints_produced
will becomevmanomaly_model_datapoints_produced
- now all the produced healthcheck metrics have
- IMPROVEMENT: Set default value for
--log_level
fromDEBUG
toINFO
to reduce logs verbosity. - IMPROVEMENT: Add alias
--log-level
to--log_level
. - FEATURE: Added
extra_filters
parameter to reader. It allows to apply global filters to all queries. - FEATURE: Added
verify_tls
parameter to reader and writer. It allows to disable TLS verification for remote endpoint. - FEATURE: Added
bearer_token
parameter to reader and writer. It allows to pass bearer token for remote endpoint for authentication. - BUGFIX: Fixed passing
workers
parameter for reader. Previously it would throw a runtime error ifworkers
was specified.
v1.5.1 #
Released: 2023-09-18
- IMPROVEMENT: Infer from the latest seen datapoint for each query. Handles the case datapoints arrive late.
v1.5.0 #
Released: 2023-08-11
- FEATURE: add
--license
and--license-file
command-line flags for license code verification. - IMPROVEMENT: Updated Python to 3.11.4 and updated dependencies.
- IMPROVEMENT: Guide documentation for Custom Model usage.
v1.4.2 #
Released: 2023-06-09
- FIX: Fix case with received metric labels overriding generated.
v1.4.1 #
Released: 2023-06-09
- IMPROVEMENT: Update dependencies.
v1.4.0 #
Released: 2023-05-06
- FEATURE: Reworked self-monitoring grafana dashboard for vmanomaly.
- IMPROVEMENT: Update python version and dependencies.
v1.3.0 #
Released: 2023-03-21
- FEATURE: Parallelized queries. See
reader.workers
param to control parallelism. By default it’s value is equal to number of queries (sends all the queries at once). - IMPROVEMENT: Updated self-monitoring dashboard.
- IMPROVEMENT: Reverted back default bind address for /metrics server to 0.0.0.0, as vmanomaly is distributed in Docker images.
- IMPROVEMENT: Silenced Prophet INFO logs about yearly seasonality.
v1.2.2 #
Released: 2023-03-19
- FIX: Fix
for
metric label to pass QUERY_KEY. - FEATURE: Added
timeout
config param to reader, writer, monitoring.push. - FIX: Don’t hang if scheduler-model thread exits.
- FEATURE: Now reader, writer and monitoring.push will not halt the process if endpoint is inaccessible or times out, instead they will increment metrics
*_response_count{code=~"timeout|connection_error"}
.
v1.2.1 #
Released: 2023-02-18
- FIX: Fixed scheduler thread starting.
- FIX: Fix rolling model fit+infer.
- BREAKING CHANGE: monitoring.pull server now binds by default on 127.0.0.1 instead of 0.0.0.0. Please specify explicitly in monitoring.pull.addr what IP address it should bind to for serving /metrics.
v1.2.0 #
Released: 2023-02-04
- FEATURE: With arg
--watch
watches for config(s) changes and reloads the service automatically. - IMPROVEMENT: Remove “provide_series” from HoltWinters model. Only Prophet model now has it, because it may produce a lot of series if “holidays” is on.
- IMPROVEMENT: if Prophet’s “provide_series” is omitted, then all series are returned.
- DEPRECATION: Config monitoring.endpoint_url is deprecated in favor of monitoring.url.
- DEPRECATION: Remove ’enable’ param from config monitoring.pull. Now /metrics server is started whenever monitoring.pull is present.
- IMPROVEMENT: include example configs into the docker image at /vmanomaly/config/*
- IMPROVEMENT: include self-monitoring grafana dashboard into the docker image under /vmanomaly/dashboard/vmanomaly_grafana_dashboard.json
v1.1.0 #
Released: 2023-01-23
- IMPROVEMENT: update Python dependencies
- FEATURE: Add multivariate IsolationForest model.
v1.0.1 #
Released: 2023-01-06
- FIX: prophet model incorrectly predicted two points in case of only one
v1.0.0-beta #
Released: 2022-12-08
- First public release is available