Migration

Introduction #

This document provides guidelines for migrating to the latest version of VictoriaMetrics Anomaly Detection (vmanomaly). It covers the key changes, compatibility considerations, and best practices to ensure a smooth transition for stateful and stateless modes of operation.

Upgrading to v1.27.1 or newer is recommended to benefit from simplified migration process.

Dry Run #

The --dryRun command-line argument allows Available from v1.27.0 to simulate the migration process without making any actual changes. This is useful for identifying potential issues and understanding the impact of the migration before applying it, e.g. dropping of existing state database or on-disk artifacts for all (or some) of the configured models and data. Starting from version v1.27.0 , the upgrade impact to any new version can be assessed by running vmanomaly with the --dryRun flag automatically. Downgrade check from v1.27.0 (or newer) to earlier versions than v1.25.3 requires setting env variable VMANOMALY_STATE_VERSION_OVERRIDE=<version>, e.g.:

      export VMANOMALY_STATE_VERSION_OVERRIDE=1.25.2
    

Compatibility Matrix #

This section outlines the compatibility of different vmanomaly versions with various components, including data, models, and configuration formats, for both stateful and stateless modes.

Refer to the global changelog for detailed information on changes in each version. Use --dryRun Available from v1.27.0 mode to check for compatibility issues before performing the actual migration. See Dry Run section for more details.

Stateful Mode #

Used if settings.restore_state is set to true. See argument details in the configuration documentation .

There are 2 types of compatibility to consider when migrating in stateful mode:

Global (in)compatibility: The new version can seamlessly read and utilize the existing state without any modifications or data loss. Or, in case of incompatibility, the existing state must be dropped completely to proceed with the migration.
Component (in)compatibility: The new version may introduce changes that affect specific components (e.g., specific models, data formats) but can still operate with the existing state with some adjustments or drop of incompatible on disk artifacts.

Compatibility classifications and automated --dryRun or /api/v1/compatibility results cover vmanomaly-managed state formats, supported readers, and built-in models . They do not validate custom model implementation or serialized dependencies, so compatibility is not guaranteed for custom models , even when the automated check reports is_compatible: true.

When upgrading from v1.30.2 or earlier to v1.30.3 or newer, custom many-to-one models that rely only on is_multivariate = True must declare topology = ModelTopology.MANY_TO_ONE before upgrading. See the custom model topology contract .

Compatibility checks are directional and use the rules bundled with the running vmanomaly version. When planning a rollback, run --dryRun or /api/v1/compatibility from the current, newer binary because an older target binary cannot discover rules added after it was released.

Source state version(s)	Target runtime version(s)	Compatibility	Notes
v1.30.4	v1.30.3 or earlier	Fully Incompatible	Runtimes released before v1.30.4 cannot recognize v1.30.4 state provenance and drop persisted state, although the v1.30.3 and v1.30.4 managed state formats are otherwise compatible. Clear the state and refit when rolling back; do not override the recorded state version.
v1.30.3	v1.30.2 or earlier	Partially Compatible	Discard and refit univariate and multivariate Temporal Envelope model dumps. Other compatible managed state remains reusable.
v1.30.0 - v1.30.2	v1.30.4 or newer	Fully Compatible	No cleanup of vmanomaly-managed built-in state is required; v1.30.4 safely restores existing multivariate Temporal Envelope checkpoints.
v1.30.3	v1.30.4 and newer	Fully Compatible	Built-in reader, data, and model state is reusable. v1.30.4 also records updated state provenance for reliable future migration checks.
v1.30.0 - v1.30.2	v1.30.3	Partially Compatible	Restored multivariate Temporal Envelope checkpoints can fail during inference. Upgrade directly to v1.30.4 or newer.
v1.29.1	v1.30.2	Fully Compatible	v1.30.0 adds Temporal Envelope state without changing existing built-in model and data artifacts.
v1.28.7	v1.29.0	Partially compatible*	Dumped models of class prophet and seasonal quantile have problems with loading to v1.29.0 due to dropped `pytz` library. Upgrading directly from v1.28.7 to v1.29.1 with a fix is suggested
v1.26.0	v1.28.7	Fully Compatible	v1.28.0 introduced rolling model class drop in favor of online models (`rolling_quantile` and `std` models), however, it does not impact compatibility, as artifacts were not produced by default for rolling models. Also, offline `mad` and `zscore` models are redirecting to their respective online counterparts since v1.28.4 .
v1.25.3	v1.26.0	Partially Compatible*	v1.25.3 introduced `forecast_at` argument for base univariate and `Prophet` models , however, itself remains backward-reversible from newer states like v1.26.2 , v1.27.0 . (All models except `isolation_forest_multivariate` class will be dropped)
v1.25.1	v1.25.2	Fully Compatible	In v1.25.1 there was a change to `vmanomaly.db` metadata database format, so migrating from v1.24.0-v1.25.0 requires deletion of a state, see note above the table
v1.24.1	v1.25.0	Partially Compatible*	In v1.25.0 there were changes to data dump layout and to `online_quantile` and `isolation_forest_multivariate` model states, so to migrate from v1.24.0-v1.24.1 it is recommended to drop the state
v1.24.0	v1.24.1	Fully Compatible	-
v1.23.3 and earlier	v1.24.0	Fully Incompatible*	*As no state (prior to v1.24.0 ) existed, it was not saved (even if on-disk mode was used). Also, see config breaking changes list in stateless mode

Clearing State #

For releases v1.27.0 and newer, the migration process is automatically handled by vmanomaly when started with settings.restore_state: true, so no manual intervention is required to clear existing state if incompatible.

However, for releases v1.24.0 - v1.26.2 , to clear the existing state (if ended with settings.restore_state: true), please manually delete the existing state database and on-disk artifacts before starting the new version of `vmanomaly - either:

Manually delete the content of VMANOMALY_MODEL_DUMPS_DIR / VMANOMALY_DATA_DUMPS_DIR folders or
Set settings.restore_state: false in the config the first run of the new version, then stop vmanomaly, set back settings.restore_state: true, and restart vmanomaly.

Stateless Mode #

Used if settings.restore_state is set to false. See argument details in the configuration documentation .

In stateless mode, the migration process is almost straightforward as there are no persistent states to manage. One may simply upgrade the vmanomaly service to the latest version and restart it, up to a slight change in the config .YAML files for backward-incompatible changes, see the list below.

Breaking Changes

v1.12.0 ARIMA model is removed from built-in models . Action: for vmanomaly v1.30.0 and newer, replace ARIMA with Temporal Envelope Available from v1.30.0 ; for older releases, use another supported seasonal model in the models section of the configuration.
v1.9.0 The sampling_period parameter is now mandatory in VmReader. This change aims to clarify and standardize the frequency of input/output in vmanomaly, thereby reducing uncertainty and aligning with user expectations; Action: Add the sampling_period parameter to your VmReader configuration, e.g.:
```
      reader:
  # Other VmReader settings...
  sampling_period: 1m
  ...
    
```

Previous Self-monitoring Next FAQ

Migration #