Anomaly detection #

Static alerts are the right choice when a value is always unacceptable or an invariant must never be broken. Anomaly detection complements them when expected behavior depends on time, workload, trend, seasonality, scale, or relationships between signals.

vmanomaly learns expected data behavior and writes a unified anomaly score to VictoriaMetrics for contextual alerting and investigation. This reduces the engineering effort spent maintaining raw-value thresholds and gives vmalert a stable, explicit decision boundary for contextual anomalies.

vmanomaly helps teams:

• reduce alert engineering and review effort across differently scaled signals;
• detect meaningful deviations in trending, seasonal, changing, or interconnected data;
• learn incrementally with online models from a stream instead of repeatedly reading and refitting large history windows;
• quickly progress from a query to a validated configuration and starting alerting rules with the UI and available AI assistance.

How it fits into VictoriaStack #

vmanomaly runs alongside the VictoriaStack databases:

flowchart LR
    subgraph Sources
        direction LR
        VM["VictoriaMetrics"]
        VL["VictoriaLogs"]
        VT["VictoriaTraces"]
    end

    A["vmanomaly<br/><hr/>learn • score • update"]

    VM -->|"MetricsQL queries"| A
    VL -->|"LogsQL stats queries"| A
    VT -->|"LogsQL stats queries"| A

    A -->|"anomaly scores as metrics"| W["VictoriaMetrics"]
    A -.->|"optional predictions for context"| W
    W --> AL["vmalert"]
    W --> G["Grafana"]

Metrics are queried directly from VictoriaMetrics. Logs and traces are analyzed as numeric time series returned by LogsQL stats queries , such as error rates or span-duration aggregates.

The results are ordinary metrics in VictoriaMetrics, preserving all input labels and values. Existing tools can therefore evaluate alerting rules and visualize the observed values, expected behavior, and anomaly scores without a separate anomaly-specific backend.

One operational signal #

anomaly_score provides a consistent operational signal across models :

• 0 means the observation matches the model expectation;
• values up to 1 remain within the expected range;
• values above 1 cross the default anomaly boundary.

The score is a continuous, de-scaled, de-trended and de-seasonalized value with an interpretable decision boundary at 1.

A rule such as anomaly_score > 1 can therefore be reused across CPU, latency, traffic, and business signals rather than maintaining a different raw-value threshold for each one.

The boundary at 1 is a stable starting point, not a substitute for operational validation. Teams can raise the alert threshold, require a duration, or aggregate scores to match the sensitivity and notification policy of a service.

Univariate models produce scores for individual input series. Multivariate models can instead produce one joint score for a group of related series. Depending on the selected model and output configuration , outputs can include the actual observed value (y), expected value (yhat), and prediction interval (yhat_lower and yhat_upper) for easier investigation and visualization.

Online-first learning #

Most built-in models support online learning. Their operating cycle is:

1. Bootstrap from enough historical behavior to establish an initial profile.
2. Score and update incrementally as new samples arrive.
3. Persist and restore the learned state so a restart does not require repeatedly querying and refitting the same history.

This spreads computation and data transfer over time and lets models adapt to new patterns without retraining from scratch. See the online model lifecycle and state restoration for configuration details.

Add domain knowledge #

A statistically unusual value is not always operationally important. Stable business policies can be attached to each input query so they apply consistently when models change:

For example, an error-rate query can reject negative values, detect only upward deviations, and ignore small percentage changes before they become anomaly alerts:

      reader:
  class: vm
  datasource_url: http://victoriametrics:8428
  sampling_period: 1m
  queries:
    error_rate:
      expr: 'sum(rate(http_requests_total{status=~"5.."}[5m]))'
      data_range: [0, 'inf']
      detection_direction: above_expected
      min_rel_dev_from_expected: [0, 20]
    

These controls encode the difference between mathematical deviation and business impact, reducing false positives and keeping detected profiles aligned with operational expectations. See all per-query parameters .

Get started #

For predefined metric sets, presets provide a configuration, Grafana dashboard, and starting alerting rules. A minimal node-exporter setup specifies only the reader and writer endpoints:

      preset: node-exporter
reader:
  datasource_url: http://victoriametrics:8428
writer:
  datasource_url: http://victoriametrics:8428
    

For a custom signal, use the built-in UI to explore historical behavior, compare models, and backtest the resulting configuration. AI Copilot , vmanomaly MCP server , and VictoriaMetrics agent skills can profile a query, suggest and fine-tune suitable models, and help produce a validated configuration plus starting alerting rules.

Then use the standard VictoriaStack workflow:

1. Write anomaly_score and optional prediction outputs to VictoriaMetrics.
2. Alert on the unified signal, starting with anomaly_score > 1.
3. Investigate alerts using generic anomaly score dashboard .
4. Tune duration, grouping, business filters, and routing as needed.

Follow the quick start for installation and the vmalert integration guide for a complete example.

Deployment and operations #

vmanomaly supports the same deployment paths as other VictoriaStack components:

• Docker ;
• VictoriaMetrics Anomaly Helm chart ;
• VMAnomaly custom resource in VictoriaMetrics Operator.

For larger workloads, vmanomaly supports horizontal scalability through sharding and high availability through replication of validated sub-configurations across instances.

See Deployment for the overall stack architecture. For production operation, see scaling and high availability , self-monitoring , and multitenancy .