Reader #

VictoriaMetrics Anomaly Detection (vmanomaly) primarily uses VmReader to ingest data. This reader focuses on fetching time-series data directly from VictoriaMetrics with the help of powerful MetricsQL expressions for aggregating, filtering and grouping your data, ensuring seamless integration and efficient data handling.

Future updates will introduce additional readers, expanding the range of data sources vmanomaly can work with.

VM reader #

Note: Starting from v1.13.0 there is backward-compatible change of queries arg of VmReader. New format allows to specify per-query parameters, like step to reduce amount of data read from VictoriaMetrics TSDB and to allow config flexibility. Please see per-query parameters section for the details.

Old format like

# other config sections ...
reader:
  class: 'vm'
  datasource_url: 'http://localhost:8428'  # source victoriametrics/prometheus
  sampling_period: "10s"  # set it <= min(infer_every) in schedulers section
  queries:
    # old format {query_alias: query_expr}, prior to 1.13, will be converted to a new format automatically
    vmb: 'avg(vm_blocks)'

will be converted to a new one with a warning raised in logs:

# other config sections ...
reader:
  class: 'vm'
  datasource_url: 'http://localhost:8428'  # source victoriametrics/prometheus
  sampling_period: '10s'
  queries:
    # old format {query_alias: query_expr}, prior to 1.13, will be converted to a new format automatically
    vmb:
      expr: 'avg(vm_blocks)'  # initial MetricsQL expression
      step: '10s'  # individual step for this query, will be filled with `sampling_period` from the root level
      data_range: ['-inf', 'inf']  # by default, no constraints applied on data range
      # new query-level arguments will be added in backward-compatible way in future releases

Per-query parameters #

Starting from v1.13.0 there is change of queries arg format. Now each query alias supports the next (sub)fields:

• expr (string): MetricsQL/PromQL expression that defines an input for VmReader. As accepted by /query_range?query=%s. i.e. avg(vm_blocks)

• step (string): query-level frequency of the points returned, i.e. 30s. Will be converted to /query_range?step=%s param (in seconds). Useful to optimize total amount of data read from VictoriaMetrics, where different queries may have different frequencies for different machine learning models to run on.

  Note: if not set explicitly (or if older config style prior to v1.13.0) is used, then it is set to reader-level sampling_period arg.

  Note: having different individual step args for queries (i.e. 30s for q1 and 2m for q2) is not yet supported for multivariate model if you want to run it on several queries simultaneously (i.e. setting queries arg of a model to [q1, q2]).

• data_range (list[float | string]): Introduced in v1.15.1, it allows defining valid data ranges for input per individual query in queries, resulting in:

  • High anomaly scores (>1) when the data falls outside the expected range, indicating a data constraint violation.
  • Lowest anomaly scores (=0) when the model’s predictions (yhat) fall outside the expected range, meaning uncertain predictions.

Per-query config example #

reader:
  class: 'vm'
  sampling_period: '1m'
  # other reader params ...
  queries:
    ingestion_rate:
      expr: 'sum(rate(vm_rows_inserted_total[5m])) by (type) > 0'
      step: '2m'  # overrides global `sampling_period` of 1m
      data_range: [10, 'inf']  # meaning only positive values > 10 are expected, i.e. a value `y` < 10 will trigger anomaly score > 1

Config parameters #

Config file example:

reader:
  class: "vm"  # or "reader.vm.VmReader" until v1.13.0
  datasource_url: "https://play.victoriametrics.com/"
  tenant_id: "0:0"
  queries:
    ingestion_rate:
      expr: 'sum(rate(vm_rows_inserted_total[5m])) by (type) > 0'
      step: '1m' # can override global `sampling_period` on per-query level
      data_range: [0, 'inf']
  sampling_period: '1m'
  query_from_last_seen_timestamp: True  # false by default
  latency_offset: '1ms'

Healthcheck metrics #

VmReader exposes several healthchecks metrics.