Reader #

VictoriaMetrics Anomaly Detection (vmanomaly) primarily uses VmReader to ingest data. This reader focuses on fetching time-series data directly from VictoriaMetrics with the help of powerful MetricsQL expressions for aggregating, filtering and grouping your data, ensuring seamless integration and efficient data handling.

Future updates will introduce additional readers, expanding the range of data sources vmanomaly can work with.

VM reader #

Note: There is backward-compatible change Available from v1.13.0 of queries arg of VmReader. New format allows to specify per-query parameters, like step to reduce amount of data read from VictoriaMetrics TSDB and to allow config flexibility. Please see per-query parameters section for the details.

Old format like

# other config sections ...
reader:
  class: 'vm'
  datasource_url: 'http://localhost:8428'  # source victoriametrics/prometheus
  sampling_period: "10s"  # set it <= min(infer_every) in schedulers section
  queries:
    # old format {query_alias: query_expr}, prior to 1.13, will be converted to a new format automatically
    vmb: 'avg(vm_blocks)'

will be converted to a new one with a warning raised in logs:

# other config sections ...
reader:
  class: 'vm'
  datasource_url: 'http://localhost:8428'  # source victoriametrics/prometheus
  sampling_period: '10s'
  queries:
    # old format {query_alias: query_expr}, prior to 1.13, will be converted to a new format automatically
    vmb:
      expr: 'avg(vm_blocks)'  # initial MetricsQL expression
      step: '10s'  # individual step for this query, will be filled with `sampling_period` from the root level
      data_range: ['-inf', 'inf']  # by default, no constraints applied on data range
      tz: 'UTC'  # by default, tz-free data is used throughout the model lifecycle
      # new query-level arguments will be added in backward-compatible way in future releases

Per-query parameters #

There is change Available from v1.13.0 of queries arg format. Now each query alias supports the next (sub)fields:

• expr (string): MetricsQL/PromQL expression that defines an input for VmReader. As accepted by /query_range?query=%s. i.e. avg(vm_blocks)

• step (string): query-level frequency of the points returned, i.e. 30s. Will be converted to /query_range?step=%s param (in seconds). Useful to optimize total amount of data read from VictoriaMetrics, where different queries may have different frequencies for different machine learning models to run on.

  Note: if not set explicitly (or if older config style prior to v1.13.0) is used, then it is set to reader-level sampling_period arg.

  Note: having different individual step args for queries (i.e. 30s for q1 and 2m for q2) is not yet supported for multivariate model if you want to run it on several queries simultaneously (i.e. setting queries arg of a model to [q1, q2]).

• data_range Available from v1.15.1 (list[float | string]): It allows defining valid data ranges for input per individual query in queries, resulting in:

  • High anomaly scores (>1) when the data falls outside the expected range, indicating a data constraint violation.
  • Lowest anomaly scores (=0) when the model’s predictions (yhat) fall outside the expected range, meaning uncertain predictions.

  Note: if not set explicitly (or if older config style prior to v1.13.0) is used, then it is set to reader-level data_range arg Available from v1.18.1

• max_points_per_query (int): Optional arg Available from v1.17.0 overrides how search.maxPointsPerTimeseries flag Available from v1.14.1 impacts vmanomaly on splitting long fit_window queries into smaller sub-intervals. This helps users avoid hitting the search.maxQueryDuration limit for individual queries by distributing initial query across multiple subquery requests with minimal overhead. Set less than search.maxPointsPerTimeseries if hitting maxQueryDuration limits. If set on a query-level, it overrides the global max_points_per_query (reader-level).

• tz Available from v1.18.0 (string): this optional argument enables timezone specification per query, overriding the reader’s default tz. This setting helps to account for local timezone shifts, such as DST, in models that are sensitive to seasonal variations (e.g., ProphetModel or OnlineQuantileModel).

Per-query config example #

reader:
  class: 'vm'
  sampling_period: '1m'
  max_points_per_query: 10000
  # other reader params ...
  queries:
    ingestion_rate:
      expr: 'sum(rate(vm_rows_inserted_total[5m])) by (type) > 0'
      step: '2m'  # overrides global `sampling_period` of 1m
      data_range: [10, 'inf']  # meaning only positive values > 10 are expected, i.e. a value `y` < 10 will trigger anomaly score > 1
      max_points_per_query: 5000 # overrides reader-level value of 10000 for `ingestion_rate` query
      tz: 'America/New_York'  # to override reader-wise `tz`

Config parameters #

Config file example:

reader:
  class: "vm"  # or "reader.vm.VmReader" until v1.13.0
  datasource_url: "https://play.victoriametrics.com/"
  tenant_id: "0:0"
  tz: 'America/New_York'
  data_range: [1, 'inf']  # reader-level
  queries:
    ingestion_rate:
      expr: 'sum(rate(vm_rows_inserted_total[5m])) by (type) > 0'
      step: '1m' # can override reader-level `sampling_period` on per-query level
      data_range: [0, 'inf']  # if set, overrides reader-level data_range
      tz: 'Australia/Sydney'  # if set, overrides reader-level tz
  sampling_period: '1m'
  query_from_last_seen_timestamp: True  # false by default
  latency_offset: '1ms'

mTLS protection #

vmanomaly supports mutual TLS (mTLS) Available from v1.16.3 for secure communication across its components, including VmReader, VmWriter, and Monitoring/Push. This allows for mutual authentication between the client and server when querying or writing data to VictoriaMetrics Enterprise, configured for mTLS.

mTLS ensures that both the client and server verify each other’s identity using certificates, which enhances security by preventing unauthorized access.

To configure mTLS, the following parameters can be set in the config:

• verify_tls: If set to a string, it functions like the -mtlsCAFile command-line argument of VictoriaMetrics, specifying the CA bundle to use. Set to True to use the system’s default certificate store.
• tls_cert_file: Specifies the path to the client certificate, analogous to the -tlsCertFile argument of VictoriaMetrics.
• tls_key_file: Specifies the path to the client certificate key, similar to the -tlsKeyFile argument of VictoriaMetrics.

These options allow you to securely interact with mTLS-enabled VictoriaMetrics endpoints.

Example configuration to enable mTLS with custom certificates:

reader:
  class: "vm"
  datasource_url: "https://your-victoriametrics-instance-with-mtls"
  # tenant_id: "0:0" uncomment and set for cluster version
  queries:
    vm_blocks_example:
      expr: 'avg(rate(vm_blocks[5m]))'
      step: 30s
  sampling_period: 30s
  verify_tls: "path/to/ca.crt"  # path to CA bundle for TLS verification
  tls_cert_file: "path/to/client.crt"  # path to the client certificate
  tls_key_file:  "path/to/client.key"  # path to the client certificate key
  # additional reader parameters ...

# other config sections, like models, schedulers, writer, ...

Healthcheck metrics #

VmReader exposes several healthchecks metrics.