For exporting data, VictoriaMetrics Anomaly Detection (vmanomaly) primarily employs the VmWriter, which writes produced anomaly scores (preserving initial labelset and optionally applying additional ones) back to VictoriaMetrics. This writer is tailored for smooth data export within the VictoriaMetrics ecosystem.

Future updates will introduce additional export methods, offering users more flexibility in data handling and integration.

VM writer #

Config parameters #


ParameterExampleDescription

class

writer.vm.VmWriter or vm starting from v1.13.0

Name of the class needed to enable writing to VictoriaMetrics or Prometheus. VmWriter is the default option, if not specified.

datasource_url

http://localhost:8481/

Datasource URL address

tenant_id

0:0

For VictoriaMetrics Cluster version only, tenants are identified by accountID or accountID:projectID. See VictoriaMetrics Cluster multitenancy docs

metric_format

__name__: "vmanomaly_$VAR"

Metrics to save the output (in metric names or labels). Must have __name__ key. Must have a value with $VAR placeholder in it to distinguish between resulting metrics. Supported placeholders:

  • $VAR – Variables that model provides, all models provide the following set: {“anomaly_score”, “y”, “yhat”, “yhat_lower”, “yhat_upper”}. Description of standard output is here. Depending on model type it can provide more metrics, like “trend”, “seasonality” etc.

  • $QUERY_KEY – E.g. “ingestion_rate”.

Other keys are supposed to be configured by the user to help identify generated metrics, e.g., specific config file name etc. More details on metric formatting are here.

for: "$QUERY_KEY"

run: "test_metric_format"

config: "io_vm_single.yaml"

import_json_path

/api/v1/import

Optional, to override the default import path

health_path

/health

Absolute or relative URL address where to check the availability of the datasource. Optional, to override the default /health path.

user

USERNAME

BasicAuth username

password

PASSWORD

BasicAuth password

timeout

5s

Timeout for the requests, passed as a string

verify_tls

false

Allows disabling TLS verification of the remote certificate.

bearer_tokentokenToken is passed in the standard format with header: Authorization: bearer {token}
bearer_token_filepath_to_filePath to a file, which contains token, that is passed in the standard format with header: Authorization: bearer {token}. Available since v1.15.9

Config example:

writer:
  class: "vm"  # or "writer.vm.VmWriter" until v1.13.0
  datasource_url: "http://localhost:8428/"
  tenant_id: "0:0"
  metric_format:
    __name__: "vmanomaly_$VAR"
    for: "$QUERY_KEY"
    run: "test_metric_format"
    config: "io_vm_single.yaml"
  import_json_path: "/api/v1/import"
  health_path: "health"
  user: "foo"
  password: "bar"

Healthcheck metrics #

VmWriter exposes several healthchecks metrics.

Metrics formatting #

There should be 2 mandatory parameters set in metric_format - __name__ and for.

__name__: PREFIX1_$VAR
for: PREFIX2_$QUERY_KEY
  • for __name__ parameter it will name metrics returned by models as PREFIX1_anomaly_score, PREFIX1_yhat_lower, etc. Vmanomaly output metrics names described here
  • for for parameter will add labels PREFIX2_query_name_1, PREFIX2_query_name_2, etc. Query names are set as aliases in config reader section in queries parameter.

It is possible to specify other custom label names needed. For example:

custom_label_1: label_name_1
custom_label_2: label_name_2

Apart from specified labels, output metrics will return labels inherited from input metrics returned by queries. For example if input data contains labels such as cpu=1, device=eth0, instance=node-exporter:9100 all these labels will be present in vmanomaly output metrics.

So if metric_format section was set up like this:

metric_format:
    __name__: "PREFIX1_$VAR"
    for: "PREFIX2_$QUERY_KEY"
    custom_label_1: label_name_1
    custom_label_2: label_name_2

It will return metrics that will look like:

{__name__="PREFIX1_anomaly_score", for="PREFIX2_query_name_1", custom_label_1="label_name_1", custom_label_2="label_name_2", cpu=1, device="eth0", instance="node-exporter:9100"}
{__name__="PREFIX1_yhat_lower", for="PREFIX2_query_name_1", custom_label_1="label_name_1", custom_label_2="label_name_2", cpu=1, device="eth0", instance="node-exporter:9100"}
{__name__="PREFIX1_anomaly_score", for="PREFIX2_query_name_2", custom_label_1="label_name_1", custom_label_2="label_name_2", cpu=1, device="eth0", instance="node-exporter:9100"}
{__name__="PREFIX1_yhat_lower", for="PREFIX2_query_name_2", custom_label_1="label_name_1", custom_label_2="label_name_2", cpu=1, device="eth0", instance="node-exporter:9100"}