Monitoring

There are 2 models to monitor VictoriaMetrics Anomaly Detection behavior - push and pull. Parameters for each of them should be specified in the config file, monitoring section.

Pull Model Config parameters #

Parameter Default Description

Parameter	Default	Description
`addr`	`"0.0.0.0"`	Server IP Address
`port`	`8080`	Port

addr

"0.0.0.0"

Server IP Address

port

8080

Port

Push Config parameters #

Parameter	Default	Description
`url`		Link where to push metrics to. Example: `"http://localhost:8480/"`
`tenant_id`		Tenant ID for cluster version. Example: `"0:0"`
`health_path`	`"health"`	Deprecated since v1.8.0. Absolute, to override `/health` path
`user`		BasicAuth username
`password`		BasicAuth password
`verify_tls`	`False`	Allows disabling TLS verification of the remote certificate.
`timeout`	`"5s"`	Stop waiting for a response after a given number of seconds.
`extra_labels`		Section for custom labels specified by user.

Monitoring section config example #

monitoring:
  pull: # Enable /metrics endpoint.
    addr: "0.0.0.0"
    port: 8080
  push:
    url: "http://localhost:8480/"
    tenant_id: "0:0" # For cluster version only
    user: "USERNAME"
    password: "PASSWORD"
    verify_tls: False
    timeout: "5s"
    extra_labels:
      job: "vmanomaly-push"
      test: "test-1"

Metrics generated by vmanomaly #

Metric	Type	Description
`vmanomaly_start_time_seconds`	Gauge	vmanomaly start time in UNIX time

Models Behaviour Metrics #

Label names description

Note: There is a new label key model_alias introduced in multi-model support v1.10.0. This label key adjustment was made to preserve unique label set production during writing produced metrics back to VictoriaMetrics.

Metric	Type	Description	Labelnames
`vmanomaly_model_runs`	Counter	How many times models ran (per model)	`stage, query_key, model_alias, scheduler_alias, preset`
`vmanomaly_model_run_duration_seconds`	Summary	How much time (in seconds) model invocations took	`stage, query_key, model_alias, scheduler_alias, preset`
`vmanomaly_model_datapoints_accepted`	Counter	How many datapoints did models accept	`stage, query_key, model_alias, scheduler_alias, preset`
`vmanomaly_model_datapoints_produced`	Counter	How many datapoints were generated by models	`stage, query_key, model_alias, scheduler_alias, preset`
`vmanomaly_models_active`	Gauge	How many models are currently inferring	`query_key, model_alias, scheduler_alias, preset`
`vmanomaly_model_runs_skipped`	Counter	How many times a run was skipped (per model)	`stage, query_key, model_alias, scheduler_alias, preset`

Writer Behaviour Metrics #

Label names description

Metric	Type	Description	Labelnames
`vmanomaly_writer_request_duration_seconds`	Summary	How much time (in seconds) did requests to VictoriaMetrics take	`url, query_key`
`vmanomaly_writer_response_count`	Counter	Response code counts we got from VictoriaMetrics	`url, query_key, code`
`vmanomaly_writer_sent_bytes`	Counter	How much bytes were sent to VictoriaMetrics	`url, query_key`
`vmanomaly_writer_request_serialize_seconds`	Summary	How much time (in seconds) did serializing take	`query_key`
`vmanomaly_writer_datapoints_sent`	Counter	How many datapoints were sent to VictoriaMetrics	`query_key`
`vmanomaly_writer_timeseries_sent`	Counter	How many timeseries were sent to VictoriaMetrics	`query_key`

Reader Behaviour Metrics #

Label names description

Metric	Type	Description	Labelnames
`vmanomaly_reader_request_duration_seconds`	Summary	How much time (in seconds) did queries to VictoriaMetrics take	`url, query_key`
`vmanomaly_reader_response_count`	Counter	Response code counts we got from VictoriaMetrics	`url, query_key, code`
`vmanomaly_reader_received_bytes`	Counter	How much bytes were received in responses	`query_key`
`vmanomaly_reader_response_parsing_seconds`	Summary	How much time (in seconds) did parsing take for each step	`step`
`vmanomaly_reader_timeseries_received`	Counter	How many timeseries were received from VictoriaMetrics	`query_key`
`vmanomaly_reader_datapoints_received`	Counter	How many rows were received from VictoriaMetrics	`query_key`

Labelnames #

stage - stage of model - ‘fit’, ‘infer’ or ‘fit_infer’ for models that do it simultaneously, see model types.
query_key - query alias from reader config section.
model_alias - model alias from models config section. Introduced in v1.10.0.
scheduler_alias - scheduler alias from schedulers config section. Introduced in v1.11.0.
preset - preset alias for forthcoming preset section compatibility. Introduced in v1.12.0.
url - writer or reader url endpoint.
code - response status code or connection_error, timeout.
step - json or dataframe reading step.