Monitoring #

There are 2 models to monitor VictoriaMetrics Anomaly Detection behavior - push and pull. Parameters for each of them should be specified in the config file, monitoring section.

Note: there was an enhancement of self-monitoring metrics for consistency across the components (v.1.17.0). Documentation was updated accordingly. Key changes included:

• Converting several self-monitoring metrics from Summary to Histogram to enable quantile calculation. This addresses the limitation of the prometheus_client’s Summary implementation, which does not support quantiles. The change ensures metrics are more informative for performance analysis. Affected metrics are:
  • vmanomaly_reader_request_duration_seconds (VmReader)
  • vmanomaly_reader_response_parsing_seconds (VmReader)
  • vmanomaly_writer_request_duration_seconds (VmWriter)
  • vmanomaly_writer_request_serialize_seconds (VmWriter)
• Adding a query_key label to the vmanomaly_reader_response_parsing_seconds metric to provide finer granularity in tracking the performance of individual queries. This metric has also been switched from Summary to Histogram to align with the other metrics and support quantile calculations.
• Adding preset and scheduler_alias keys to VmReader and VmWriter metrics for consistency in multi-scheduler setups.
• Renaming Counters vmanomaly_reader_response_count to vmanomaly_reader_responses and vmanomaly_writer_response_count to vmanomaly_writer_responses.

Pull Model Config parameters #

Push Config parameters #

Monitoring section config example #

monitoring:
  pull: # Enable /metrics endpoint.
    addr: "0.0.0.0"
    port: 8080
  push:
    url: "http://localhost:8480/"
    tenant_id: "0:0" # For cluster version only
    user: "USERNAME"
    password: "PASSWORD"
    verify_tls: False
    timeout: "5s"
    extra_labels:
      job: "vmanomaly-push"
      test: "test-1"

mTLS protection #

Starting from v1.16.3, vmanomaly components such as VmWriter support mTLS to ensure secure communication with VictoriaMetrics Enterprise, configured with mTLS.

For detailed guidance on configuring mTLS parameters such as verify_tls, tls_cert_file, and tls_key_file, please refer to the mTLS protection section in the Reader documentation. The configuration principles apply consistently across all these vmanomaly components.

Metrics generated by vmanomaly #

• Startup metrics
• Reader metrics
• Model metrics
• Writer metrics

Startup metrics #

Back to metric sections

Reader behaviour metrics #

Label names description

Note: additional labels (scheduler_alias, preset) were added to writer and reader metrics in v1.17.0 to improve consistency across the components. Also, metrics vmanomaly_reader_request_duration_seconds and vmanomaly_reader_response_parsing_seconds changed their type to Histogram (was Summary prior to v1.17.0).

Back to metric sections

Models behaviour metrics #

Label names description

Note: There is a new label key model_alias introduced in multi-model support v1.10.0. This label key adjustment was made to preserve unique label set production during writing produced metrics back to VictoriaMetrics.

Note: as a part of self-monitoring metrics enhancement (v.1.17.0), new metrics, like vmanomaly_model_run_errors, was added. Some of them changed the type (Summary -> Histogram), like vmanomaly_model_run_duration_seconds.

Back to metric sections

Writer behaviour metrics #

Label names description

Note: additional labels (scheduler_alias, preset) were added to writer and reader metrics in v1.17.0 to improve consistency across the components. Also, metrics vmanomaly_writer_request_duration_seconds and vmanomaly_writer_request_serialize_seconds changed their type to Histogram (was Summary prior to v1.17.0).

Back to metric sections

Labelnames #

• stage - stage of model - ‘fit’, ‘infer’ or ‘fit_infer’ for models that do it simultaneously, see model types.
• query_key - query alias from reader config section.
• model_alias - model alias from models config section. Introduced in v1.10.0.
• scheduler_alias - scheduler alias from schedulers config section. Introduced in v1.11.0.
• preset - preset alias for preset mode of vmanomaly. Introduced in v1.12.0.
• url - writer or reader url endpoint.
• code - response status code or connection_error, timeout.
• step - json or dataframe reading step.

Back to metric sections

Logs generated by vmanomaly #

The vmanomaly service logs operations, errors, and performance for its components (service, reader, writer), alongside self-monitoring metrics updates. Below is a description of key logs for each component and the related metrics affected.

{{X}} indicates a placeholder in the log message templates described below, which will be replaced with the appropriate entity during logging.

Note: Applicable to version v1.17.1 or newer.

Note: By default, vmanomaly uses the INFO logging level. You can change this by specifying the --loggerLevel argument. See command-line arguments here.

• Startup logs
• Reader logs
• Service logs
• Writer logs

Startup logs #

The vmanomaly service logs important information during the startup process. This includes checking for the license, validating configurations, and setting up schedulers, readers, and writers. Below are key logs that are generated during startup, which can help troubleshoot issues with the service’s initial configuration or license validation.

License check. If no license key or file is provided, the service will fail to start and log an error message. If a license file is provided but cannot be read, the service logs a failure. Log messages:

Please provide a license code using --license or --licenseFile arg, or as VM_LICENSE_FILE env. See https://victoriametrics.com/products/enterprise/trial/ to obtain a trial license.

failed to read file {{args.license_file}}: {{error_message}}

Config validation. If the service’s configuration fails to load or does not meet validation requirements, an error message is logged and the service will exit. If the configuration is loaded successfully, a message confirming the successful load is logged. Log messages:

Config validation failed, please fix these errors: {{error_details}}

Config has been loaded successfully.

Model and data directory setup. The service checks the environment variables VMANOMALY_MODEL_DUMPS_DIR and VMANOMALY_DATA_DUMPS_DIR to determine where to store models and data. If these variables are not set, models and data will be stored in memory. Please find the on-disk mode details here. Log messages:

Using ENV MODEL_DUMP_DIR=`{{model_dump_dir}}` to store anomaly detection models.

ENV MODEL_DUMP_DIR is not set. Models will be kept in RAM between consecutive `fit` calls.

Using ENV DATA_DUMP_DIR=`{{data_dump_dir}}` to store anomaly detection data.

ENV DATA_DUMP_DIR is not set. Models' training data will be stored in RAM.

Scheduler and service initialization. After configuration is successfully loaded, the service initializes schedulers and services for each defined scheduler_alias. If there are issues with a specific scheduler (e.g., no models or queries found to attach to a scheduler), a warning is logged. When schedulers are initialized, the service logs a list of active schedulers. Log messages:

Scheduler {{scheduler_alias}} wrapped and initialized with {{N}} model spec(s).

No model spec(s) found for scheduler `{{scheduler_alias}}`, skipping setting it up.

Active schedulers: {{list_of_schedulers}}.

Back to logging sections

Reader logs #

The reader component logs events during the process of querying VictoriaMetrics and retrieving the data necessary for anomaly detection. This includes making HTTP requests, handling SSL, parsing responses, and processing data into formats like DataFrames. The logs help to troubleshoot issues such as connection problems, timeout errors, or misconfigured queries.

Starting a healthcheck request. When the reader component initializes, it checks whether the VictoriaMetrics endpoint is accessible by sending a request for _vmanomaly_healthcheck. Log messages:

[Scheduler {{scheduler_alias}}] Max points per timeseries set as: {{vm_max_datapoints_per_ts}}

[Scheduler {{scheduler_alias}}] Reader endpoint SSL error {{url}}: {{error_message}}

[Scheduler {{scheduler_alias}}] Reader endpoint inaccessible {{url}}: {{error_message}}

[Scheduler {{scheduler_alias}}] Reader endpoint timeout {{url}}: {{error_message}}

No data found (False). Based on query_from_last_seen_timestamp VmReader flag. A warning log is generated when no data is found in the requested range. This could indicate that the query was misconfigured or that no new data exists for the time period requested. Log message format:

[Scheduler {{scheduler_alias}}] No data between {{start_s}} and {{end_s}} for query "{{query_key}}"

No unseen data found (True). Based on query_from_last_seen_timestamp VmReader flag. A warning log is generated when no new data is returned (i.e., all data has already been seen in a previous inference step(s)). This helps in identifying situations where data for inference has already been processed. Based on VmReader’s adjust flag. Log messages:

[Scheduler {{scheduler_alias}}] No unseen data between {{start_s}} and {{end_s}} for query "{{query_key}}"

Connection or timeout errors. When the reader fails to retrieve data due to connection or timeout errors, a warning log is generated. These errors could result from network issues, incorrect query endpoints, or VictoriaMetrics being temporarily unavailable. Log message format:

[Scheduler {{scheduler_alias}}] Error querying {{query_key}} for {{url}}: {{error_message}}

Max datapoints warning. If the requested query range (defined by fit_every or infer_every scheduler args) exceeds the maximum number of datapoints allowed by VictoriaMetrics, a warning log is generated, and the request is split into multiple intervals (option available since v1.14.1). This ensures that the request does not violate VictoriaMetrics’ constraints. Log messages:

[Scheduler {{scheduler_alias}}] Query "{{query_key}}" from {{start_s}} to {{end_s}} with step {{step}} may exceed max datapoints per timeseries and will be split...

Multi-tenancy warnings. If the reader detects any issues related to missing or misconfigured multi-tenancy labels (supported since v1.16.2), a warning log is generated to indicate the issue. See additional details here. Log message format:

The label vm_account_id was not found in the label set of {{query_key}}, but tenant_id='multitenant' is set in reader configuration...

Metrics updated in read operations. During successful query execution process, the following reader self-monitoring metrics are updated:

• vmanomaly_reader_request_duration_seconds: Records the time (in seconds) taken to complete the query request.

• vmanomaly_reader_responses: Tracks the number of response codes received from VictoriaMetrics.

• vmanomaly_reader_received_bytes: Counts the number of bytes received in the response.

• vmanomaly_reader_response_parsing_seconds: Records the time spent parsing the response into different formats (e.g., JSON or DataFrame).

• vmanomaly_reader_timeseries_received: Tracks how many timeseries were retrieved in the query result.

• vmanomaly_reader_datapoints_received: Counts the number of datapoints retrieved in the query result.

Metrics skipped in case of failures. If an error occurs (connection or timeout), vmanomaly_reader_received_bytes, vmanomaly_reader_timeseries_received, and vmanomaly_reader_datapoints_received are not incremented because no valid data was received.

Back to logging sections

Service logs #

The model component (wrapped in service) logs operations during the fitting and inference stages for each model spec attached to particular scheduler scheduler_alias. These logs inform about skipped runs, connection or timeout issues, invalid data points, and successful or failed model operations.

Skipped runs. When there are insufficient valid data points to fit or infer using a model, the run is skipped and a warning log is generated. This can occur when the query returns no new data or when the data contains invalid values (e.g., NaN, INF). The skipped run is also reflected in the vmanomaly_model_runs_skipped metric. Log messages:

When there are insufficient valid data points (at least 1 for online models and 2 for offline models)

[Scheduler {{scheduler_alias}}] Skipping run for stage 'fit' for model '{{model_alias}}' (query_key: {{query_key}}): Not enough valid data to fit: {{valid_values_cnt}}

When all the received timestamps during an infer call have already been processed, meaning the anomaly_score has already been produced for those points

[Scheduler {{scheduler_alias}}] Skipping run for stage 'infer' for model '{{model_alias}}' (query_key: {{query_key}}): No unseen data to infer on.

When the model fails to produce any valid or finite outputs (such as anomaly_score)

[Scheduler {{scheduler_alias}}] Skipping run for stage 'infer' for model '{{model_alias}}' (query_key: {{query_key}}): No (valid) datapoints produced.

Errors during model execution. If the model fails to fit or infer data due to internal service errors or model spec misconfigurations, an error log is generated and the error is also reflected in the vmanomaly_model_run_errors metric. This can occur during both fit and infer stages. Log messages:

[Scheduler {{scheduler_alias}}] Error during stage 'fit' for model '{{model_alias}}' (query_key: {{query_key}}): {{error_message}}

[Scheduler {{scheduler_alias}}] Error during stage 'infer' for model '{{model_alias}}' (query_key: {{query_key}}): {{error_message}}

Model instance created during inference. In cases where an online model instance is created during the inference stage (without a prior fit, a feature introduced in v1.15.2), a debug log is produced. This helps track models that are created dynamically based on incoming data. Log messages:

[Scheduler {{scheduler_alias}}] Model instance '{{model_alias}}' created for '{{query_key}}' during inference.

Successful model runs. When a model successfully fits, logs track the number of valid datapoints processed and the time taken for the operation. These logs are accompanied by updates to self-monitoring metrics like vmanomaly_model_runs, vmanomaly_model_run_duration_seconds, vmanomaly_model_datapoints_accepted, and vmanomaly_model_datapoints_produced. Log messages:

For non-rolling models

[Scheduler {{scheduler_alias}}] Fitting on {{valid_values_cnt}}/{{total_values_cnt}} valid datapoints for "{{query_key}}" using model "{{model_alias}}".

[Scheduler {{scheduler_alias}}] Model '{{model_alias}}' fit completed in {{model_run_duration}} seconds for {{query_key}}.

For rolling models (combined stage)

[Scheduler {{scheduler_alias}}] Fit-Infer on {{datapoint_count}} points for "{{query_key}}" using model "{{model_alias}}".

Metrics updated in model runs. During successful fit or infer operations, the following self-monitoring metrics are updated for each run:

• vmanomaly_model_runs: Tracks how many times the model ran (fit, infer, or fit_infer) for a specific query_key.

• vmanomaly_model_run_duration_seconds: Records the total time (in seconds) for the model invocation, based on the results of the query_key.

• vmanomaly_model_datapoints_accepted: The number of valid datapoints processed by the model during the run.

• vmanomaly_model_datapoints_produced: The number of datapoints generated by the model during inference.

• vmanomaly_models_active: Tracks the number of models currently available for infer for a specific query_key.

Metrics skipped in case of failures. If a model run fails due to an error or if no valid data is available, the metrics such as vmanomaly_model_datapoints_accepted, vmanomaly_model_datapoints_produced, and vmanomaly_model_run_duration_seconds are not updated.

Back to logging sections

Writer logs #

The writer component logs events during the process of sending produced data (like anomaly_score metrics) to VictoriaMetrics. This includes data preparation, serialization, and network requests to VictoriaMetrics endpoints. The logs can help identify issues in data transmission, such as connection errors, invalid data points, and track the performance of write requests.

Starting a write request. A debug level log is produced when the writer component starts the process of writing data to VictoriaMetrics. It includes details like the number of datapoints, bytes of payload, and the query being written. This is useful for tracking the payload size and performance at the start of the request. Log messages:

[Scheduler {{scheduler_alias}}] POST {{url}} with {{N}} datapoints, {{M}} bytes of payload, for {{query_key}}

No valid data points. A warning log is generated if there are no valid datapoints to write (i.e., all are NaN or unsupported like INF). This indicates that the writer will not send any data to VictoriaMetrics. Log messages:

[Scheduler {{scheduler_alias}}] No valid datapoints to save for metric: {{query_key}}

Connection, timeout, or I/O errors. When the writer fails to send data due to connection, timeout, or I/O errors, an error log is generated. These errors often arise from network problems, incorrect URLs, or VictoriaMetrics being unavailable. The log includes details of the failed request and the reason for the failure. Log messages:

[Scheduler {{scheduler_alias}}] Cannot write {{N}} points for {{query_key}}: connection error {{url}} {{error_message}}

[Scheduler {{scheduler_alias}}] Cannot write {{N}} points for {{query_key}}: timeout for {{url}} {{error_message}}

[Scheduler {{scheduler_alias}}] Cannot write {{N}} points for {{query_key}}: I/O error for {{url}} {{error_message}}

Multi-tenancy warnings. If the tenant_id is set to multitenant but the vm_account_id label is missing from the query result, or vice versa, a warning log is produced (supported since v1.16.2). This helps in debugging label set issues that may occur due to the multi-tenant configuration - see this section for details. Log messages:

The label vm_account_id was not found in the label set of {{query_key}}, but tenant_id='multitenant' is set in writer...

The label set for the metric {{query_key}} contains multi-tenancy labels, but the write endpoint is configured for single-tenant mode (tenant_id != 'multitenant')...

Metrics updated in write operations. During the successful write process of non-empty data, the following self-monitoring metrics are updated:

• vmanomaly_writer_request_duration_seconds: Records the time (in seconds) taken to complete the write request.

• vmanomaly_writer_sent_bytes: Tracks the number of bytes sent in the request.

• vmanomaly_writer_responses: Captures the HTTP response code returned by VictoriaMetrics. In case of connection, timeout, or I/O errors, a specific error code (connection_error, timeout, or io_error) is recorded instead.

• vmanomaly_writer_request_serialize_seconds: Records the time taken for data serialization.

• vmanomaly_writer_datapoints_sent: Counts the number of valid datapoints that were successfully sent.

• vmanomaly_writer_timeseries_sent: Tracks the number of timeseries sent to VictoriaMetrics.

Metrics skipped in case of failures. If an error occurs (connection, timeout, or I/O error), only vmanomaly_writer_request_duration_seconds is updated with appropriate error code.

Back to logging sections