Choose a model for your metrics #

• SRE: detect an unexpected rise in CPU utilization for one instance under steady load.
• APM: detect a latency increase for a route whose normal response time is stable. Use a seasonal recipe if traffic drives regular daily changes .
• Hardware: detect an unusual temperature change on one sensor. Retain fixed safety limits for overheating.
• Business: detect a drop in a conversion rate. Use a seasonal recipe when the expected rate has daily or weekly patterns .

Choose the profile below by inspecting the query’s output after aggregation or rate calculation.

• CPU utilization across instances with different normal ranges, learned separately for each instance.
• Request latency for a stable route, after choosing an aggregation and sample interval that avoid sparse or very noisy results.
• Sensor temperatures around a stable operating point.
• Business ratios with enough observations per sample and no strong daily or weekly variation.

The important property is a stable average and spread, without frequent extreme values. Standard deviation is useful for describing the usual variation around that average. Check the threshold against your own data to understand its effect on false positives .

Strong patterns, frequent outliers, or sparse samples can make this baseline unreliable.

• Large repeating peaks : a single expected range either becomes too wide to catch incidents or flags normal peaks repeatedly. Use Online Seasonal Quantile or Temporal Envelope .
• Frequent or very large outliers : they pull the average and widen its range, potentially hiding later incidents. Consider Online MAD .
• A sustained trend or a new operating level: this model can initially flag a changepoint, then gradually absorb it. Use Temporal Envelope when these changes are normal.
• Mostly zero or sparse counts: check the aggregation interval and event volume first. A standard deviation around a near-zero baseline may not express business significance.

Two instances, two CPU incidents.

Open the prepared metrics playground .

1. Select Execute Query to inspect the two synthetic CPU-utilization series.
2. Select Detect Anomalies to run Online Z-score with the prepared settings.
3. Inspect checkout-01 on January 10, 2026, 12:00–15:00 UTC and January 20, 2026, 06:00–09:00 UTC. The rise and fall should cross the expected range and have scores above 1. The other instance should remain within its own range.

This online model keeps learning as new data arrives, so regular refits are optional.

Times in this guide are UTC. The UI may display the same instants in your selected timezone.

What to expect: gentle oscillations and small noisy deviations stay inside each instance’s expected range, without a strong daily or weekly pattern. Each three-hour incident affects twelve observations on checkout-01, with consecutive flagged observations connected by a line when Streaks is enabled. Scores may taper as the model learns from the incident. The other instance remains healthy. Values use ratios: 0.45 means 45% CPU utilization. The query generates demonstration data.

The query depends only on timestamps and fixed constants. Its small noise is generated by a deterministic mixing function, so the value at a timestamp is unchanged when the fit and display windows differ. Seeded rand functions reproduce a sequence from the beginning of a request. Changing the request start can assign different values to the same timestamps.

If using your own UI , copy the query from the configuration below and the settings from this table. The public playground needs network access. The configuration remains readable offline.

These reader, models, and schedulers sections use a periodic scheduler to score current data continuously. The playground above replays its fixed historical dates with the same fit window and cadence. Keep the writer and license configuration from the Quickstart . Use a test destination for the demo results written by this configuration.

      reader:
  class: vm
  datasource_url: https://play.victoriametrics.com/select/0/prometheus
  sampling_period: 15m
  queries:
    cpu_utilization:
      expr: |
        WITH (
          # Timestamp-based noise stays the same across fit and display windows.
          mixed(seed) = sin(time() / 900 * 12.9898 + seed) * 43758.5453,
          noise(seed) = 0.05 * (mixed(seed) - floor(mixed(seed)) - 0.5),
          # Subtle waves (0.3 percentage points each), without a calendar pattern.
          fluctuation(seed) = noise(seed)
            + 0.003 * sin(time() / (11 * 3600) + seed)
            + 0.003 * sin(time() / (5 * 3600) + seed * 0.7),
          rise = (time() >= bool 1768046400) * (time() < bool 1768057200),
          fall = (time() >= bool 1768888800) * (time() < bool 1768899600)
        )
        label_set(0.45 + fluctuation(42) + 0.25 * rise - 0.20 * fall,
          "__name__", "demo_cpu_utilization_ratio", "instance", "checkout-01", "service", "checkout")
        or label_set(0.20 + fluctuation(43),
          "__name__", "demo_cpu_utilization_ratio", "instance", "checkout-02", "service", "checkout")
      detection_direction: both  # When spikes and drops matter. Use above_expected or below_expected for one direction.
      data_range: [0, 1]  # Utilization is expressed as a ratio.

models:
  zscore_steady_cpu:  # Choose a unique alias for this model configuration.
    class: zscore_online  # Select the Online Z-score model.
    queries: [cpu_utilization]  # Connect to one or more aliases defined in reader.queries.
    schedulers: [steady_cpu_demo]  # Connect to one or more aliases defined in schedulers.
    z_threshold: 3  # Set the expected range to three standard deviations from the mean.
    min_n_samples_seen: 96  # One day's worth of 15-minute samples per instance.
    decay: 0.998          # Slower adaptation. History weight halves in ~3.6d at 15m steps.
    history_strength: 1.0  # Give fit history its ordinary weight.

schedulers:
  steady_cpu_demo:
    class: periodic  # Run continuously on incoming data.
    fit_window: 3d
    fit_every: 1000w  # Learning continues between these very infrequent full refits.
    infer_every: 15m
    

queries and schedulers explicitly bind the model to this example. Query-level business policies apply consistently if you later compare models.

For production, replace the synthetic expression with your utilization query, for example 1 - avg by (instance, service) (rate(node_cpu_seconds_total{mode="idle"}[5m])) if those labels exist in your data. Adjust the inference cadence to your response-time requirements. The deliberately long fit_every: 1000w highlights online learning: the model keeps updating between full refits. fit_window supplies the initial history at each fit. The model continues learning from observations after that window.

Give new series time to warm up. min_n_samples_seen: 96 requires 96 valid observations per series, including fit data, before non-neutral scoring. At 15m, that is one day’s worth of regularly sampled data. Missing samples extend the wait. Until warmup completes, the score is 0 because the model needs more history. Check warmup status before interpreting zero scores as healthy. The three-day fit normally warms up both demo instances. This recipe uses 96 to allow more warmup than the default 16.

Adjust sensitivity. z_threshold: 3 places the initial expected range three standard deviations from the mean. Raise it to tolerate larger fluctuations or lower it to investigate smaller changes. Adjust this starting value to your data and investigation needs. Review false positives and false negatives (missed incidents) on representative data before enabling alerts.

Choose how quickly to adapt. This recipe uses decay: 0.998 for slower adaptation, helping limit abrupt changes in the expected range after brief incidents. Older information loses half its weight after about 346 new observations (roughly 3.6 days at 15m). Values closer to 1 adapt more slowly. Setting 1 retains all observations since the latest fit. Faster adaptation can help track an accepted new level, but can also absorb a persistent incident sooner. Retest after changing the sample cadence.

Weight fit history when appropriate. history_strength: 1 gives the fit data its ordinary weight. With representative, similar history, 2 makes that initial evidence twice as influential against subsequent updates. A one-week fit with strength 2 still covers only one week and supplies the same number of valid warmup samples. Use it to adjust adaptation while retaining enough history to cover the patterns you need.

Apply domain knowledge to each query. This example uses detection_direction: both to show a rise and a fall. Use above_expected if only CPU rises matter for your service, or below_expected for a conversion-rate drop. Consider min_dev_from_expected to ignore negligible changes: for a utilization ratio, 0.02 means two percentage points. Place these policies under reader.queries.<alias>.

Open the spikes-only comparison . This keeps the same data and model, names the query cpu_utilization, and sets Query overrides to detection_direction: above_expected, data_range: [0, 1], and min_dev_from_expected: [0.02, 0.02]. Select Detect Anomalies: only the January 10 rise should score above 1, while the January 20 fall should score 0. The two-percentage-point minimum deviation is smaller than either incident, so direction is what changes this example’s results. To apply these policies in YAML, replace the query’s policy fields with:

      detection_direction: above_expected
data_range: [0, 1]
min_dev_from_expected: [0.02, 0.02]  # Two percentage points of utilization.
    

Match scheduling to the decision. The example’s 15m step and inference cadence are for an easy-to-inspect demo. Short incidents may require finer sampling. Slower business KPIs may need coarser aggregation. Keep Exact enabled for causal online backtests and set Infer Every deliberately. A periodic refit resets learned state using its fit window. Check whether that history contains incidents that could become part of the new baseline.

Separate detection from paging. An anomaly score is evidence of unusual behavior. Combine it with impact, persistence, and existing safety or SLO rules when deciding to page. See the alerting guide and alert-fatigue guidance .

• Request latency with a stable typical level and sporadic slow samples.
• Processing or queue-wait durations interrupted by occasional long jobs.
• Sensor readings where a few extreme measurements should not distort the baseline.

The baseline should describe most observations. Robustness limits how much an extreme value changes that baseline. The extreme value can still receive a high anomaly score.

Predictable patterns, multiple operating modes, or sparse samples may need a different model or query.

• Strong daily or weekly variation: use Online Seasonal Quantile or Temporal Envelope , so normal peaks are compared with their calendar context.
• Normal growth or frequent accepted level changes: use Temporal Envelope . MAD initially treats an unfamiliar level as unusual and can later learn it. Decide whether a deployment-related change should be accepted based on its impact.
• Several normal operating modes: one median and spread can poorly describe their separate ranges. Check whether labels or query aggregation can separate the workloads.
• Sparse or nearly constant results: check the sampling interval and set a meaningful minimum deviation. A very narrow learned range can make tiny changes look significant.

Two latency spikes, then a lasting increase.

Open the MAD latency playground .

1. Select Execute Query to inspect one synthetic latency series, named checkout_latency.
2. Select Detect Anomalies to run Online MAD with the prepared settings.
3. Inspect the two spikes on January 8 and January 16, 2026, 12:00–12:30 UTC, then the lasting increase starting January 16, 18:00 UTC. MAD should flag both spikes and the beginning of the new level.

This online model keeps learning as new data arrives, so regular refits are optional.

What to expect: values use seconds: 0.2 means 200 ms. The series has timestamp-based noise, two 30-minute spikes, and a permanent 60 ms increase. There is no trend or calendar pattern to learn. As new observations arrive, the expected range can move and widen until the new level is no longer flagged. Use latency or SLO checks to confirm that the regression has been resolved.

Compare with Online Z-score . The comparison keeps the query, policies, dates, schedule, and decay the same, while changing the model and its threshold. The large spikes widen the Z-score range, which can hide the smaller increase that follows. Compare the expected ranges at the start of the level change. MAD’s threshold and Z-score’s z_threshold measure different spreads and need separate tuning.

These reader, models, and schedulers sections use a periodic scheduler for continuous scoring. The playground above replays the dated latency example with the same fit window and cadence. Keep the writer and license settings from the Quickstart and use a test writer destination, as in the previous recipe.

      reader:
  class: vm
  datasource_url: https://play.victoriametrics.com/select/0/prometheus
  sampling_period: 15m
  queries:
    checkout_latency:
      expr: |
        WITH (
          # Fixed timestamp-based noise, in seconds.
          mixed = sin(time() / 900 * 12.9898 + 42) * 43758.5453,
          noise = 0.02 * (mixed - floor(mixed) - 0.5),
          spike_1 = (time() >= bool 1767873600) * (time() < bool 1767875400),
          spike_2 = (time() >= bool 1768564800) * (time() < bool 1768566600),
          changed = time() >= bool 1768586400
        )
        label_set(0.20 + noise + 0.80 * (spike_1 + spike_2) + 0.06 * changed,
          "__name__", "demo_request_duration_seconds", "quantile", "0.95",
          "route", "/checkout", "service", "checkout")
      detection_direction: above_expected  # Flag slower requests while allowing faster ones.
      min_dev_from_expected: [0.02, 0.02]  # Ignore deviations smaller than 20 ms.
      data_range: [0, 5]  # This demo service has a five-second request timeout.

models:
  mad_checkout_latency:
    class: mad_online
    queries: [checkout_latency]
    schedulers: [latency_demo]
    threshold: 5.0  # Multiplier of median absolute deviation, not standard deviation.
    min_n_samples_seen: 96  # One day's worth of 15-minute samples.
    decay: 0.99  # Favor recent observations so sustained changes can become normal.
    history_strength: 1.0
    compression: 100  # Starting accuracy/memory tradeoff for approximate quantiles.

schedulers:
  latency_demo:
    class: periodic  # Run continuously on incoming data.
    fit_window: 3d
    fit_every: 1000w  # Learning continues between these very infrequent full refits.
    infer_every: 15m
    

For the Z-score comparison, replace class: mad_online with class: zscore_online, replace threshold: 5.0 with z_threshold: 3.0, and remove compression. Keep the query policies and schedule unchanged. Replace the generated expression with your latency query when applying the recipe to real data. Preserve its units and aggregation meaning.

Start with representative history and warmup. The three-day fit contains healthy data and exceeds min_n_samples_seen: 96. New series need 96 valid observations before non-neutral scores. Missing observations delay warmup. Include ordinary variability in fit history. A perfectly flat fit can produce an overly narrow range.

Tune sensitivity in MAD units. threshold: 5.0 is a starting point for this query. Increase it to tolerate more variation or decrease it to investigate smaller departures. MAD measures distance from the median, so tune its threshold for the spread of your data. Check false positives and missed incidents on representative data.

Set business significance in the query’s units. detection_direction: above_expected allows latency improvements. min_dev_from_expected: [0.02, 0.02] measures a 20 ms minimum distance from the expected value. A sample must also cross the model’s expected range to be flagged. The two limits work together. See minimum deviation .

Decide how long a new level should remain unusual. With decay: 0.99, recent observations gain influence and a persistent increase can be absorbed. Values closer to 1 retain older evidence longer. Setting history_strength above 1 strengthens the initial fit history without adding time coverage. A shorter refit interval can also reset the baseline from recent data. Backtest these settings against the time you need to investigate a regression. Use impact or SLO checks to confirm recovery.

Keep persistence separate from severity. Streaks controls how consecutive flags are displayed in the UI. Configure alert persistence separately. The two large spikes can still be anomalies even though they barely move MAD’s baseline. Decide whether to page using impact and persistence, and retain latency or SLO rules for regressions that must stay actionable after the model adapts. See the alerting guide .

• Service traffic that switches between fairly steady opening-hours and overnight levels.
• Scheduled batch-processing throughput with repeatable active and idle periods.
• Hardware power usage that follows a stable shift schedule.
• Business activity with repeatable daily or weekly operating windows.
• Custom production cycles: for a line whose product mix and processing/cleaning schedule repeat every two weeks, three-hour slots can capture the usual throughput or power usage for each stage.

The normal level and spread should be fairly consistent when the same time slot returns. Place bucket boundaries at changes between operating periods, so each bucket mostly contains one normal level.

Smooth curves, mixed time slots, or changing baselines can make step-shaped predictions a poor fit.

• Smooth seasonal curves , such as sinusoids: one nearly horizontal range covers the whole bucket. If the signal moves substantially within that interval, the expected value can be too high at one end and too low at the other. Narrow bounds can flag normal movement, while wide bounds can hide smaller incidents. Consider Temporal Envelope for smooth patterns.
• Neighboring hours with different normal levels: a two-hour bucket can mix an hour near 50 requests/s with one near 200. Its shared center may represent neither hour well, and its range can become wide enough to accept both levels at the wrong time. Smaller, correctly aligned buckets help, provided enough history is available for each.
• Normal growth , shifting operating hours, or frequent level changes: use Temporal Envelope when these changes are expected. Fixed seasonal slots work best when the same operating periods recur.
• Little repeated history or sparse samples: a bucket cannot learn a dependable range from a few observations. Check aggregation and cycle coverage before making the buckets smaller.

Mixing different normal phases can widen the range and hide incidents. Robust quantiles reduce the influence of extreme values. Use suitable bucket boundaries to preserve the time context.

See the smooth-curve limitation. Open the smooth-curve example with four-hour buckets . Select Detect Anomalies, then inspect January 10, 2026, 10:30–11:30 UTC. A 30 requests/s drop in otherwise smooth daily traffic can stay inside the broad range learned for 08:00–12:00. Compare the same curve with one-hour buckets . Smaller buckets reveal the drop but can also flag normal within-hour movement. Set the Streaks minimum to 1 to inspect isolated flags.

These links use one region and the query below, shown over January 3–17, 2026, UTC to keep the slowdown near the center. They retain the recipe’s 14-day fit history, quantiles, and 10% relative-deviation policy. Only the bucket width differs between the two links. For the coarse model settings, replace the query in the configuration below and set min_subseason: 4h.

      WITH (
  hour_utc = (time() % 86400) / 3600,
  # A smooth daily cycle from 50 to 200 requests/s, peaking at noon.
  baseline = 125 + 75 * sin((hour_utc - 6) / 24 * 6.283185307179586),
  mixed = sin(time() / 900 * 12.9898 + 42) * 43758.5453,
  noise = 10 * (mixed - floor(mixed) - 0.5),
  slowdown = (time() >= bool 1768041000) * (time() < bool 1768044600)
)
label_set(baseline + noise - 30 * slowdown,
  "__name__", "demo_http_requests_per_second", "region", "eu-west", "service", "checkout")
    

Two regions, daily plateaus, and values arriving at the wrong time.

Open the plateau-pattern playground .

1. Select Execute Query to inspect checkout_request_rate for two regions. Both switch between overnight and opening-hours plateaus.
2. Select Detect Anomalies to run Online Seasonal Quantile with one-hour buckets.
3. Inspect eu-west on January 10, 2026, 09:00–10:00 UTC, when traffic drops to the overnight level, and January 20, 02:00–05:00 UTC, when it rises to the opening-hours level. Both incidents should cross the expected range. The regular daily transitions and eu-east should remain unflagged.

This online model keeps learning as new data arrives, so regular refits are optional.

What to expect: the expected range steps between the low and high plateaus at 09:00 and 19:00 UTC. The daytime incident affects four observations, and the nighttime incident affects twelve. Values use requests per second. The two regions have separate expected ranges, even though one query returns both series.

Compare with two-hour buckets . Only min_subseason changes from 1h to 2h. Inspect 08:00–10:00 and 18:00–20:00 UTC: each coarse bucket now covers an hour from each plateau. Compare the wider range and less representative expected value with the one-hour version. The January 10 drop can disappear into that mixed range, even though its business meaning is unchanged.

These reader, models, and schedulers sections use a periodic scheduler for continuous scoring. The playground above replays the dated plateau example with the same fit window and cadence. Keep the writer and license settings from the Quickstart , and use a test writer destination.

      reader:
  class: vm
  datasource_url: https://play.victoriametrics.com/select/0/prometheus
  sampling_period: 15m
  queries:
    checkout_request_rate:
      expr: |
        WITH (
          hour_utc = (time() % 86400) / 3600,
          open = (hour_utc >= bool 9) * (hour_utc < bool 19),
          baseline = 50 + 150 * open,
          # Small timestamp-based noise, in requests per second.
          mixed(seed) = sin(time() / 900 * 12.9898 + seed) * 43758.5453,
          noise(seed) = 10 * (mixed(seed) - floor(mixed(seed)) - 0.5),
          daytime_drop = (time() >= bool 1768035600) * (time() < bool 1768039200),
          nighttime_rise = (time() >= bool 1768874400) * (time() < bool 1768885200)
        )
        label_set(baseline + noise(42) - 150 * daytime_drop + 150 * nighttime_rise,
          "__name__", "demo_http_requests_per_second", "region", "eu-west", "service", "checkout")
        or label_set(0.6 * baseline + noise(43),
          "__name__", "demo_http_requests_per_second", "region", "eu-east", "service", "checkout")
      detection_direction: both  # Unexpected traffic rises and falls both matter.
      min_rel_dev_from_expected: [10, 10]  # Ignore deviations smaller than 10% below/above the expected value.
      data_range: [0, 500]  # Demo intake's configured capacity is 500 requests/s.

models:
  quantile_daily_traffic:  # Choose a unique model alias.
    class: quantile_online  # Select Online Seasonal Quantile.
    queries: [checkout_request_rate]  # Connect to one or more reader query aliases.
    schedulers: [daily_traffic_demo]  # Connect to one or more scheduler aliases.
    seasonal_interval: 1d  # Repeat the same daily pattern.
    min_subseason: 1h  # Keep a separate range for each clock hour.
    season_starts_from: '2026-01-01'  # Align the demo's buckets to UTC midnight.
    quantiles: [0.25, 0.5, 0.75]  # Robust lower, central, and upper estimates.
    iqr_threshold: 2.0  # Widen the range to tolerate ordinary within-bucket noise.
    global_smoothing: 0.0  # Keep the time-specific ranges separate from the overall range.
    min_n_samples_seen: 96  # Total valid observations required before non-neutral scoring.
    decay: 0.99  # Let each time slot gradually adapt as it is observed again.
    history_strength: 1.0  # Give the initial fit its ordinary weight.
    compression: 100  # Starting accuracy/memory tradeoff for approximate quantiles.
    clip_predictions: true  # Keep predicted ranges within the query's valid data range.

schedulers:
  daily_traffic_demo:
    class: periodic  # Run continuously on incoming data.
    fit_window: 14d  # Fourteen daily cycles with 56 samples per hour slot.
    fit_every: 1000w  # Learning continues between these very infrequent full refits.
    infer_every: 15m
    

For the coarse-bucket comparison, change only min_subseason: 1h to min_subseason: 2h. For real traffic, replace the generated expression with your request-rate query, for example sum by (region, service) (rate(http_requests_total{service="checkout"}[5m])) if those labels exist. Choose the cycle and buckets from the resulting series, after aggregation.

Align buckets with operating periods. min_subseason sets the bucket width independently of the query step. Here 1h holds four 15m observations from the same clock hour each day. It divides seasonal_interval: 1d into 24 slots. A two-hour bucket mixes the 08:00 overnight hour with the 09:00 opening hour. Smaller buckets can follow a smooth curve more closely, but need more repeated cycles to estimate each slot reliably.

Use custom cycles when the operation calls for them. The cycle need not be a day or a week. For the two-week production plan above, seasonal_interval: 2w and min_subseason: 3h create 112 separate slots. Set season_starts_from to the plan’s actual starting date and time, and use several complete cycles as fit history. For example, fit_window: 6w covers three repetitions. The slot width must divide the full cycle evenly, and its boundaries should keep different operating stages separate.

Cover every slot before relying on it. The 14d fit supplies fourteen daily cycles and about 56 observations per hour slot. min_n_samples_seen: 96 checks the total observations seen by a model. Inspect each bucket separately to confirm that it has enough observations. For a weekly schedule, use a weekly cycle with enough history to observe each weekday and weekend slot repeatedly. Increasing history_strength cannot supply missing weekdays or unseen operating periods.

Use quantiles and IQR together. The middle quantiles [0.25, 0.5, 0.75] reduce the influence of a few extreme values. iqr_threshold expands the resulting range using the spread within each bucket. Increasing it tolerates more variation but can hide incidents. Very wide quantiles near 0 and 1 can let rare extremes widen the range. Neither choice fixes a bucket that contains two different normal levels.

Scale business tolerance with the expected traffic. min_rel_dev_from_expected: [10, 10] ignores deviations smaller than 10% below or above the expected value. That is about 5 requests/s when 50 is expected, or 20 requests/s when 200 is expected. The percentage is measured from the expected value. A sample must exceed both this tolerance and the model’s expected range to be flagged.

Distinguish global blending from a smooth seasonal curve. global_smoothing: 0 uses each slot’s own distribution. Increasing it blends in the distribution across all times, which can reduce the contrast between overnight and opening-hours predictions. The bucket boundaries still produce steps.

Match time and cadence to the schedule. The generated pattern and bucket alignment use UTC. The UI may display those same instants in another timezone. Check local operating hours, daylight-saving changes, and the cycle reference point when applying the recipe. Keep the query step and inference cadence deliberate: changing them changes how many observations each slot receives.

Let accepted changes adapt gradually. decay below 1 gives new observations more influence when their slot is updated. A slot seen only during one clock hour adapts on a different wall-clock timescale from a continuously updated Z-score baseline. If changing levels or moving schedules are part of normal behavior, consider Temporal Envelope .

• Service traffic with smooth daily demand, weekday/weekend differences, and gradual growth.
• CPU or power usage whose usual range changes with workload and calendar schedules.
• Business activity with recurring calendar effects and an evolving normal level.
• Metrics affected by accepted traffic migrations or capacity changes, when a new level should eventually become normal.

The model learns what the metric usually does at that time and how its baseline evolves. Decide which persistent shifts should become normal based on their business impact.

A simpler baseline, a custom cycle, or a different comparison better matches the question.

• Stable data without calendar patterns: start with Online Z-score or Online MAD . Their simpler ranges can be easier to inspect and tune.
• Fixed plateaus or custom-duration cycles dominate: consider Online Seasonal Quantile , especially for schedules such as a repeating two-week production plan with three-hour slots. Temporal Envelope’s seasonal presets follow calendar patterns. Seasonal Quantile also supports custom-duration cycles.
• The relationship between signals matters: use multivariate models . An independently plausible CPU value and RAM value can still form an unusual pair.
• One entity differs from comparable entities now: use a peer-group model when the comparison should be with its peers at the same timestamp.

A growing service, daily and weekend patterns, a new level, and two brief incidents.

Open the Temporal Envelope playground .

1. Select Execute Query to inspect checkout_evolving_rate. The two-week view shows smooth daily cycles over a rising baseline, with lower weekend traffic.
2. Select Detect Anomalies to fit and run Temporal Envelope chronologically.
3. Inspect the lasting increase from January 12, 2026, 00:00 UTC, then the surge on January 18, 10:00–13:00 UTC and drop on January 20, 02:00–05:00 UTC. Compare the learned expected range before and after the level change, and during the two brief incidents.

This online model keeps learning as new data arrives, so regular refits are optional.

What to expect: the center follows the daily curve and gradual growth. The January 12 increase is a changepoint. It can receive high scores before the model adapts. The later short surge and drop are separate departures from the new baseline. Scores can fade during an incident, and returning to normal can briefly be flagged while the model catches up. The query is deterministic, and values use requests per second. Use impact and SLO checks to confirm recovery after a lasting change, as the score can fall through adaptation.

These reader, models, and schedulers sections use a periodic scheduler for continuous scoring. The playground above replays the dated traffic example with the same fit window and cadence. Keep the writer and license settings from the Quickstart , and use a test writer destination.

      reader:
  class: vm
  datasource_url: https://play.victoriametrics.com/select/0/prometheus
  sampling_period: 15m
  queries:
    checkout_evolving_rate:
      expr: |
        WITH (
          hour_utc = (time() % 86400) / 3600,
          days_since_start = (time() - 1767225600) / 86400,
          # Unix epoch was Thursday. Number weekdays from Monday=0 to Sunday=6.
          weekday = floor(time() / 86400 + 3) % 7,
          weekend = weekday >= bool 5,
          daily = 60 * sin((hour_utc - 6) / 24 * 6.283185307179586),
          baseline = 150 + 2 * days_since_start + daily - 25 * weekend,
          mixed = sin(time() / 900 * 12.9898 + 42) * 43758.5453,
          noise = 10 * (mixed - floor(mixed) - 0.5),
          migration = time() >= bool 1768176000,
          surge = (time() >= bool 1768730400) * (time() < bool 1768741200),
          drop = (time() >= bool 1768874400) * (time() < bool 1768885200)
        )
        label_set(baseline + noise + 35 * migration + 100 * surge - 70 * drop,
          "__name__", "demo_http_requests_per_second", "region", "eu-west", "service", "checkout")
      detection_direction: both  # Investigate unexpected surges and drops.
      min_rel_dev_from_expected: [10, 10]  # Require at least 10% deviation from the expected value.
      data_range: [0, 500]  # Demo intake's configured capacity is 500 requests/s.

models:
  envelope_checkout_rate:  # Choose a unique model alias.
    class: temporal_envelope  # Select univariate Temporal Envelope.
    queries: [checkout_evolving_rate]  # Connect to one or more reader query aliases.
    schedulers: [evolving_traffic_demo]  # Connect to one or more scheduler aliases.
    seasonalities: [hod_smooth, weekpart_plateau]  # Daily curves and weekday/weekend levels.
    quantiles: [0.25, 0.75]  # Lower and upper residual quantiles. The center is learned separately.
    iqr_threshold: 2.0  # Control the width of the expected range.
    alpha: 0.005  # Trend Reactivity: how quickly the baseline follows lasting changes.
    loss_reactivity: 3.0  # Limit how strongly brief deviations influence learning.
    changepoint_window: 16  # Look for a lasting shift over 16 observations, about four hours here.
    min_n_samples_seen: 96  # Require one day's worth of valid observations before non-neutral scoring.
    clip_predictions: true  # Keep predicted ranges within the query's valid data range.

schedulers:
  evolving_traffic_demo:
    class: periodic  # Run continuously on incoming data.
    fit_window: 28d  # Four repetitions of the weekly pattern.
    fit_every: 1000w  # Learning continues between these very infrequent full refits.
    infer_every: 15m
    

For production, replace the generated expression with your request-rate query and choose history that covers the calendar patterns you enable. The deliberately long fit_every: 1000w highlights learning between full refits. Adjust sampling and scheduling to the decisions you need to make.

Check each behavior separately. Inspect normal daily/weekend cycles, adaptation to the migration, and detection of the brief surge and drop. Recheck all three after changing one setting.

Use representative calendar context. Enable patterns covered by fit history, and set the query’s tz for local operating hours. Add holiday context when the metric responds to those events and representative history is available.

For range width, reactivity, shift detection, and warmup settings, use the Temporal Envelope parameter reference . Keep impact or SLO alerts for changes that must remain actionable after the model adapts.

• SRE: RAM utilization stays high while CPU falls on a replica where they usually rise and fall together.
• APM: request rate falls while CPU remains high. Either value may look ordinary alone, but their combination can indicate wasted work or a stuck process.
• Hardware: fan speed no longer rises as temperature increases on a host.
• Business: purchase rate falls while checkout attempts stay steady.

Inputs need consistent identities, timestamps, and grouping. Keep a stable set of related signals in each group. For changing populations of comparable entities, use peer-group models .

• Light-job rate, heavy-job rate, and CPU when processing cost depends on the work mix.
• Request rates for different endpoint classes and resource use when endpoints have different costs.
• Checkout attempts and completed purchases when their relationship is more useful than either count alone.

Check the relationship in healthy data first. Compare signals for the same scope and interval. Include representative workload mixes and enough observations for the model to learn their variation.

The inputs are unrelated, membership changes, or a known rule already describes the relationship.

• Each signal needs its own decision: use a per-series recipe .
• You need to compare interchangeable replicas now: use a peer-group model . Here, each pool is compared with its own learned job-mix behavior.
• A stable cost equation is already known: monitor its residual directly. A learned model can help when the expected relationship and its variation differ between pools and must be estimated from data.
• Input identities or timing are inconsistent: fix aggregation, alignment, and missing channels before fitting a joint model.

Two worker pools, three named queries, and a change in their relationship. Open the multivariate playground .

1. Select Execute Query. Each query returns one series per pool, giving six input series in total.
2. Select Detect Anomalies to fit one model per service and pool pair, then score observations chronologically.
3. Inspect January 15, 2026, 10:00–16:00 UTC on render-a. Job rates and CPU stay within their usual ranges, but CPU no longer matches the work mix. Compare with healthy render-b.

The online model continues learning after the initial fit. Frequent full refits are optional.

What to expect: each pool receives one joint anomaly-score series. Five of the six incident observations are flagged, forming one streak after the first hourly observation. Healthy data remains unflagged in this replay. All six incident observations remain inside all three channels’ predicted ranges. The joint decision uses their combination.

Compare with simpler baselines. Run the same inputs through independent models to fit six separate envelopes with the recipe’s range settings, or try the standard univariate range with quantiles [0.25, 0.75], iqr_threshold: 2, and scale [1, 1]. You can also try CPU divided by total job rate . All comparisons use the same fit history, replay dates, cadence, and causal scoring.

These are raw-score counts for this generated example. Each pool timestamp is counted once, even if several independent channels flag it. The incident redistributes light and heavy jobs while preserving total job rate and CPU, so their ratio is unchanged. The tighter independent settings also flag healthy variation. This does not establish an advantage for every dataset.

When a rule is enough: the simulator uses a known cost equation to generate CPU. For render-a, expected CPU is 0.12 + 0.001 * light_jobs + 0.008 * heavy_jobs. For render-b, it is 0.08 + 0.002 * light_jobs + 0.006 * heavy_jobs. A residual rule using these exact coefficients catches all six incident observations. The learned model’s value is estimating relationships and normal variation from history when those costs are not already known. Compare it with available rules and independent models on your own incidents and healthy periods.

The three queries retain matching service and pool labels. groupby creates two models, each with three input channels. Keep job_type out of groupby so light and heavy jobs remain in the same model. Keep the writer and license configuration from the Quickstart .

      reader:
  datasource_url: https://play.victoriametrics.com/select/0/prometheus
  sampling_period: 1h
  queries:
    light_jobs:
      expr: |
        WITH (
          mixed(seed) = sin(time() / 3600 * 12.9898 + seed) * 43758.5453,
          unit(seed) = mixed(seed) - floor(mixed(seed)),
          total(seed) = 95 + 10 * unit(seed),
          mix(seed) = 0.10 + 0.20 * unit(seed + 10),
          noise(seed) = 0.002 * (unit(seed + 20) - 0.5),
          incident = (time() >= bool 1768471200) * (time() < bool 1768492800),
          reported_mix(seed, event) = 0.10 + 0.20 * ((unit(seed + 10) + 0.5 * event) % 1),
          light(seed, event) = total(seed) * (1 - reported_mix(seed, event)),
          heavy(seed, event) = total(seed) * reported_mix(seed, event),
          cpu_a = 0.12 + total(42) * (0.001 + 0.007 * mix(42)) + noise(42),
          cpu_b = 0.08 + total(43) * (0.002 + 0.004 * mix(43)) + noise(43)
        )
        label_set(light(42, incident), "__name__", "demo_completed_jobs_per_second", "service", "image-processing", "pool", "render-a", "job_type", "thumbnail")
        or label_set(light(43, 0), "__name__", "demo_completed_jobs_per_second", "service", "image-processing", "pool", "render-b", "job_type", "thumbnail")
      detection_direction: both
      data_range: [0, 200]
    heavy_jobs:
      expr: |
        WITH (
          mixed(seed) = sin(time() / 3600 * 12.9898 + seed) * 43758.5453,
          unit(seed) = mixed(seed) - floor(mixed(seed)),
          total(seed) = 95 + 10 * unit(seed),
          mix(seed) = 0.10 + 0.20 * unit(seed + 10),
          noise(seed) = 0.002 * (unit(seed + 20) - 0.5),
          incident = (time() >= bool 1768471200) * (time() < bool 1768492800),
          reported_mix(seed, event) = 0.10 + 0.20 * ((unit(seed + 10) + 0.5 * event) % 1),
          light(seed, event) = total(seed) * (1 - reported_mix(seed, event)),
          heavy(seed, event) = total(seed) * reported_mix(seed, event),
          cpu_a = 0.12 + total(42) * (0.001 + 0.007 * mix(42)) + noise(42),
          cpu_b = 0.08 + total(43) * (0.002 + 0.004 * mix(43)) + noise(43)
        )
        label_set(heavy(42, incident), "__name__", "demo_completed_jobs_per_second", "service", "image-processing", "pool", "render-a", "job_type", "full-resolution")
        or label_set(heavy(43, 0), "__name__", "demo_completed_jobs_per_second", "service", "image-processing", "pool", "render-b", "job_type", "full-resolution")
      detection_direction: both
      data_range: [0, 60]
    worker_cpu:
      expr: |
        WITH (
          mixed(seed) = sin(time() / 3600 * 12.9898 + seed) * 43758.5453,
          unit(seed) = mixed(seed) - floor(mixed(seed)),
          total(seed) = 95 + 10 * unit(seed),
          mix(seed) = 0.10 + 0.20 * unit(seed + 10),
          noise(seed) = 0.002 * (unit(seed + 20) - 0.5),
          incident = (time() >= bool 1768471200) * (time() < bool 1768492800),
          reported_mix(seed, event) = 0.10 + 0.20 * ((unit(seed + 10) + 0.5 * event) % 1),
          light(seed, event) = total(seed) * (1 - reported_mix(seed, event)),
          heavy(seed, event) = total(seed) * reported_mix(seed, event),
          cpu_a = 0.12 + total(42) * (0.001 + 0.007 * mix(42)) + noise(42),
          cpu_b = 0.08 + total(43) * (0.002 + 0.004 * mix(43)) + noise(43)
        )
        label_set(cpu_a, "__name__", "demo_cpu_utilization_ratio", "service", "image-processing", "pool", "render-a")
        or label_set(cpu_b, "__name__", "demo_cpu_utilization_ratio", "service", "image-processing", "pool", "render-b")
      detection_direction: both
      data_range: [0, 1]
  class: vm
models:
  envelope_worker_relationship:  # A model alias chosen for this recipe.
    class: temporal_envelope_multivariate
    queries: [light_jobs, heavy_jobs, worker_cpu]  # Connect the three named queries.
    schedulers: [worker_relationship_demo]  # Connect the schedule below.
    groupby: [service, pool]  # One three-channel model per worker pool.
    seasonalities: []  # Work mix varies without a calendar pattern.
    dependency_rank: 3  # Use dependency features for all three inputs.
    score_aggregation: l2  # Combine channel deviations into one group score.
    quantiles: [0.35, 0.65]  # Central residual quantiles for this example.
    iqr_threshold: 1.5  # Check sensitivity against healthy data and incidents.
    scale: [1.0, 1.0]  # Keep the learned ranges without extra widening.
    alpha: 0.005
    loss_reactivity: 3.0
    changepoint_window: 16
    min_n_samples_seen: 24  # One day of aligned hourly observations.
    random_state: 42  # Reproducible dependency features.
    clip_predictions: true
    provide_series: [anomaly_score, y, yhat, yhat_lower, yhat_upper]
schedulers:
  worker_relationship_demo:
    class: periodic  # Score current observations continuously.
    fit_window: 14d
    fit_every: 1000w  # Learning continues between full refits.
    infer_every: 1h
    

Replace the generated expressions with your completed-job rates and CPU query for production. Keep their units and aggregate all three to the same service and pool labels. The playground replays fixed historical dates using the same fit window and cadence as this periodic configuration. Hourly samples keep the demonstration readable. Use a finer cadence for shorter incidents, and revisit warmup and timing settings when changing it.

Group by the workload you investigate. groupby: [service, pool] keeps each pool’s job rates and CPU together. Leave query aliases, metric names, and job classes out of groupby. Add a cluster label if pool names repeat across clusters.

Keep inputs aligned. Use the same sampling step and aggregation scope. Inspect gaps, duplicate series, and label changes before interpreting scores. Monitor collection health separately.

Inspect the combination. Channel diagnostics help explain a joint score, even when no individual channel crosses its expected range. This recipe requests them through provide_series. Retain only [anomaly_score] if that is all your alerting needs.

Include normal work mixes. Fit on representative workloads. Enable supported seasonalities when calendar timing matters, and keep meaningful query-level business policies for each input.

• SRE: one of eight replicas uses more CPU than the others under comparable load.
• APM: one host has higher request latency than hosts serving similar work.
• Hardware: one disk runs hotter than comparable disks in the same operating conditions.

Separate different workloads, hardware classes, or regions into appropriate populations. A strict majority must behave normally. A shared failure across every peer may look normal to a peer comparison. Add a model that follows each metric over time when shared shifts also matter.

• Replicas of the same service with comparable traffic, resource limits, and deployment roles.
• Hosts of the same hardware class performing similar work.
• Disks, sensors, or other equipment operating under comparable conditions.

Normalize or separate different capacities and workloads before comparing them. A CPU ratio alone cannot account for different request costs or limits. Keep a strict healthy majority in each population.

A member’s own history, a relationship between signals, or a shared change matters more.

• All replicas shift together: add a per-series model or an impact alert. A whole service becoming slower or busier can leave the peer relationships unchanged.
• Members have different legitimate baselines: separate the populations using suitable labels, normalize the query, or use per-series models. Examples include different instance sizes, leader/follower roles, or deliberately uneven traffic.
• Several related metrics describe one entity: use a multivariate model to learn their relationship.
• Too few valid peers or no dependable healthy majority: improve the population definition or use another comparison. Missing collection data also needs its own monitoring.

Seven to eight replicas, one steady hot peer, and a spike across the healthy majority. Open the peer-group playground .

1. Select Execute Query. The named query replica_cpu returns peers labeled by service, region, and instance. Eight identities appear over the replay, with seven or eight present at a time.
2. Select Detect Anomalies. Use the peer-group investigation view to inspect checkout-08, which uses more CPU than its peers throughout the view, despite having a stable individual history.
3. Inspect January 15, 2026, 09:00 UTC, when a healthy checkout-07 joins. Its accepted range is available immediately from the existing group’s learned state. checkout-06 leaves at January 16, 00:00 UTC, while the remaining peers continue to be compared.
4. Inspect January 16, 10:00–13:00 UTC, when all healthy replicas rise together from about 50% to 65% CPU. checkout-08 stays near 80%. The accepted peer range should follow the majority, leaving the healthy replicas unflagged and the steady hot peer still unusual. Use a temporal model or service-level alert when this majority-wide change matters.

What to expect: one population model produces a separate anomaly-score series for each replica. checkout-08 should be flagged throughout, even while its distance from the majority shrinks during the spike. Other replicas, including the new member, should stay inside the accepted peer range. The range follows the majority at each timestamp. Values use ratios, so 0.8 means 80% CPU utilization.

Keep the writer and license configuration from the Quickstart . This periodic configuration runs continuously. The playground replays the fixed dates above with the same fit window and cadence.

      reader:
  datasource_url: https://play.victoriametrics.com/select/0/prometheus
  sampling_period: 15m
  queries:
    replica_cpu:  # One CPU-utilization series per comparable replica.
      expr: |
        WITH (
          mixed(seed) = sin(time() / 900 * 12.9898 + seed) * 43758.5453,
          noise(seed) = 0.004 * (mixed(seed) - floor(mixed(seed)) - 0.5),
          majority_load = 0.15 * (time() >= bool 1768557600) * (time() < bool 1768568400),
          level = 0.50 + 0.04 * sin(time() / 86400 * 6.283185307179586)
            + 0.015 * sin(time() / (5 * 3600)),
          peer(name, seed, offset) = label_set(level + noise(seed) + offset,
            "__name__", "demo_cpu_utilization_ratio", "instance", name,
            "service", "checkout", "region", "eu-west")
        )
        peer("checkout-01", 42, -0.015 + majority_load)
        or peer("checkout-02", 43, -0.010 + majority_load)
        or peer("checkout-03", 44, -0.005 + majority_load)
        or peer("checkout-04", 45, 0.000 + majority_load)
        or peer("checkout-05", 46, 0.005 + majority_load)
        or (peer("checkout-06", 47, 0.010 + majority_load) if on() (time() < 1768521600))
        or (peer("checkout-07", 48, 0.015 + majority_load) if on() (time() >= 1768467600))
        or peer("checkout-08", 49, 0.300)
      detection_direction: above_expected  # Investigate peers using unusually high CPU.
      min_dev_from_expected: [0.05, 0.05]  # Five percentage points from the expected peer value.
      data_range: [0, 1]  # CPU utilization expressed as a ratio.

models:
  peer_checkout_cpu:  # A model alias chosen for this recipe.
    class: peer_outlier
    queries: [replica_cpu]  # Connect the query defining this peer population.
    schedulers: [replica_peer_demo]  # Connect the schedule below.
    groupby: [service, region]  # Keep instance identity within the population.
    min_peer_count: 5  # Minimum aligned peers needed for a valid snapshot.
    epsilon_quantile: 0.75  # Learn a reference for the usual population spread.
    tolerance: 4.0  # Control how far a peer may lie from its normal neighbors.
    decay: 0.999  # Give recent population snapshots more influence.
    min_n_samples_seen: 32  # Effective valid snapshots for the group, independent of peer count.
    clip_predictions: true  # Keep the accepted range inside the query's data range.
    provide_series: [anomaly_score, y, yhat, yhat_lower, yhat_upper]  # Diagnostics for the demo.

schedulers:
  replica_peer_demo:
    class: periodic  # Compare current population snapshots continuously.
    fit_window: 1d
    fit_every: 1000w  # Learning continues between these very infrequent full refits.
    infer_every: 15m
    

For production, replace the generated expression with your CPU query. For example, use 1 - avg by (service, region, instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) if those labels exist and the instances perform comparable work. Preserve identity labels and aggregate duplicate inputs before comparing peers.

Keep peer identity out of the grouping labels. groupby: [service, region] defines a population for each service and region. The instance label identifies members inside it. Grouping by instance would split the replicas into separate populations. Each named query keeps its own populations, even when their grouping labels match.

Distinguish a new peer from a new population. A peer joining an existing, warmed-up group uses that group’s state. A new service or region needs its own fit and warmup. Here, min_n_samples_seen: 32 requires effective history from valid snapshots with at least five aligned peers. Decay discounts older snapshots. A snapshot with eight peers still contributes one observation. Gaps or too few peers delay learning and scoring, so monitor collection health separately.

Set the operational difference you care about. This recipe investigates high CPU and uses a five-percentage-point minimum distance from the expected peer value. A value must also cross the accepted peer range to be flagged. Use both or below_expected when unexpectedly low utilization matters too. Inspect healthy deployment and autoscaling periods before adjusting sensitivity with tolerance or epsilon_quantile. See the Peer Outlier reference .

Keep shared failures visible. The January 16 increase demonstrates a change affecting the healthy majority, while the unusual replica stays at its own level. Add a temporal model , a capacity rule, or an SLO alert alongside the peer comparison. Review the population when traffic routing, instance sizes, or deployment roles change.