Graphite #

VictoriaMetrics components like vmagent, vminsert or single-node can receive inserts from Graphite-compatible agents (such as StatsD). VictoriaMetrics single-node and vmselect support Graphite query language. It can be integrated with Grafana via Graphite datasource. See How Grammarly Improved Monitoring by Over 10x with VictoriaMetrics by switching from Graphite and keeping all the same Grafana dashboards and agents.

See full list of Graphite-related configuration flags by running:

      /path/to/victoria-metrics-prod --help | grep graphite

Ingesting #

Enable Graphite receiver by setting -graphiteListenAddr command line flag:

      /path/to/victoria-metrics-prod -graphiteListenAddr=:2003

Now, VictoriaMetrics host name and specified port can be used as destination address in Graphite-compatible agents (e.g. graphiteHost in StatsD).

Try writing a data sample via Graphite plaintext protocol to local VictoriaMetrics using nc:

      echo "foo.bar.baz;tag1=value1;tag2=value2 123 `date +%s`" | nc -N localhost 2003
    

Metrics can be sanitized during ingestion according to Prometheus naming convention by passing -graphite.sanitizeMetricName command-line flag to VictoriaMetrics. The following modifications are applied to the ingested samples when this flag is passed to VictoriaMetrics:

remove redundant dots, e.g: metric..name => metric.name
replace characters not matching a-zA-Z0-9:_. chars with _

VictoriaMetrics sets the current time to the ingested samples if the timestamp is omitted. An arbitrary number of lines delimited by \n (aka newline char) can be sent in one go.

VictoriaMetrics single-node or vmselect can read the ingested data back. Try reading the data via /api/v1/export endpoint:

      curl -G 'http://localhost:8428/api/v1/export' -d 'match=foo.bar.baz'
    

Note, we’re using :8428 port here, as it is default port where VictoriaMetrics single-node listens for user requests.

The /api/v1/export endpoint should return the following response:

      {"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277406000]}
    

Querying #

VictoriaMetrics single-node or vmselect support the following query APIs:

Selecting Graphite metrics #

VictoriaMetrics supports __graphite__ pseudo-label for selecting time series with Graphite-compatible filters in MetricsQL. For example, {__graphite__="foo.*.bar"} is equivalent to {__name__=~"foo[.][^.]*[.]bar"}, but works faster and is easier to use when migrating from Graphite to VictoriaMetrics. See docs for Graphite paths and wildcards. VictoriaMetrics also supports label_graphite_group function for extracting the given groups from Graphite metric name.

The __graphite__ pseudo-label supports e.g. alternate regexp filters such as (value1|...|valueN). They are transparently converted to {value1,...,valueN} syntax used in Graphite. This allows using multi-value template variables in Grafana inside __graphite__ pseudo-label. For example, Grafana expands {__graphite__=~"foo.($bar).baz"} into {__graphite__=~"foo.(x|y).baz"} if $bar template variable contains x and y values. In this case the query is automatically converted into {__graphite__=~"foo.{x,y}.baz"} before execution.

VictoriaMetrics also supports Graphite query language - see Render API.

Graphite API usage #

To integrate with Graphite datasource in Grafana, VictoriaMetrics supports the following Graphite querying APIs:

All Graphite handlers can be prepended with /graphite prefix. For example, both /graphite/metrics/find and /metrics/find should work.

VictoriaMetrics accepts optional query args: extra_label=<label_name>=<label_value> and extra_filters[]=series_selector for all the Graphite APIs. These args can be used for limiting the scope of time series visible to the given tenant. It is expected that the extra_label query arg is automatically set by auth proxy sitting in front of VictoriaMetrics. See vmauth and vmgateway as examples of such proxies.

VictoriaMetrics supports __graphite__ pseudo-label for filtering time series with Graphite-compatible filters in MetricsQL. See Selecting Graphite metrics.

Render API #

VictoriaMetrics supports Graphite Render API subset at /render endpoint, which is used by Graphite datasource in Grafana. When configuring Graphite datasource in Grafana, the Storage-Step HTTP request header must be set to a step between Graphite data points stored in VictoriaMetrics. For example, Storage-Step: 10s would mean 10 seconds distance between Graphite datapoints stored in VictoriaMetrics.

Known Incompatibilities with `graphite-web` #

Timestamp Shifting: VictoriaMetrics does not support shifting response timestamps outside the request time range as graphite-web does. This limitation impacts chained functions with time modifiers, such as timeShift(summarize). For more details, refer to this issue.
Non-deterministic series order: due to the distributed nature of metrics processing, functions within the seriesLists family can produce non-deterministic results. To ensure consistent results, arguments for these functions must be wrapped with a sorting function. For instance, the function divideSeriesLists(series_list_1, series_list_2) should be modified to divideSeriesLists(sortByName(series_list_1), sortByName(series_list_2)). See this issue for details. The affected functions include:
- aggregateSeriesLists
- diffSeriesLists
- multiplySeriesLists
- divideSeriesLists

Metrics API #

VictoriaMetrics supports the following handlers from Graphite Metrics API:

VictoriaMetrics accepts the following additional query args at /metrics/find and /metrics/expand:

label - for selecting arbitrary label values. By default, label=__name__, i.e. metric names are selected.
delimiter - for using different delimiters in metric name hierarchy. For example, /metrics/find?delimiter=_&query=node_* would return all the metric name prefixes that start with node_. By default delimiter=..

Tags API #

VictoriaMetrics supports the following handlers from Graphite Tags API:

Previous Prometheus Next Datadog