Prometheus #

VictoriaMetrics integrates with Prometheus as remote storage for writes.

To send data from Prometheus to VictoriaMetrics, add the following lines to your Prometheus config file:

      remote_write:
  - url: http://<victoriametrics-addr>:8428/api/v1/write
    

Replace <victoriametrics-addr> with the VictoriaMetrics hostname or IP address.

For cluster version use vminsert address:

      http://<vminsert-addr>:8480/insert/<tenant>/prometheus
    

Replace <vminsert-addr> with the hostname or IP address of vminsert service.

If you have more than 1 vminsert, configure load-balancing. Replace <tenant> based on your multitenancy settings.

Then apply the new config restart Prometheus or hot-reload it:

      kill -HUP `pidof prometheus`
    

Prometheus stores incoming data locally and also sends a copy to the remote storage. This means even if the remote storage is down, data is still available locally for as long as set in --storage.tsdb.retention.time.

If you send data from more than one Prometheus instance, add the following lines into global section of Prometheus config:

      global:
  external_labels:
    datacenter: dc-123
    

This adds a label datacenter=dc-123 to each sample sent to remote storage. You can use any label name; datacenter is just an example. The value should be different for each Prometheus instance so you can filter and group time series by it.

For Prometheus servers handling a high load (200k+ samples per second), use these settings:

      remote_write:
  - url: http://<victoriametrics-addr>:8428/api/v1/write
    queue_config:
      max_samples_per_send: 10000
      capacity: 20000
      max_shards: 30
    

Using remote write can increase Prometheus memory use by around 25%. If memory usage is too high, try reducing max_samples_per_send and capacity params. These two settings work closely together, so adjust carefully. Read more about tuning remote write for Prometheus.

It is recommended upgrading Prometheus to v2.12.0 or newer, since previous versions may have issues with remote_write.

Take a look at vmagent and vmalert, which can be used as faster and less resource-hungry alternative to Prometheus.