VictoriaMetrics integrates with Prometheus as remote storage for writes.
To send data from Prometheus to VictoriaMetrics, add the following lines to your Prometheus config file:
remote_write:
- url: http://<victoriametrics-addr>:8428/api/v1/write
Replace <victoriametrics-addr>
with the VictoriaMetrics hostname or IP address.
For cluster version use vminsert address:
http://<vminsert-addr>:8480/insert/<tenant>/prometheus
Replace <vminsert-addr>
with the hostname or IP address of vminsert service.
If you have more than 1 vminsert, configure load-balancing.
Replace <tenant>
based on your multitenancy settings.
Then apply the new config restart Prometheus or hot-reload it:
kill -HUP `pidof prometheus`
Prometheus stores incoming data locally and also sends a copy to the remote storage.
This means even if the remote storage is down, data is still available locally for as long as set in --storage.tsdb.retention.time
.
If you send data from more than one Prometheus instance, add the following lines into global
section
of Prometheus config:
global:
external_labels:
datacenter: dc-123
This adds a label datacenter=dc-123
to each sample sent to remote storage.
You can use any label name; datacenter is just an example. The value should be different for each Prometheus instance
so you can filter and group time series by it.
For Prometheus servers handling a high load (200k+ samples per second), use these settings:
remote_write:
- url: http://<victoriametrics-addr>:8428/api/v1/write
queue_config:
max_samples_per_send: 10000
capacity: 20000
max_shards: 30
Using remote write can increase Prometheus memory use by around 25%. If memory usage is too high, try reducing
max_samples_per_send
and capacity
params. These two settings work closely together, so adjust carefully.
Read more about tuning remote write for Prometheus.
It is recommended upgrading Prometheus to v2.12.0 or newer,
since previous versions may have issues with remote_write
.
Take a look at vmagent and vmalert, which can be used as faster and less resource-hungry alternative to Prometheus.