vmctl supports remote-read mode for migrating data from remote databases that support Prometheus remote read API .

vmctl remote-read allows migrating data from the following systems:

Remote read API has two implementations of remote read API: default (SAMPLES) and streamed (STREAMED_XOR_CHUNKS). Streamed version is more efficient but has lower adoption.

See ./vmctl remote-read --help for details and the full list of flags.

Note

Migration via remote read protocol allows to fetch data via API. This is usually a resource intensive operation for Thanos and may be slow or expensive in terms of resources.

The importing process example for local installation of Prometheus:

      ./vmctl remote-read \
--remote-read-src-addr=http://<prometheus>:9091 \
--remote-read-filter-time-start=2021-10-18T00:00:00Z \
--remote-read-step-interval=hour \
--vm-addr=http://<victoria-metrics>:8428 \
    

See how to configure –vm-addr .

Filtering #

Filtering by time can be configured via flags --remote-read-filter-time-start and --remote-read-filter-time-end in RFC3339 format.

Filtering by labels can be configured via flags --remote-read-filter-label and --remote-read-filter-label-value. For example, --remote-read-filter-label=tenant and --remote-read-filter-label-value="team-eu" will select only series with tenant="team-eu" label-value pair.

Filtering flags can be provided multiple times Available from v1.129.0 to narrow down the selection of timeseries to migrate. For example:

      ./vmctl remote-read \
    --remote-read-filter-label=tenant --remote-read-filter-label-value="team-eu" \
    --remote-read-filter-label=__name__ --remote-read-filter-label-value="cpu_.*"
    

will select only timeseries with tenant="team-eu" label and metric names matching cpu_.* regex.

Configuration #

Migrating big volumes of data may result in remote read client reaching the timeout. Increase the value of --remote-read-http-timeout (default 5m) command-line flag when seeing timeouts or context canceled errors.

Flag --remote-read-step-interval allows splitting export data into chunks to reduce pressure on the source --remote-read-src-addr. Valid values are month, day, hour, minute.

Flag --remote-read-use-stream defines whether to use SAMPLES or STREAMED_XOR_CHUNKS mode. Mode STREAMED_XOR_CHUNKS is much less resource intensive for the source, but not many databases support it. By default, is uses SAMPLES mode.

See general vmctl migration tips .

See ./vmctl remote-read --help for details and full list of flags:

      NAME:
   vmctl remote-read - Migrate time series via Prometheus remote-read protocol

USAGE:
   vmctl remote-read [command options]

OPTIONS:
   -s                                                                                 Whether to run in silent mode. If set to true no confirmation prompts will appear. (default: false)
   --verbose                                                                          Whether to enable verbosity in logs output. (default: false)
   --disable-progress-bar                                                             Whether to disable progress bar during the import. (default: false)
   --pushmetrics.url value [ --pushmetrics.url value ]                                Optional URL to push metrics. See https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#push-metrics
   --pushmetrics.interval value                                                       Interval for pushing metrics to every -pushmetrics.url (default: 10s)
   --pushmetrics.extraLabel value [ --pushmetrics.extraLabel value ]                  Extra labels to add to pushed metrics. In case of collision, label value defined by flag will have priority. Flag can be set multiple times, to add few additional labels. For example, -pushmetrics.extraLabel='instance="foo"' adds instance="foo" label to all the metrics pushed to every -pushmetrics.url
   --pushmetrics.header value [ --pushmetrics.header value ]                          Optional HTTP headers to add to pushed metrics. Flag can be set multiple times, to add few additional headers.
   --pushmetrics.disableCompression                                                   Whether to disable compression when pushing metrics. (default: false)
   --remote-read-concurrency value                                                    Number of concurrently running remote read readers (default: 1)
   --remote-read-filter-time-start value                                              The time filter in RFC3339 format to select timeseries with timestamp equal or higher than provided value. E.g. '2020-01-01T20:07:00Z'
   --remote-read-filter-time-end value                                                The time filter in RFC3339 format to select timeseries with timestamp equal or lower than provided value. E.g. '2020-01-01T20:07:00Z'
   --remote-read-filter-label value [ --remote-read-filter-label value ]              Prometheus label name to filter timeseries by. E.g. '__name__' will filter timeseries by name. (default: __name__)
   --remote-read-filter-label-value value [ --remote-read-filter-label-value value ]  Prometheus regular expression to filter label from "remote-read-filter-label-value" flag. (default: .*)
   --remote-read                                                                      Use Prometheus remote read protocol (default: false)
   --remote-read-use-stream                                                           Defines whether to use SAMPLES or STREAMED_XOR_CHUNKS mode. By default, is uses SAMPLES mode. See https://prometheus.io/docs/prometheus/latest/querying/remote_read_api/#streamed-chunks (default: false)
   --remote-read-step-interval value                                                  The time interval to split the migration into steps. For example, to migrate 1y of data with '--remote-read-step-interval=month' vmctl will execute it in 12 separate requests from the beginning of the time range to its end. To reverse the order use '--remote-read-filter-time-reverse'. Requires setting '--remote-read-filter-time-start'. Valid values are 'month','week','day','hour','minute'.
   --remote-read-filter-time-reverse                                                  Whether to reverse the order of time intervals split by '--remote-read-step-interval' cmd-line flag. When set, the migration will start from the newest to the oldest data. (default: false)
   --remote-read-src-addr value                                                       Remote read address to perform read from.
   --remote-read-user value                                                           Remote read username for basic auth [$REMOTE_READ_USERNAME]
   --remote-read-password value                                                       Remote read password for basic auth [$REMOTE_READ_PASSWORD]
   --remote-read-http-timeout value                                                   Timeout defines timeout for HTTP requests made by remote read client (default: 0s)
   --remote-read-headers value                                                        Optional HTTP headers to send with each request to the corresponding remote source storage 
      For example, --remote-read-headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding remote source storage. 
      Multiple headers must be delimited by '^^': --remote-read-headers='header1:value1^^header2:value2'
   --remote-read-cert-file value       Optional path to client-side TLS certificate file to use when connecting to -remote-read-src-addr
   --remote-read-key-file value        Optional path to client-side TLS key to use when connecting to -remote-read-src-addr
   --remote-read-CA-file value         Optional path to TLS CA file to use for verifying connections to -remote-read-src-addr. By default, system CA is used
   --remote-read-server-name value     Optional TLS server name to use for connections to remoteReadSrcAddr. By default, the server name from -remote-read-src-addr is used
   --remote-read-insecure-skip-verify  Whether to skip TLS certificate verification when connecting to the remote read address (default: false)
   --remote-read-disable-path-append   Whether to disable automatic appending of the /api/v1/read suffix to --remote-read-src-addr (default: false)
   --vm-addr value                     VictoriaMetrics address to perform import requests. 
      Should be the same as --httpListenAddr value for single-node version or vminsert component. 
      When importing into the clustered version do not forget to set additionally --vm-account-id flag. 
      Please note, that vmctl performs initial readiness check for the given address by checking /health endpoint. (default: "http://localhost:8428")
   --vm-user value        VictoriaMetrics username for basic auth [$VM_USERNAME]
   --vm-password value    VictoriaMetrics password for basic auth [$VM_PASSWORD]
   --vm-account-id value  AccountID is an arbitrary 32-bit integer identifying namespace for data ingestion (aka tenant). 
      AccountID is required when importing into the clustered version of VictoriaMetrics. 
      It is possible to set it as accountID:projectID, where projectID is also arbitrary 32-bit integer. 
      If projectID isn't set, then it equals to 0
   --vm-concurrency value                             Number of workers concurrently performing import requests to VM (default: 2)
   --vm-compress                                      Whether to apply gzip compression to import requests (default: true)
   --vm-batch-size value                              How many samples importer collects before sending the import request to VM (default: 200000)
   --vm-significant-figures value                     The number of significant figures to leave in metric values before importing. See https://en.wikipedia.org/wiki/Significant_figures. Zero value saves all the significant figures. This option may be used for increasing on-disk compression level for the stored metrics. See also --vm-round-digits option (default: 0)
   --vm-round-digits value                            Round metric values to the given number of decimal digits after the point. This option may be used for increasing on-disk compression level for the stored metrics (default: 100)
   --vm-extra-label value [ --vm-extra-label value ]  Extra labels, that will be added to imported timeseries. In case of collision, label value defined by flag will have priority. Flag can be set multiple times, to add few additional labels.
   --vm-rate-limit value                              Optional data transfer rate limit in bytes per second.
      By default, the rate limit is disabled. It can be useful for limiting load on configured via '--vm-addr' destination. (default: 0)
   --vm-cert-file value             Optional path to client-side TLS certificate file to use when connecting to '--vm-addr'
   --vm-key-file value              Optional path to client-side TLS key to use when connecting to '--vm-addr'
   --vm-CA-file value               Optional path to TLS CA file to use for verifying connections to '--vm-addr'. By default, system CA is used
   --vm-server-name value           Optional TLS server name to use for connections to '--vm-addr'. By default, the server name from '--vm-addr' is used
   --vm-insecure-skip-verify        Whether to skip tls verification when connecting to '--vm-addr' (default: false)
   --vm-backoff-retries value       How many import retries to perform before giving up. (default: 10)
   --vm-backoff-factor value        Factor to multiply the base duration after each failed import retry. Must be greater than 1.0 (default: 1.8)
   --vm-backoff-min-duration value  Minimum duration to wait before the first import retry. Each subsequent import retry will be multiplied by the '--vm-backoff-factor'. (default: 2s)
   --help, -h                       show help