Relabeling cookbook #

This /targets page helps answer the following questions:

1. Why are some targets not being scraped?

• The last error column shows the reason why a target is not being scraped.
• Click the endpoint link to open the target URL in your browser.
• Click the response link to view the response vmagent received from the target.

2. What labels does a specific target have?

The labels column shows the labels for each target. These labels are attached to all metrics scraped from that target.

You can click the label column of the target to see the original labels before any relabeling was applied.

This option is only available when the component is started with the -promscrape.dropOriginalLabels=false flag.

3. Why does a target have a certain set of labels?

Click the target link in the debug relabeling column. This opens a step-by-step view of how the relabeling rules were applied to the original labels.

This option is only available when the component is started with the -promscrape.dropOriginalLabels=false flag.

4. How are metric relabeling rules applied to scraped metrics?

Click the metrics link in the debug relabeling column. This shows how the metric relabeling rules were applied, step by step.

Each column on the page shows important details:

• state: shows if the target is currently up or down.
• scrapes: number of times the target was scraped.
• errors: number of failed scrapes.
• last scrape: when the last scrape happened.
• last scrape size: size of the last scrape.
• duration: time taken for the last scrape.
• samples: number of metrics exposed by the target during the last scrape.

This page shows all discovered targets .

This option is only available when the component is started with the -promscrape.dropOriginalLabels=false flag.

It helps answer the following questions:

1. Why are some targets dropped during service discovery or showing unexpected labels?

Click the debug link in the debug relabeling column for a dropped target. This opens a step-by-step view of how target relabeling rules were applied to that target’s original labels.

2. What were the original labels before relabeling?

The discovered labels column shows the original labels for each discovered target.

How to drop discovered targets #

To drop a particular discovered target, use the following options:

• action: drop: drops scrape targets with labels matching the if series selector
• action: keep: keeps scrape targets with labels matching the if series selector, while dropping all other targets

Here are examples of these options:

• This config discovers pods in Kubernetes and drops all pods with names starting with the test- prefix:

        scrape_configs:
    - job_name: prod_pods_only
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
        - if: '{__meta_kubernetes_pod_name=~"test-.*"}'
          action: drop
      

  Try the above config

• This config keeps only pods with names starting with the backend- prefix:

        scrape_configs:
    - job_name: backend_pods
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
        - if: '{__meta_kubernetes_pod_name=~"backend-.*"}'
          action: keep
      

  Try the above config

See also useful tips for target relabeling .

How to modify scrape URLs in targets #

URLs for scrape targets are composed of the following parts:

• Scheme (e.g. http, https) is available during target relabeling in a special label - __scheme__. By default, it’s set to http but can be overridden either by specifying the scheme option at scrape_config level or by updating the __scheme__ label during relabeling.
• Host and port (e.g. host12:3456) is available during target relabeling in a special label - __address__. Its value depends on the service discovery type . Sometimes this value needs to be modified. In this case, just update the __address__ label during relabeling to the needed value.
  • The port part is optional. If it is missing, it’s automatically set depending on the scheme (80 for http or 443 for https). The host:port part from the final __address__ label is automatically set to the instance label. The __address__ label can contain the full scrape URL (e.g. http://host:port/metrics/path?query_args). In this case the __scheme__ and __metrics_path__ labels are ignored.
• URL path (e.g. /metrics) is available during target relabeling in a special label - __metrics_path__. By default, it’s set to /metrics and can be overridden either by specifying the metrics_path option at scrape_config level or by updating the __metrics_path__ label during relabeling.
• Query args (e.g. ?foo=bar&baz=xyz) are available during target relabeling in special labels with the __param_ prefix.
  • Take ?foo=bar&baz=xyz for example. There will be two special labels: __param_foo="bar" and __param_baz="xyz". The query args can be specified either via the params section at scrape_config or by updating/setting the corresponding __param_* labels during relabeling.

The resulting scrape URL looks like the following:

      <__scheme__> + "://" + <__address__> + <__metrics_path__> + <"?" + query_args_from_param_labels>
    

Given the scrape URL construction rules above, the following config discovers pod targets in Kubernetes and constructs a per-target scrape URL as https://<pod_name>/metrics/container?name=<container_name>:

      scrape_configs:
  - job_name: k8s
    kubernetes_sd_configs:
      - role: pod
    metrics_path: /metrics/container
    relabel_configs:
      - target_label: __scheme__
        replacement: https
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: __address__
      - source_labels: [__meta_kubernetes_pod_container_name]
        target_label: __param_name
    

Try the above config

How to remove labels from targets #

To remove some labels from targets discovered by the scrape job, use either:

• action: labeldrop: drops labels with names matching the given regex option
• action: labelkeep: drops labels with names not matching the given regex option

For example:

      scrape_configs:
  - job_name: k8s
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - action: labelmap
        regex: "__meta_kubernetes_pod_label_(.+)"
        replacement: "pod_label_$1"
      - action: labeldrop
        regex: "pod_label_team_.*"
    

Try the above config

The job above will:

1. discover pods in Kubernetes
2. extract pod-level labels (e.g. app.kubernetes.io/name and team)
3. prefix them with pod_label_ and add them as labels to all scraped metrics
4. drop all labels starting with pod_label_team_
5. drop all labels starting with __ (this is done by default by VictoriaMetrics)

Note that:

• Labels that start with __ are removed automatically after relabeling, so you don’t need to drop them with relabeling rules.
• Do not remove instance and job labels, since this may result in duplicate scrape targets with identical sets of labels.
• The regex option must match the whole label name from start to end, not just a part of it.

How to remove labels from a subset of targets #

To remove some labels from a subset of discovered targets while keeping the rest of the targets unchanged, use the if series selector with action: labeldrop or action: labelkeep relabeling rule.

As an illustration:

• The job below discovers Kubernetes pods and removes any labels starting with pod_internal_ but only for targets matching the {__address__=~"pod123.+"} selector

      scrape_configs:
  - job_name: k8s
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - action: labeldrop
        if: '{__address__=~"pod123.+"}'
        regex: "pod_internal_.*"
    

Try the above config

How to remove prefixes from target label names #

You can modify target-labels including removing prefixes with the action: labelmap option.

For example, Kubernetes service discovery automatically adds special __meta_kubernetes_pod_label_<labelname> labels for each pod-level label.

All labels with the prefix __ will be dropped automatically. To extract and keep only the <labelname> part of this special label, you can use action: labelmap combined with regex and replacement options:

      scrape_configs:
  - job_name: k8s
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - action: labelmap
        regex: "__meta_kubernetes_pod_label_(.+)"
        replacement: "$1"
    

Try the above config

The regex contains a capture group (.+). This capture group can be referenced inside the replacement option with the $N syntax, such as $1 for the first capture group.

This config will create a new label with the name extracted from the regex capture group (.+) for all metrics scraped from the discovered pods.

Note that:

• The regex option must match the whole label name from start to end, not just a part of it.

How to extract label parts #

Relabeling allows extracting parts from label values and storing them into arbitrary labels. This is performed with:

• source_labels: the label(s) whose values are used to compute the new value for target_label,
• target_label: the label we want to modify or create,
• replacement: the value that will be computed and assigned to the target_label,
• regex: the regular expression to be applied to the value of source_labels.

Let’s take this case:

      scrape_configs:
  - job_name: k8s
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_container_name]
        regex: "[^/]+/(.+)"
        replacement: "team_$1"
        target_label: owner_team
    

Try the above config

The job above discovers pod targets in Kubernetes , and performs these actions:

1. Extracts the value of __meta_kubernetes_pod_container_name label (e.g. foo/bar),
2. Matches it against the regex [^/]+/(.+),
3. Computes the new value as team_$1 with $1 capture from regex (.+),
4. Stores the result in the owner_team label.

Note that:

• The regex option must match the whole label value from start to end, not just a part of it.
• If source_labels contains multiple labels, their values are joined with a ; separator (customized by the separator option) before being matched against the regex.

How to add labels to scrape targets #

To add or update labels on scrape targets during discovery, use these options:

• target_label: specifies the label name to add or update
• replacement: specifies the value to assign to this label

For example, this config adds a environment="production" label to all discovered pods in Kubernetes :

      scrape_configs:
  - job_name: k8s
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - target_label: "environment"
        replacement: "production"
    

Try the above config

If a label from the scrape configuration (target_label) conflicts with a label from the scraped metric (scrape-time label), the original scrape-time label is renamed by adding an exported_ prefix.

To avoid this renaming and instead let the scrape-time labels take priority (overriding target labels), set honor_labels: true in the scrape configuration.

For example, this config adds a environment="production" label to all discovered pods, but if any pod already exports a environment label, that value will override the target label:

      scrape_configs:
  - job_name: k8s
    kubernetes_sd_configs:
      - role: pod
    honor_labels: true
    relabel_configs:
      - target_label: "environment"
        replacement: "production"
    

See also useful tips for target relabeling .

How to copy labels in scrape targets #

Labels can be copied using the following options:

• source_labels: specifies which labels to copy from
• target_label: specifies the destination label to receive the value

The following config copies the __meta_kubernetes_pod_name label to the pod label for all discovered pods in Kubernetes :

      scrape_configs:
  - job_name: k8s
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
    

Try the above config

If source_labels contains multiple labels, their values are joined with a ; delimiter by default. Use the separator option to change this delimiter.

For example, this config combines pod name and container port into the host_port label for all discovered pod targets in Kubernetes :

      scrape_configs:
  - job_name: k8s
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels:
          [
            __meta_kubernetes_pod_name,
            __meta_kubernetes_pod_container_port_number,
          ]
        separator: ":"
        target_label: host_port
    

Try the above config

How to modify instance and job #

instance and job labels are automatically added by single-node VictoriaMetrics and vmagent for each discovered target.

• The job label is set to the job_name value specified in the corresponding scrape_config.
• The instance label is set to the host:port part of the __address__ label value after target-level relabeling. The __address__ label value depends on the type of service discovery and can be overridden during relabeling.

Modifying instance and job labels works like other target-labels by using target_label and replacement options:

      scrape_configs:
  - job_name: k8s
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - target_label: job
        replacement: kubernetes_pod_metrics
    

Try the above config

How to remove labels from scraped metrics #

Removing labels from scraped metrics is a good idea to avoid high cardinality and high churn rate issues.

This can be done with either of the following actions:

• action: labeldrop: drops labels with names matching the given regex option
• action: labelkeep: drops labels with names not matching the given regex option

Let’s see this in action:

• Remove labels with names starting with the kubernetes_ prefix from all scraped metrics:

        metric_relabel_configs:
    - action: labeldrop
      regex: "kubernetes_.*"
      

  Try the above config

The regex option must match the whole label name from start to end, not just a part of it.

Note that:

• Labels that start with __ are removed automatically after relabeling, so you don’t need to drop them with relabeling rules.

How to remove labels from metrics subset #

You can remove certain labels from some metrics without affecting other metrics by using the if parameter with labeldrop action. The if parameter is a series selector - it looks at the metric name and labels of each scraped time series.

For instance, this config below removes the cpu and mode labels, but only from the node_cpu_seconds_total metric where mode="idle":

      metric_relabel_configs:
  - action: labeldrop
    if: 'node_cpu_seconds_total{mode="idle"}'
    regex: "cpu|mode"
    

Try the above config

How to add labels to scraped metrics #

You can add custom labels to scraped metrics using target_label to set the label name and the replacement field to set the label value. For example:

• Add a region="us-east-1" label to all scraped metrics:

        metric_relabel_configs:
    - target_label: region
      replacement: us-east-1
      

  Try the above config

• Add a team="platform" label only for metrics from jobs that match web-.* and are not in the staging environment :

        metric_relabel_configs:
    - if: '{job=~"web-.*", environment!="staging"}'
      target_label: team
      replacement: platform
      

  Try the above config

How to change label values in scraped metrics #

To change the label values of scraped metrics, we use the following fields:

• target_label: the label we want to modify (if it exists) or create,
• source_labels: the label(s) whose values are used to compute the new value for target_label,
• replacement: the value that will be computed and assigned to the target_label.

Below are a few illustrations:

• Add prod_ prefix to all values of the job label across all scraped metrics:

        metric_relabel_configs:
    - source_labels: [job]
      target_label: job
      replacement: prod_$1
      

  Try the above config

• Add prod_ prefix to job label values only for metrics matching {job=~"api-service-.*",env!="dev"}:

        metric_relabel_configs:
    - if: '{job=~"api-service-.*",env!="dev"}'
      source_labels: [job]
      target_label: job
      replacement: prod_$1
      

  Try the above config

How to rename scraped metrics #

The metric name is actually the value of a special label called __name__ (see Key Concepts ). So renaming a metric is performed in the same way as changing a label value. Let’s take some examples:

• Rename node_cpu_seconds_total to vm_node_cpu_seconds_total across all the scraped metrics:

        metric_relabel_configs:
    - if: "node_cpu_seconds_total"
      replacement: vm_node_cpu_seconds_total
      target_label: __name__
      

  Try the above config

• Rename all metrics starting with http_ to start with web_ instead (e.g. http_requests_total → web_requests_total):

        metric_relabel_configs:
    - source_labels: [__name__]
      regex: "http_(.*)"
      replacement: web_$1
      target_label: __name__
      

  Try the above config

• Replace all dashes (-) in metric names with underscores (_) (e.g. nginx-ingress-latency → nginx_ingress_latency):

        metric_relabel_configs:
    - source_labels: [__name__]
      action: replace_all
      regex: "-"
      replacement: "_"
      target_label: __name__
      

  Try the above config

How to drop metrics during scrape #

All examples above work at the label level: adding, dropping, or changing label values of scraped metrics. You can also drop entire metrics. This is especially beneficial for metrics that result in high cardinality or high churn rate .

Instead of labeldrop or labelkeep actions, we use drop or keep actions in the metric_relabel_configs section:

• action: drop: drops all metrics that match the if series selector
• action: keep: drops all metrics that don’t match the if series selector

For example, the following config drops all metrics with names starting with container_:

      metric_relabel_configs:
  - if: '{__name__=~"container_.*"}'
    action: drop
    

Try the above config

Note that the relabeling config is specified under the metric_relabel_configs section instead of relabel_configs section. They serve different purposes:

• The scrape_configs[].relabel_configs apply before scraping, modifying or filtering targets. Any changes here affect all metrics from that target.
• The scrape_configs[].metric_relabel_configs apply after scraping, modifying or filtering individual metrics.