Using Grafana with vmgateway is a great way to provide multi-tenant access to your metrics. vmgateway provides a way to authenticate users using JWT tokens issued by an external identity provider. Those tokens can include information about the user and the tenant they belong to, which can be used to restrict access to metrics to only those that belong to the tenant.
Prerequisites #
- Identity service that can issue JWT tokens
- Grafana
- VictoriaMetrics single-node or cluster version
- vmgateway
- An active license key. You can obtain a trial license key here.
Configure identity service #
The identity service must be able to issue JWT tokens with the following vm_access
claim:
{
"vm_access": {
"tenant_id": {
"account_id": 0,
"project_id": 0
}
}
}
See details about all supported options in the vmgateway documentation.
Configuration example for Keycloak #
Keycloak is an open source identity service that can be used to issue JWT tokens.
Log in with admin credentials to your Keycloak instance
Go to
Clients
->Create
.
UseOpenID Connect
asClient Type
.
Specifygrafana
asClient ID
.
ClickNext
.Enable
Client authentication
.
EnableAuthorization
.
ClickNext
.Add Grafana URL as
Root URL
. For example,http://localhost:3000/
.
ClickSave
.Go to
Clients
->grafana
->Credentials
.
Copy the value ofClient secret
. It will be used later in Grafana configuration.Go to
Clients
->grafana
->Client scopes
.
Click atgrafana-dedicated
->Add mapper
->By configuration
->User attribute
.
Configure the mapper as followsName
asvm_access
.Token Claim Name
asvm_access
.User Attribute
asvm_access
.Claim JSON Type
asJSON
. EnableAdd to ID token
andAdd to access token
.
Go to
Users
-> select user to configure claims ->Attributes
.
Specifyvm_access
asKey
.
For the purpose of this example, we will use 2 users:
Configure grafana #
To forward JWT tokens Grafana must be configured to use OpenID Connect authentication as follows:
[auth.generic_oauth]
enabled = true
allow_sign_up = true
name = keycloak
client_id = {CLIENT_ID_FROM_IDENTITY_PROVIDER}
client_secret = {SECRET_FROM_IDENTITY_PROVIDER}
scopes = openid profile email
auth_url = http://localhost:3001/realms/{KEYCLOAK_REALM}/protocol/openid-connect/auth
token_url = http://localhost:3001/realms/{KEYCLOAK_REALM}/protocol/openid-connect/token
api_url = http://localhost:3001/realms/{KEYCLOAK_REALM}/protocol/openid-connect/userinfo
After restarting Grafana with the new config you should be able to log in using your identity provider.
Start vmgateway #
Multi-tenant access for VictoriaMetrics cluster #
Now starting vmgateway with enabled authentication is as simple as adding the -enable.auth=true
flag.
In order to enable multi-tenant access, you must also specify the -clusterMode=true
flag.
./bin/vmgateway \
-licenseFile=./vm-license.key
-enable.auth=true \
-clusterMode=true \
-write.url=http://localhost:8480 \
-read.url=http://localhost:8481
With this configuration vmgateway will use the vm_access
claim from the JWT token to restrict access to metrics.
For example, if the JWT token contains the following vm_access
claim:
{
"vm_access": {
"tenant_id": {
"account_id": 0,
"project_id": 0
}
}
}
Note: in case
project_id
is not specified, default value0
is used.
Then vmgateway will proxy request to an endpoint with the following path:
http://localhost:8480/select/0:0/
This allows to restrict access to specific tenants without having to create separate datasources in Grafana, or manually managing access at another proxy level.
Multi-tenant access for single-node VictoriaMetrics #
In order to use multi-tenant access with single-node VictoriaMetrics, you can use token claims such as extra_labels
or extra_filters
filled dynamically by using Identity Provider’s user information.
vmgateway uses those claims and enhanced Prometheus querying API
to provide additional filtering capabilities.
For example, the following claims can be used to restrict user access to specific metrics:
{
"vm_access": {
"extra_labels": {
"team": "dev"
},
"extra_filters": ["{env=~\"aws|gcp\",cluster!=\"production\"}"]
}
}
This will add the following query args to the proxied request:
extra_labels=team=dev
extra_filters={env=~"aws|gcp",cluster!="production"}
With this configuration VictoriaMetrics will add the following filters to every query: {team="dev", env=~"aws|gcp", cluster!="production"}
.
So when user will try to query vm_http_requests_total
query will be transformed to vm_http_requests_total{team="dev", env=~"aws|gcp", cluster!="production"}
.
Token signature verification #
It is also possible to enable JWT token signature verification at
vmgateway.
To do this by using OpenID Connect discovery endpoint you need to specify the -auth.oidcDiscoveryEndpoints
flag. For example:
./bin/vmgateway \
-licenseFile=./vm-license.key
-enable.auth=true \
-clusterMode=true \
-write.url=http://localhost:8480 \
-read.url=http://localhost:8481
-auth.oidcDiscoveryEndpoints=http://localhost:3001/realms/master/.well-known/openid-configuration
Now vmgateway will print the following message on startup:
2023-03-13T14:45:31.552Z info VictoriaMetrics/app/vmgateway/main.go:154 using 2 keys for JWT token signature verification
That means that vmgateway has successfully fetched the public keys from the OpenID Connect discovery endpoint.
It is also possible to provide the public keys directly via the -auth.publicKeys
flag. See the vmgateway documentation for details.
Use Grafana to query metrics #
Create a new Prometheus datasource in Grafana with the following URL http://<vmgateway>:8431
.
URL should point to the vmgateway instance.
In the “Type and version” section it is recommended to set the type to “Prometheus” and the version to at least “2.24.x”:
This allows Grafana to use a more efficient API to get label values.
You can also use VictoriaMetrics Grafana datasource plugin. See installation instructions here.
Enable Forward OAuth identity
flag.
Now you can use Grafana to query metrics from the specified tenant.
Users with vm_access
claim will be able to query metrics from the specified tenant.
Test multi-tenant access #
For the test purpose we will setup the following services as docker-compose manifest:
- Grafana
- Keycloak
- vmagent to generate test metrics
- VictoriaMetrics cluster
- vmgateway configured to work in cluster mode
- VictoriaMetrics single node
- vmgateway configured to work in single node mode
version: '3'
services:
keycloak:
image: quay.io/keycloak/keycloak:21.0
command:
- start-dev
ports:
- 3001:8080
environment:
KEYCLOAK_ADMIN: admin
KEYCLOAK_ADMIN_PASSWORD: change_me
grafana:
image: grafana/grafana:10.4.2
network_mode: host
volumes:
- ./grafana.ini:/etc/grafana/grafana.ini
- grafana_data:/var/lib/grafana/
vmsingle:
image: victoriametrics/victoria-metrics:v1.106.0
command:
- -httpListenAddr=0.0.0.0:8429
vmstorage:
image: victoriametrics/vmstorage:v1.106.0-cluster
vminsert:
image: victoriametrics/vminsert:v1.106.0-cluster
command:
- -storageNode=vmstorage:8400
- -httpListenAddr=0.0.0.0:8480
vmselect:
image: victoriametrics/vmselect:v1.106.0-cluster
command:
- -storageNode=vmstorage:8401
- -httpListenAddr=0.0.0.0:8481
vmagent:
image: victoriametrics/vmagent:v1.106.0
volumes:
- ./scrape.yaml:/etc/vmagent/config.yaml
command:
- -promscrape.config=/etc/vmagent/config.yaml
- -remoteWrite.url=http://vminsert:8480/insert/0/prometheus/api/v1/write
- -remoteWrite.url=http://vmsingle:8429/api/v1/write
vmgateway-cluster:
image: victoriametrics/vmgateway:v1.106.0-enterprise
ports:
- 8431:8431
volumes:
- ./vm-license.key:/opt/vm-license.key
command:
- -licenseFile=/opt/vm-license.key
- -license.forceOffline=true
- -enable.auth=true
- -clusterMode=true
- -write.url=http://vminsert:8480
- -read.url=http://vmselect:8481
- -httpListenAddr=0.0.0.0:8431
- -auth.oidcDiscoveryEndpoints=http://keycloak:8080/realms/master/.well-known/openid-configuration
vmgateway-single:
image: victoriametrics/vmgateway:v1.106.0-enterprise
ports:
- 8432:8431
volumes:
- ./vm-license.key:/opt/vm-license.key
command:
- -licenseFile=/opt/vm-license.key
- -enable.auth=true
- -write.url=http://vmsingle:8429
- -read.url=http://vmsingle:8429
- -httpListenAddr=0.0.0.0:8431
- -auth.oidcDiscoveryEndpoints=http://keycloak:8080/realms/master/.well-known/openid-configuration
volumes:
grafana_data:
For the test purpose vmagent will be configured to scrape metrics from the following targets(scrape.yaml
contents):
scrape_configs:
- job_name: stat
metric_relabel_configs:
- if: "{instance =~ 'vmgateway.*'}"
action: replace
target_label: team
replacement: admin
- if: "{instance =~ 'localhost.*'}"
action: replace
target_label: team
replacement: dev
static_configs:
- targets:
- localhost:8429
- vmgateway-single:8431
- vmgateway-cluster:8431
Relabeling rules will add the team
label to the scraped metrics in order to test multi-tenant access.
Metrics from localhost
will be labeled with team=dev
and metrics from vmgateway
will be labeled with team=admin
.
vmagent will write data into VictoriaMetrics single-node and cluster(with tenant 0:0
).
Grafana datasources configuration will be the following:
Let’s login as user with team=dev
labels limitation set via claims.
Using vmgateway-cluster
results into No data
response as proxied request will go to tenant 0:1
.
Since vmagent is only configured to write to 0:0
No data
is an expected response.
Switching to vmgateway-single
does have data. Note that it is limited to metrics with team=dev
label.
Now lets login as user with team=admin
.
Both cluster and single node datasources now return metrics for team=admin
.
Using oAuth for remote write with vmagent #
vmagent can be configured to use oAuth for remote write. This is in order to add authentication to the write requests.
In order to create a client for vmagent to use, follow the steps below:
Log in with admin credentials to your Keycloak instance
Go to
Clients
->Create
.
UseOpenID Connect
asClient Type
.
Specifyvmagent
asClient ID
.
ClickNext
.Enable
Client authentication
.
EnableAuthorization
.
ClickNext
.Leave URLs section empty as vmagent will not use any. Click
Save
.Go to
Clients
->vmagent
->Credentials
.
Copy the value ofClient secret
. It will be used later in vmagent configuration.Go to
Clients
->vmagent
->Client scopes
.
Click atvmagent-dedicated
->Add mapper
->By configuration
->User attribute
.
Configure the mapper as followsName
asvm_access
.Token Claim Name
asvm_access
.User Attribute
asvm_access
.Claim JSON Type
asJSON
. EnableAdd to ID token
andAdd to access token
.
Go to
Service account roles
-> click onservice-account-vmagent
.Go to
Attributes
tab and add an attribute. Specifyvm_access
asKey
.
Specify{"tenant_id" : {"account_id": 0, "project_id": 0 }}
as a value.
ClickSave
.
Once iDP configuration is done, vmagent configuration needs to be updated to use oAuth for remote write:
vmagent:
image: victoriametrics/vmagent:v1.106.0
volumes:
- ./scrape.yaml:/etc/vmagent/config.yaml
- ./vmagent-client-secret:/etc/vmagent/oauth2-client-secret
command:
- -promscrape.config=/etc/vmagent/config.yaml
- -remoteWrite.url=http://vmgateway-cluster:8431/api/v1/write
- -remoteWrite.url=http://vmgateway-single:8431/api/v1/write
- -remoteWrite.oauth2.clientID={CLIENT_ID}
- -remoteWrite.oauth2.clientSecretFile=/etc/vmagent/oauth2-client-secret
- -remoteWrite.oauth2.tokenUrl=http://keycloak:8080/realms/master/protocol/openid-connect/token
- -remoteWrite.oauth2.scopes=openid
It is required to replace {CLIENT_ID}
with the client ID and provide the client secret in vmagent-client-secret
file.
Note that vmagent will use the same token for both single-node and cluster vmgateway. vmgateway running in cluster mode
will use tenant information from the token to route the request to the correct tenant. vmgateway running in single-node mode
will just verify token validity.