In today’s blog post, I walk you through the setup on how to send Cilium metrics to Azure Managed Prometheus.
Our setup covers two scenarios. The first one is an Azure Kubernetes Service cluster using Cilium via the BYOCNI (Bring Your Own CNI) option, and the second one is a K3s single node cluster running on a Raspberry PI 5 using Cilium.
In both scenarios, we utilize Grafana Alloy to scrape and ingest the Prometheus metrics as we do not want to use Azure’s integrated toolchain.
Prerequisites
As prerequisites, we need, in general, an Azure Managed Prometheus and an Azure Managed Grafana workspace.
-> https://learn.microsoft.com/en-us/azure/azure-monitor/metrics/azure-monitor-workspace-manage?tabs=azure-portal&WT.mc_id=AZ-MVP-5000119#create-an-azure-monitor-workspace
-> https://learn.microsoft.com/en-us/azure/managed-grafana/quickstart-managed-grafana-portal?WT.mc_id=AZ-MVP-5000119
Additionally, for the Kubernetes cluster setup, an Azure Service Principal is required.
For the Azure Kubernetes Service cluster setup, we use the Azure Managed Identity of the Azure Kubernetes Service cluster node pool.
We assign both identities the Monitoring Metrics Publisher role on the data collection rule of our Azure Managed Prometheus workspace. This enables the metrics ingestion for both identities.
-> https://learn.microsoft.com/en-us/azure/azure-monitor/metrics/prometheus-remote-write-virtual-machines?WT.mc_id=AZ-MVP-5000119&tabs=managed-identity%2Cprom-vm#assign-the-monitoring-metrics-publisher-role-to-the-application
-> https://learn.microsoft.com/en-us/azure/azure-monitor/metrics/prometheus-remote-write-virtual-machines?WT.mc_id=AZ-MVP-5000119&tabs=entra-application%2Cprom-vm#assign-the-monitoring-metrics-publisher-role-to-the-application-1
The last prerequisite is to ensure that the Cilium metrics for the Cilium agent and operator are enabled.
-> https://docs.cilium.io/en/stable/observability/metrics/#installation
Grafana Alloy Configuration
The Grafana Alloy deployment is done via the Helm Chart provided by Grafana. Hence, we use the following values in a YAML file to configure the Alloy installation.
alloy:
configMap:
create: false
name: alloy-config
key: config.alloy
envFrom:
- secretRef:
name: alloy-azure-config
extraEnv:
- name: CLUSTER
value: k8s
- name: REGEX_METRICS_FILTER
value: ".+"
The YAML file contains three configuration parameters: configMap, envFrom, and extraEnv.
In the configMap parameter, we instruct the Helm chart not to create the config map in the Kubernetes namespace. Within the envFrom parameter, we reference the Kubernetes secret that contains either the Azure Service Principal credentials or the Azure Managed Identity information, depending on our setup. The last parameter provides two additional environment variables for our Alloy setup. First, the environment variable CLUSTER adds the Kubernetes cluster name as an additional Prometheus label to each Prometheus metric. The second one, REGEX_METRICS_FILTER, provides the ability to only ingest Prometheus metrics to Azure Managed Prometheus that match the defined regular expression.
Before we dive into the config map let us have a look into the Kubernetes secret. For the Azure Kubernetes Service cluster setup, the secret contains the client ID of the managed identity, and the endpoint URL of the Azure Managed Prometheus workspace.
apiVersion: v1
kind: Secret
metadata:
name: alloy-azure-config
namespace: grafana-alloy
labels:
app: grafana-alloy
data:
CLIENT_ID: <BASE64_VALUE>
ENDPOINT_URL: <BASE64_VALUE>
The Kubernetes cluster setup with the Azure Service Principal requires different values. Besides the endpoint URL, it is the client ID and client secret of the Azure Service Principal, as well as the Entra tenant ID.
apiVersion: v1
kind: Secret
metadata:
name: alloy-azure-config
namespace: grafana-alloy
labels:
app: grafana-alloy
data:
CLIENT_ID: <BASE64_VALUE>
CLIENT_SECRET: <BASE64_VALUE>
ENDPOINT_URL: <BASE64_VALUE>
TENANT_ID: <BASE64_VALUE>
Now, we can dive into the config map for our Grafana Alloy installation, which consists of six parts: logging, discovery.kubernetes, discovery.relabel, prometheus.scrape, prometheus.relabel, and prometheus.remote_write.
logging {
level = "info"
format = "json"
}
The log level can be set later to warn or error, but to the beginning, info is helpful for troubleshooting purposes. Instead of using the default logfmt format, we switch to json as a structured log format.
discovery.kubernetes "pods" {
role = "pod"
namespaces {
own_namespace = false
names = ["kube-system"]
}
selectors {
role = "pod"
field = "spec.nodeName=" + coalesce(sys.env("HOSTNAME"), constants.hostname)
}
}
We configure our Grafana Alloy installation to discover only pods for metrics scraping in the kube-system namespace. Additionally, we instruct Grafana Alloy to restrict the pod discovery to the same Kubernetes node as we run Grafana Alloy in the default daemon set setup.
For the discovery.relabel part we will focus only on the additional rules that have been added to the default scraping rules for pods mentioned in the Cilium example configuration.
...
rule {
source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_part_of"]
action = "keep"
regex = `cilium`
}
...
Ensuring that we only keep Cilium agent and operator pods in the scraping target list, we check if the Kubernetes label app.kubernetes.io/part-of contains cilium as a value. These scraping targets are kept in the list, and everything else is removed. When we run “kubectl get pods -n kube-system -l ‘app.kubernetes.io/part-of=cilium’” upfront, we get the potential targets for a quick cross-check already.
❯ kubectl get pods -n kube-system -l 'app.kubernetes.io/part-of=cilium' NAME READY STATUS RESTARTS AGE cilium-envoy-5qfs7 1/1 Running 8 (3d10h ago) 30d cilium-node-init-8kdjx 1/1 Running 17 (3d10h ago) 96d cilium-operator-659bffc68c-vcnhl 1/1 Running 9 (3d10h ago) 30d cilium-q4rkw 1/1 Running 8 (3d10h ago) 30d hubble-relay-5d5965474b-hxbxj 1/1 Running 8 (3d10h ago) 30d hubble-ui-7fd6bc845b-l6hz9 2/2 Running 16 (3d10h ago) 30d
Besides this rule, we add a rule that adds the cluster name as a label to every scraping target.
...
rule {
replacement = sys.env("CLUSTER")
target_label = "cluster"
}
...
In the Prometheus scraping part, we set three specific configurations.
prometheus.scrape "pods" {
job_name = "kubernetes-pods"
honor_labels = true
targets = discovery.relabel.pods.output
forward_to = [prometheus.relabel.pods.receiver]
scrape_interval = "30s"
scheme = "http"
}
First, we configure Grafana Alloy to honor labels and then set the scraping scheme to http. Furthermore, we increase the scraping interval from 60 to 30 seconds.
After the scraping part comes the prometheus.relabel part for filtering which Prometheus metrics we want to ingest into the Azure Managed Prometheus workspace.
prometheus.relabel "pods" {
forward_to = [prometheus.remote_write.azure_managed_prometheus.receiver]
rule {
source_labels = ["__name__"]
action = "keep"
regex = sys.env("REGEX_METRICS_FILTER")
}
}
Eventually, we reached the final part prometheus.remote_write for the metrics ingestion. Depending on whether we run Grafana Alloy on Azure Kubernetes Service or on another Kubernetes distribution, the configuration differs.
The Azure Kubernetes Service configuration uses the azuread block for the managed identity authentication.
prometheus.remote_write "azure_managed_prometheus" {
endpoint {
url = sys.env("ENDPOINT_URL")
azuread {
cloud = "AzurePublic"
managed_identity {
client_id = sys.env("CLIENT_ID")
}
}
}
}
Whereas the Kubernetes configuration uses the oauth2 block for the Azure Service Principal authentication.
prometheus.remote_write "azure_managed_prometheus" {
endpoint {
url = sys.env("ENDPOINT_URL")
oauth2 {
client_id = sys.env("CLIENT_ID")
client_secret = sys.env("CLIENT_SECRET")
token_url = "https://login.microsoftonline.com/" + sys.env("TENANT_ID") + "/oauth2/v2.0/token"
scopes = ["https://monitor.azure.com/.default"]
}
}
}
After walking you through the Grafana Alloy configuration, it is now time to deploy Grafana Alloy to our two Kubernetes clusters.
Rollout and Metrics Ingestion
The aforementioned configuration files can be found on my GitHub repository.
-> https://github.com/neumanndaniel/kubernetes/tree/master/cilium/prometheus-metrics
Before we install Grafana Alloy via its Helm Chart, we create a namespace, called grafana-alloy, the config map, and the secret containing either the managed identity or service principal information.
For the Azure Kubernetes Service cluster, we run the following commands
kubectl create namespace grafana-alloy kubectl apply -f "./aks/config-map.yaml" kubectl create secret generic alloy-azure-config -n grafana-alloy --from-literal=ENDPOINT_URL="<ENDPOINT_URL>" --from-literal=CLIENT_ID="<MANAGED_IDENTITY_CLIENT_ID>"
For the K3s cluster, we run a slightly different set of commands as we cannot use an Azure Managed Identity.
kubectl create namespace grafana-alloy kubectl apply -f "./k8s/config-map.yaml" kubectl create secret generic alloy-azure-config -n grafana-alloy --from-literal=ENDPOINT_URL="<ENDPOINT_URL>" --from-literal=CLIENT_ID="<SPN_CLIENT_ID>" --from-literal=CLIENT_SECRET="<SPN_CLIENT_SECRET>" --from-literal=TENANT_ID="<ENTRA_TENANT_ID>"
Now, we are ready to deploy Grafana Alloy to the Kubernetes clusters.
AKS:
helm repo add grafana https://grafana.github.io/helm-charts helm repo update helm upgrade --install grafana-alloy grafana/alloy --version "1.0.3" \ --wait \ --namespace "grafana-alloy" \ -f "./aks/grafana-alloy.yaml"
K3s:
helm repo add grafana https://grafana.github.io/helm-charts helm repo update helm upgrade --install grafana-alloy grafana/alloy --version "1.0.3" \ --wait \ --namespace "grafana-alloy" \ -f "./k8s/grafana-alloy.yaml"
Afterward, we check if the Grafana Alloy pods are up and running as seen in the screenshots below.
The next step is the verification of a working metrics ingestion within the Azure Managed Prometheus workspace. This can either happen via the Prometheus explorer or the Metrics tab.
Within the Prometheus explorer’s grid view, we see that the custom cluster label is added correctly to every scraping target.
In the case that no metrics arrive in the Azure Managed Prometheus workspace, Grafana Alloy provides a debug UI to examine the ingestion pipeline. All you need is a port forward to one of the Grafana Alloy pods.
❯ kubectl port-forward grafana-alloy-fr6ms --address localhost 12345:12345 ❯ open http://localhost:12345/graph
The last step is logging in to the Azure Managed Grafana workspace to import the Cilium dashboard for presenting the various Cilium metrics.
-> https://github.com/cilium/cilium/tree/v1.17.4/install/kubernetes/cilium/files/cilium-agent/dashboards
-> https://github.com/cilium/cilium/tree/v1.17.4/install/kubernetes/cilium/files/cilium-operator/dashboards
Once the Cilium dashboards have been imported successfully, we look at the metrics of our Cilium installations.
Summary
Sending Cilium metrics to an Azure Managed Prometheus workspace is a bit more work than using the Azure Monitor integration for it. However, it provides two tremendous advantages from my point of view, using Grafana Alloy, as in this example, or the Prometheus Operator. First, we have full control over the entire configuration to customize it to our needs. Second, we can easily exchange the Prometheus backend.
Do not get me wrong, the entire integrated Azure toolchain for monitoring an Azure Kubernetes Service or an Azure Arc-enabled Kubernetes cluster is great and might fit your needs. However, when you want to customize or fine-tune the out of box solutions, it gets complicated, and you might be better served using self-managed solutions to accomplish your goals.
The example configurations can be found on my GitHub repository.
-> https://github.com/neumanndaniel/kubernetes/tree/master/cilium/prometheus-metrics