In Cilium, IP addresses that do not belong to the Pod CIDR or Kubernetes Service CIDR range, and some special ranges like the Kubernetes API server, are recognized as the reserved:world identity. So, to say they do not belong to the Kubernetes cluster scope, known to Cilium itself.
-> https://docs.cilium.io/en/stable/gettingstarted/terminology/#special-identities
When you start using DNS-based Cilium network policies, you automatically add additional metadata information, identity labels, to the IP addresses that the FQDN resolves to.
-> https://docs.cilium.io/en/stable/security/dns/
However, you might want to add additional metadata information to IP addresses that are part of the reserved:world identity and not covered by a DNS-based Cilium network policy. The question is now, how do you do that?
Let us have a look into an Azure Kubernetes Service cluster, running Cilium in BYOCNI mode, and the two special IP addresses that Azure uses for the internal DNS service and the Instance Metadata Service short IMDS.
If you have not configured a custom DNS server for the Virtual Network that the Azure Kubernetes Service cluster uses, then CoreDNS and the Virtual Machine Scale Set instances are using Azure’s internal DNS service that operates under the 168.63.129.16 IP address. Azure’s IMDS service operates under the 169.254.169.254 IP address.
Looking into the network traffic of the kube-system namespace using Cilium’s Hubble UI, we see outbound traffic from the CoreDNS pods to the IP address 168.63.129.16 on port 53.
As not everyone is aware of those two Azure-specific IP addresses, we want to provide additional context to them.
In Cilium we can achieve this by using Cilium’s CIDR groups. The following two CIDR groups make Cilium aware of the two IP addresses.
apiVersion: cilium.io/v2alpha1
kind: CiliumCIDRGroup
metadata:
name: azure-imds
labels:
k8s-app: azure-imds
spec:
externalCIDRs:
- 169.254.169.254/32
---
apiVersion: cilium.io/v2alpha1
kind: CiliumCIDRGroup
metadata:
name: azure-internal-dns
labels:
k8s-app: azure-internal-dns
spec:
externalCIDRs:
- 168.63.129.16/32
Once applied to the Azure Kubernetes Service cluster, we see the additional metadata information for the IP 168.63.129.16.
According to the CIDR group definition, the destination labels cidrgroup:io.cilium.policy.cidrgroupname/azure-internal-dns and cidrgroup:k8s-app=azure-internal-dns are added. Also, the destination identity changes to the one for the CIDR group.
This allows us to provide additional context for IP addresses that reside outside of the Kubernetes cluster scope.
The example CIDR group definition can be found on my GitHub repository.