Today, we talk about how to block egress traffic with Cilium cluster-wide network policies on Azure Kubernetes Service. For this, we need an Azure Kubernetes Service cluster with Cilium installed via the bring-your-own CNI approach.
Azure CNI powered by Cilium unfortunately only partially supports Cilium network policies. However, Cilium cluster-wide network policies and Cilium CIDR groups are not officially supported. Even the required custom resource definitions are installed on the Azure Kubernetes Service cluster.
Can I use CiliumNetworkPolicy custom resources instead of Kubernetes NetworkPolicy resources?
CiliumNetworkPolicy custom resources are partially supported. Customers may use FQDN filtering as part of the Advanced Container Networking Services feature bundle.
The Cilium version I used is the RC.1 of the upcoming 1.17 release. Why not the latest stable version 1.16? I will outline this later in this blog post.
Egress traffic blocking is done to prevent network traffic to malicious network entities, to country-level network CIDR ranges, called geo-blocking, etc.
Using Cilium cluster-wide network policies with Cilium CIDR groups is an easy way to achieve this without setting up additional infrastructure or services. As security, including network security, is a multi-layer approach today’s presented solution is only one building block but a powerful one.
Configuration
The entire configuration in this example is kept simple and can become a bit more complex when it should be covering the geo-blocking approach. Geo-blocking requires constant updates to the Cilium CIDR group and includes the usage of third-party services like MaxMind.
In my example, I am using the three IP addresses of my blog and added a /32 at the end to define them in CIDR annotation.
The corresponding Cilium CIDR group template is shown below.
apiVersion: cilium.io/v2alpha1 kind: CiliumCIDRGroup metadata: name: egress-traffic-blocking labels: policy: egress-traffic-blocking spec: externalCIDRs: - 217.160.0.92/32 - 217.160.0.111/32 - 217.160.223.1/32
Without being referenced by a Cilium cluster-wide or Cilium network policy a Cilium CIDR group is just another resource in Kubernetes and does not affect egress traffic at all.
apiVersion: cilium.io/v2 kind: CiliumClusterwideNetworkPolicy metadata: name: egress-traffic-blocking labels: policy: egress-traffic-blocking spec: endpointSelector: {} enableDefaultDeny: egress: false egressDeny: - toCIDRSet: - cidrGroupRef: egress-traffic-blocking
As seen above in the Cilium cluster-wide network policy we select every possible endpoint in the Azure Kubernetes Service cluster this policy should be applied, to block egress traffic.
We set the enableDefaultDeny setting for egress to false as we want to allow every other egress traffic except for the ranges defined in the Cilium CIDR group.
Under egressDeny, we reference the Cilium CIDR group. If we want, we could add additional CIDR group references to the template.
After rolling out both templates to the Azure Kubernetes Service cluster, we test the egress traffic blocking in an application pod using the curl command and hubble observe to monitor if the egress traffic is blocked as intended.
root@bash:/# curl https://www.danielstechblog.de ------------------------------------------------------------------------------------------------------------------------------------------ ❯ hubble observe --drop-reason-desc POLICY_DENY -f Jan 15 07:19:58.346: default/bash:57152 (ID:21894) <> 217.160.0.92:443 (ID:16777218) policy-verdict:L3-Only EGRESS DENIED (TCP Flags: SYN) Jan 15 07:19:58.346: default/bash:57152 (ID:21894) <> 217.160.0.92:443 (ID:16777218) Policy denied by denylist DROPPED (TCP Flags: SYN)
As seen above the egress traffic is blocked successfully by the defined Cilium cluster-wide network policy.
Difference between Cilium 1.16 and 1.17
There have been some heavy investments into Cilium’s CIDR group capabilities for the upcoming 1.17 release.
First and foremost, how Cilium handles the identities for CIDR ranges. Before 1.17, Cilium created for every CIDR range its own identity. In my example, it would be three identities as seen below.
❯ kubectl exec -it cilium-6dddh -c cilium-agent -- cilium-dbg identity list ID LABELS 1 reserved:host reserved:kube-apiserver 2 reserved:world ... 16777217 cidr:217.160.0.111/32 reserved:world 16777218 cidr:217.160.223.1/32 reserved:world 16777219 cidr:217.160.0.92/32 reserved:world
With only a few hundred CIDR ranges in a CIDR group, this should not be a concern. However, in the scenario of geo-blocking, it can be tens of thousands of CIDR ranges which actually brings Cilium before 1.17 to its limits.
In Cilium 1.17 an identity is only created for the entire CIDR group and not for every CIDR range that is part of the CIDR group.
❯ kubectl exec -it cilium-vplkw -c cilium-agent -- cilium-dbg identity list ID LABELS 1 reserved:host reserved:kube-apiserver 2 reserved:world ... 16777218 cidrgroup:io.cilium.policy.cidrgroupname/egress-traffic-blocking reserved:world
Looking back to the geo-blocking scenario, that means we would only have one identity for all the tens of thousands of CIDR ranges which dramatically improves performance and scaling.
Summary
Doing egress traffic blocking on Azure Kubernetes Service using Cilium is straightforward with its CIDR groups and cluster-wide network policies. Especially, with the improvements in the upcoming 1.17 release.
The examples can be found on my GitHub repository.
-> https://github.com/neumanndaniel/kubernetes/tree/master/cilium/egress-traffic-blocking