This is the second part of a three-part series about “An experiment – Enable Cilium native routing on Azure Kubernetes Service BYOCNI”.
We will focus today on how to enable Cilium native routing with WireGuard Transparent Encryption on Azure Kubernetes Service BYOCNI.
Enable Cilium native routing with WireGuard Transparent Encryption
In this example, we use 100.64.0.0/10 as the pod CIDR range and have the following Helm chart configuration, condensed on the most important settings.
aksbyocni: enabled: true autoDirectNodeRoutes: false encryption: enabled: true type: wireguard nodeEncryption: true ipv4NativeRoutingCIDR: 100.64.0.0/10 routingMode: native
When we apply this configuration to our Cilium installation, we get an error message.
Error: UPGRADE FAILED: execution error at (cilium/templates/cilium-operator/deployment.yaml:49:48): RoutingMode must be set to tunnel when aksbyocni.enabled=true
The Helm chart does not allow native routing when using Azure Kubernetes Service BYOCNI. Looking into the Helm chart, it is the only reason for having “aksbyocni” set to “true”. No other Cilium settings are affected by that.
Let us try it again with a new configuration.
aksbyocni: enabled: false autoDirectNodeRoutes: false encryption: enabled: true type: wireguard nodeEncryption: true ipv4NativeRoutingCIDR: 100.64.0.0/10 routingMode: native
Now the deployment works, and we should have Cilium running with native routing on Azure Kubernetes Service BYOCNI.
Before I switched the Cilium installation from encapsulation to native routing, I gathered the configuration status via cilium-dbg.
❯ kubectl exec -it cilium-dlrh7 -- cilium-dbg status ... KubeProxyReplacement: True [eth0 10.10.0.12 fe80::20d:3aff:fedf:60c4 (Direct Routing)] ... CNI Chaining: none CNI Config file: successfully wrote CNI configuration file to /host/etc/cni/net.d/05-cilium.conflist Cilium: Ok 1.18.6 (v1.18.6-95896696) ... Cilium health daemon: Ok IPAM: IPv4: 34/254 allocated from 100.64.0.0/24, ... Routing: Network: Tunnel [vxlan] Host: BPF Attach Mode: TCX Device Mode: veth Masquerading: BPF [eth0] 100.64.0.0/24 [IPv4: Enabled, IPv6: Disabled] ... Encryption: Wireguard [NodeEncryption: Enabled, cilium_wg0 (Pubkey: OnJFcH+/tGSyXhhiMVJvpUpbQ5ctX7kqPmoNj/cTEGI=, Port: 51871, Peers: 3)] ...
In the output, we see under Routing that the Cilium installation uses VXLAN for encapsulation.
Additionally, I ran a quick network connection test between two pods on different Kubernetes nodes and observed the test with the Hubble CLI.
❯ hubble observe -P --protocol tcp --from-pod kube-system/ping -f Feb 13 21:20:47.557: kube-system/ping:45284 (ID:30081) -> go-webapp/go-webapp-55f76547b5-xkpm7:8080 (ID:14226) to-endpoint FORWARDED (TCP Flags: SYN) Feb 13 21:20:47.557: kube-system/ping:45284 (ID:30081) -> go-webapp/go-webapp-55f76547b5-xkpm7:8080 (ID:14226) policy-verdict:all EGRESS ALLOWED (TCP Flags: SYN) Feb 13 21:20:47.557: kube-system/ping:45284 (ID:30081) -> go-webapp/go-webapp-55f76547b5-xkpm7:8080 (ID:14226) to-overlay FORWARDED (TCP Flags: SYN) Feb 13 21:20:47.557: kube-system/ping:45284 (ID:30081) -> go-webapp/go-webapp-55f76547b5-xkpm7:8080 (ID:14226) to-overlay FORWARDED (TCP Flags: ACK) Feb 13 21:20:47.557: kube-system/ping:45284 (ID:30081) -> go-webapp/go-webapp-55f76547b5-xkpm7:8080 (ID:14226) to-overlay FORWARDED (TCP Flags: ACK, PSH) Feb 13 21:20:47.558: kube-system/ping:45284 (ID:30081) -> go-webapp/go-webapp-55f76547b5-xkpm7:8080 (ID:14226) to-endpoint FORWARDED (TCP Flags: ACK) Feb 13 21:20:47.558: kube-system/ping:45284 (ID:30081) -> go-webapp/go-webapp-55f76547b5-xkpm7:8080 (ID:14226) to-endpoint FORWARDED (TCP Flags: ACK, PSH) Feb 13 21:20:47.562: kube-system/ping:45284 (ID:30081) -> go-webapp/go-webapp-55f76547b5-xkpm7:8080 (ID:14226) to-overlay FORWARDED (TCP Flags: ACK, FIN) Feb 13 21:20:47.562: kube-system/ping:45284 (ID:30081) -> go-webapp/go-webapp-55f76547b5-xkpm7:8080 (ID:14226) to-endpoint FORWARDED (TCP Flags: ACK, FIN) Feb 13 21:20:47.562: kube-system/ping:45284 (ID:30081) -> go-webapp/go-webapp-55f76547b5-xkpm7:8080 (ID:14226) to-endpoint FORWARDED (TCP Flags: ACK)
You can see in the Hubble CLI output that the packets are sent to the overlay -> to-overlay.
After the switch to the native routing mode, I ran cilium-dbg and executed the test again.
❯ kubectl exec -it cilium-6sggq -- cilium-dbg status ... KubeProxyReplacement: True [eth0 10.10.0.12 fe80::20d:3aff:fedf:60c4 (Direct Routing)] ... CNI Chaining: none CNI Config file: successfully wrote CNI configuration file to /host/etc/cni/net.d/05-cilium.conflist Cilium: Ok 1.18.6 (v1.18.6-95896696) ... Cilium health daemon: Ok IPAM: IPv4: 34/254 allocated from 100.64.0.0/24, ... Routing: Network: Native Host: BPF Attach Mode: TCX Device Mode: veth Masquerading: BPF [eth0] 100.64.0.0/10 [IPv4: Enabled, IPv6: Disabled] ... Encryption: Wireguard [NodeEncryption: Enabled, cilium_wg0 (Pubkey: NCgjS0OgiVNOQZyPk+tNQ83zpH7m9kYK16AgkX3VSws=, Port: 51871, Peers: 3)] ...
Now, under Routing, we see that the Cilium installation uses the native routing mode.
❯ hubble observe -P --protocol tcp --from-pod kube-system/ping -f Feb 13 21:35:13.969: kube-system/ping:38502 (ID:30081) -> go-webapp/go-webapp-55f76547b5-fcpg2:8080 (ID:14226) policy-verdict:all EGRESS ALLOWED (TCP Flags: SYN) Feb 13 21:35:13.969: kube-system/ping:38502 (ID:30081) -> go-webapp/go-webapp-55f76547b5-fcpg2:8080 (ID:14226) to-network FORWARDED (TCP Flags: SYN) Feb 13 21:35:13.969: kube-system/ping:38502 (ID:30081) -> go-webapp/go-webapp-55f76547b5-fcpg2:8080 (ID:14226) to-endpoint FORWARDED (TCP Flags: SYN) Feb 13 21:35:13.970: kube-system/ping:38502 (ID:30081) -> go-webapp/go-webapp-55f76547b5-fcpg2:8080 (ID:14226) to-network FORWARDED (TCP Flags: ACK) Feb 13 21:35:13.970: kube-system/ping:38502 (ID:30081) -> go-webapp/go-webapp-55f76547b5-fcpg2:8080 (ID:14226) to-endpoint FORWARDED (TCP Flags: ACK) Feb 13 21:35:13.970: kube-system/ping:38502 (ID:30081) -> go-webapp/go-webapp-55f76547b5-fcpg2:8080 (ID:14226) to-network FORWARDED (TCP Flags: ACK, PSH) Feb 13 21:35:13.970: kube-system/ping:38502 (ID:30081) -> go-webapp/go-webapp-55f76547b5-fcpg2:8080 (ID:14226) to-endpoint FORWARDED (TCP Flags: ACK, PSH) Feb 13 21:35:13.974: kube-system/ping:38502 (ID:30081) -> go-webapp/go-webapp-55f76547b5-fcpg2:8080 (ID:14226) to-endpoint FORWARDED (TCP Flags: ACK, FIN) Feb 13 21:35:13.974: kube-system/ping:38502 (ID:30081) -> go-webapp/go-webapp-55f76547b5-fcpg2:8080 (ID:14226) to-network FORWARDED (TCP Flags: ACK, FIN) Feb 13 21:35:13.975: kube-system/ping:38502 (ID:30081) -> go-webapp/go-webapp-55f76547b5-fcpg2:8080 (ID:14226) to-endpoint FORWARDED (TCP Flags: ACK)
Also, the observed network traffic via Hubble CLI shows that the packets are sent to the network directly -> to-network.
Summary
Using Cilium native routing with WireGuard Transparent Encryption works on Azure Kubernetes Service BYOCNI when we set certain configuration options for native routing and explicitly set the “aksbyocni” to “false” to bypass the routing mode check in Cilium’s Helm chart.
This solution still uses encapsulation, WireGuard tunnel, but provides secure network communication by encrypting the network traffic.
Outlook
In the third and last part, we have look at Cilium native routing with Azure Route Server and BGP, and then sum up the topic of enabling Cilium native routing on Azure Kubernetes Service BYOCNI.