After looking at Cilium’s transparent encryption performance for WireGuard and ztunnel on Azure Kubernetes Service BYOCNI, we now look at IPsec.
-> https://www.danielstechblog.io/an-experiment-enable-cilium-native-routing-on-azure-kubernetes-service-byocni-part-2/
-> https://www.danielstechblog.io/cilium-ztunnel-native-mtls-for-cilium-on-azure-kubernetes-service-byocni/
As in the tests before, the MTU has been left untouched. In other words, jumbo frames have not been used.
Set up IPsec in Cilium
Similar to the ztunnel encryption option, IPsec must be configured before we can enable transparent IPsec encryption. Fortunately, Cilium provides out-of-the-box capabilities for it.
-> https://docs.cilium.io/en/stable/security/network/encryption-ipsec/#generate-import-the-psk
Once we have generated the required IPsec configuration, we roll out Cilium with IPsec encryption enabled.
... encryption: enabled: true type: ipsec ...
Keep in mind that the L7 proxy option requires the DNS proxy to operate in transparent mode when IPsec is used as an encryption option.
-> https://docs.cilium.io/en/stable/security/network/encryption-ipsec/#dependencies
Performance comparison
In the table below, you find the test summary, which is also available as a Markdown file.
All tests are available in detail as Markdown files, outlining the test setup configuration and which test commands have been executed.
| Test | Geneve | Geneve + IPsec | Native Routing + IPsec | Native Routing |
|---|---|---|---|---|
| TCP Throughput (Gbit/s) | 7.2 | 2.4 | 2.6 | 12.0 |
| UDP Throughput (Gbit/s) | 1.6 | 0.8 | 1.0 | 2.0 |
| UDP Throughput Lost Datagrams (%) | 3.1 | 0.4 | 1.4 | 1.2 |
| Bidirectional Throughput (TX-C/RX-C) (Gbit/s) | 4.0/2.4 | 0.5/2.3 | 2.0/0.5 | 12.0/11.0 |
| TCP Latency (Mean µs/Transaction Rate) | 164/6095 | 198/5054 | 183/5468 | 124/ 8072 |
| UDP Latency (Mean µs/Transaction Rate) | 161/6220 | 197/5067 | 180/5539 | 122/ 8158 |
| TCP Stream (Throughput 10^6bits/sec) | 6809 | 2368 | 2552 | 11927 |
Looking at the test results, IPsec provides the same performance as the WireGuard encryption option in Cilium. Cilium’s IPsec option operates in IPsec tunnel mode, which creates an overlay. Both IPsec and WireGuard are outperformed by the new ztunnel option.
Summary
Depending on your requirements, IPsec might be the only encryption option you can use in Cilium when, for instance, FIPS 140-2/-3 certification is required.
All the test results, configurations, and summaries are available on my GitHub repository.
-> https://github.com/neumanndaniel/kubernetes/blob/master/cilium/network-performance-testing/