Site icon Daniel's Tech Blog

Migrating an AKS BYOCNI cluster from Cilium’s cluster scope to multi-pool IPAM

danielstechblog

In today’s blog post, we have a look at a new feature in Cilium 1.20 that allows the migration from cluster scope to multi-pool IPAM.

The cluster scope IPAM is the default mode in Cilium when you install Cilium yourself. For cloud provider specific installations like on AWS, Azure, and GCP, you use the cloud provider optimized IPAM mode.

Before we dive into the new Cilium feature, we discuss why you should use the multi-pool IPAM mode instead of the default cluster scope one.

Cluster scope vs. multi-pool IPAM

Let us start with what both modes have in common. Regardless of whether you are using the cluster scope or multi-pool IPAM option, Cilium is responsible for the CIDR configuration. That provides you with flexible configuration options for your IPAM. Both options equally support tunnel, VXLAN/Geneve, and direct routing modes. Another commonality is support for multiple CIDR ranges per cluster (cluster level), and here it stops for the cluster scope IPAM.

The multi-pool IPAM supports two additional features/scenarios: multiple CIDR ranges per node and dynamic CIDR/IP allocation. Whereas the cluster scope only allows adding additional CIDR ranges when running out of IP addresses, the multi-pool option, thanks to its dynamic allocation feature, can also be used to exchange CIDR ranges as long as they do not overlap.

Besides that, with the multi-pool IPAM, we can assign CIDR ranges to selected Kubernetes nodes based on a label selector like the topology.kubernetes.io/zone label.

You can see that multi-pool IPAM is the most configurable and flexible option when it comes to running Cilium on a Kubernetes cluster that does not rely on external CIDR configuration.

-> https://docs.cilium.io/en/stable/network/concepts/ipam/
-> https://docs.cilium.io/en/stable/network/concepts/ipam/cluster-pool/
-> https://docs.cilium.io/en/stable/network/concepts/ipam/multi-pool/

When your Kubernetes cluster uses the cluster scope IPAM, and you want to use the multi-pool IPAM now, Cilium 1.20 has you covered. Since Cilium 1.20, a migration option from cluster scope IPAM to multi-pool IPAM is now included.

In the next section, we migrate an Azure Kubernetes Service BYOCNI cluster with Cilium to multi-pool IPAM.

Multi-pool IPAM migration

The Cilium project provides a migration guideline with all the required details.

-> https://docs.cilium.io/en/stable/network/concepts/ipam/cluster-pool-to-multi-pool/

In our case, when migrating an Azure Kubernetes Service BYOCNI cluster to multi-pool IPAM, we run some checks beforehand, as seen below.

$ kubectl get ciliumpodippools.cilium.io -A
No resources found

$ kubectl get ciliumnodes.cilium.io aks-default-13458874-vmss000058 -ojson | jq .spec.ipam
{
  "podCIDRs": [
    "100.64.10.0/24"
  ],
  "pools": {}
}

First, we check for already existing Cilium pod IP pools and second have a look at one of the node’s IPAM status. All looks good, and we can proceed with the migration.

...
ipam:
  operator:
    clusterPoolIPv4PodCIDRList:
      - 100.64.0.0/10
...

The previous cluster scope IPAM configuration will be replaced by the new multi-pool one.

...
ipam:
  mode: multi-pool
  operator:
    autoCreateCiliumPodIPPools:
      default:
        ipv4:
          cidrs:
            - 100.64.0.0/10
          maskSize: 24
extraConfig:
  enable-cluster-pool-to-multi-pool-migration: "true"
rollOutCiliumPods: false
...

I recommend letting the Cilium Operator create the new default Cilium pod IP pool; you can always edit it later. For instance, applying labels to it.

Besides that, the mode is specified as multi-pool, and we use the new migration feature in Cilium 1.20.

Furthermore, according to the migration guideline, we disable for the migration the automatic rollout of the Cilium pods to not interfere with the Cilium Operator’s work.

After applying the new Cilium configuration to our Azure Kubernetes Service cluster, we run the same checks again.

$ kubectl get ciliumpodippools.cilium.io default -o yaml
apiVersion: cilium.io/v2alpha1
kind: CiliumPodIPPool
metadata:
  creationTimestamp: "2026-11-05T08:17:40Z"
  generation: 1
  name: default
  resourceVersion: "3039518"
  uid: f915ca73-5cdb-4003-91fd-8b51d72b97c8
spec:
  allowFirstIP: false
  allowLastIP: false
  ipv4:
    cidrs:
    - 100.64.0.0/10
    maskSize: 24

$ kubectl get ciliumnodes.cilium.io aks-default-13458874-vmss000058 -ojson | jq .spec.ipam
{
  "pools": {
    "allocated": [
      {
        "cidrs": [
          "100.64.10.0/24"
        ],
        "pool": "default"
      }
    ],
    "requested": [
      {
        "needed": {
          "ipv4-addrs": 24
        },
        "pool": "default"
      }
    ]
  }
}

Now, we see that the Cilium Operator created the Cilium pod IP pool successfully. Also, the nodes received their new IPAM configuration, and the same assigned CIDR range as before is preserved.

Summary

Thanks to Cilium’s new migration feature in 1.20, there is nothing to speak against using multi-pool IPAM on your Kubernetes clusters. The migration effort is nearly zero and straightforward. So, a small change that provides flexibility to your Kubernetes clusters.

Exit mobile version