At Build this year Microsoft announced the Custom VNET with Azure CNI integration for Azure Kubernetes Service.
-> https://azure.microsoft.com/en-us/blog/kubernetes-on-azure/
Even this was some month ago I would like to walk you through the necessary planning and deployment steps for the bring your own Virtual Network option here.
Before we start a deployment, we must ensure to meet the network prerequisites for the custom VNET scenario.
- The VNET for the AKS cluster must allow outbound internet connectivity.
- Do not create more than one AKS cluster in the same subnet.
- Advanced networking for AKS does not support VNETs that use Azure Private DNS Zones.
- AKS clusters may not use 169.254.0.0/16, 172.30.0.0/16, or 172.31.0.0/16 for the Kubernetes service address range.
- The service principal used for the AKS cluster must have Contributor permissions to the resource group containing the existing VNET.
The next step is to calculate the necessary IP address space that is needed for the AKS cluster deployment. This depends heavily on the number of nodes in the AKS cluster and the number of pods per node. An AKS cluster can have a maximum of 100 nodes and 110 pods maximum per each node. Per default the maximum pod limit in the advanced networking scenario is 30 per each node. You can adjust the limit during the initial deployment of the AKS cluster itself. For the calculation details we will have a look at the following part of the Azure AKS documentation.
-> https://docs.microsoft.com/en-us/azure/aks/networking-overview#plan-ip-addressing-for-your-cluster
I created a custom Virtual Network with a 10.0.0.0/8 address space including a subnet with a 10.240.0.0/16 address space for the AKS cluster deployment. The Kubernetes service CIDR address space is defined as 10.0.0.0/16 with the Kubernetes DNS service IP address 10.0.0.10. You must ensure in a bring you own Virtual Network deployment that the Kubernetes service CIDR is not used by any other network in Azure or on-premises the Virtual Network will gets connect to. Otherwise you may experience network / routing issues.
You have three options to deploy an AKS cluster into a custom VNET. The Azure portal, Azure CLI or an Azure Resource Manager template.
The Azure portal experience is very straight forward. In the Networking section select Advanced for the network configuration and select the VNET as also the corresponding subnet. Finally, the Kubernetes service address range, the Kubernetes DNS service IP address and the Docker Bridge address needs to be defined. That’s all, but it does not support to set the pod limit.
The Azure CLI is similar instead of the VNET and subnet selection. In the CLI command you reference to the Virtual Network subnet via the resource id and you can specify the pod limit.
az aks create --name aks-cluster --resource-group aks --network-plugin azure --max-pods 30 --service-cidr 10.0.0.0/16 --dns-service-ip 10.0.0.10 --docker-bridge-address 172.17.0.1/16 --vnet-subnet-id /subscriptions/{SUBSCRIPTION ID}/resourceGroups/{RESOURCE GROUP NAME}/providers/Microsoft.Network/virtualNetworks/{VIRTUAL NETWORK NAME}/subnets/{SUBNET NAME}
If you need an ARM template to get started, have a look at the following one.
-> https://github.com/neumanndaniel/armtemplates/blob/master/container/aks.json
An ARM template deployment can be kicked off through the Azure CLI or Azure PowerShell. Here is the Azure CLI example.
az group deployment create --resource-group aks --template-uri https://raw.githubusercontent.com/neumanndaniel/armtemplates/master/container/aks.json --parameters ./aks.parameters.json --verbose
Beside the ARM template you will need an ARM template parameter file. The parameter file includes all your parameters values like AKS cluster name and the VNET configuration.
After the deployment we will see that the AKS cluster was successfully deployed into the custom VNET.
Now, we can connect the AKS custom VNET through Virtual Network peering with the rest of our Azure and on-premises infrastructure.
-> https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-peering-overview