UPDATE: Checkout the new blog post covering AKS multiple node pools in Terraform Azure provider version 1.37.
The multiple node pool feature for AKS is currently in preview.
-> https://docs.microsoft.com/en-us/azure/aks/use-multiple-node-pools
Multiple node pools enable us to define a different amount of worker nodes, a different VM size or even that the nodes in the node pool should run across different AZs (Availability Zones) in an Azure region per node pool.
Today we walk through the steps to deploy multiple AKS node pools with Terraform.
Looking at the Terraform documentation for AKS, a node pool gets defined in the azurerm_kubernetes_cluster resource with an agent_pool_profile argument block.
...
agent_pool_profile {
name = "nodepool1"
count = 2
vm_size = "Standard_D2s_v3"
type = "VirtualMachineScaleSets"
availability_zones = ["1", "2"]
max_pods = 250
os_type = "Linux"
os_disk_size_gb = 128
vnet_subnet_id = var.vnet_subnet_id
}
...
The easiest, but not the most efficient way is to add a second agent_pool_profile argument block.
...
agent_pool_profile {
name = "nodepool1"
count = 2
vm_size = "Standard_D2s_v3"
type = "VirtualMachineScaleSets"
availability_zones = ["1", "2"]
max_pods = 250
os_type = "Linux"
os_disk_size_gb = 128
vnet_subnet_id = var.vnet_subnet_id
}
agent_pool_profile {
name = "nodepool2"
count = 3
vm_size = "Standard_D4s_v3"
type = "VirtualMachineScaleSets"
availability_zones = ["1", "2", "3"]
max_pods = 250
os_type = "Linux"
os_disk_size_gb = 128
vnet_subnet_id = var.vnet_subnet_id
}
...
As you do not want to repeat yourself, there is another better way to define multiple node pools in your AKS TF file.
Since Terraform 0.12 we got the dynamic block functionality that is the perfect solution to keep the agent_pool_profile argument block DRY.
-> https://www.terraform.io/docs/configuration/expressions.html#dynamic-blocks
Using the dynamic block construct changes our node pool definition in the AKS TF file slightly.
...
dynamic "agent_pool_profile" {
for_each = var.agent_pool_configuration
content {
name = format("nodepool%d", agent_pool_profile.key + 1)
count = agent_pool_profile.value["agent_count"]
vm_size = agent_pool_profile.value["vm_size"]
type = "VirtualMachineScaleSets"
availability_zones = agent_pool_profile.value["zones"]
max_pods = 250
os_type = "Linux"
os_disk_size_gb = 128
vnet_subnet_id = var.vnet_subnet_id
}
}
...
The variable for the node pool is defined by a list object.
...
variable "agent_pool_configuration" {
description = "The list object to configure one or several node pools with number of worker nodes, worker node VM size and Availability Zones."
type = list(object({
agent_count = number
vm_size = string
zones = list(string)
}))
}
...
In our case the list object contains options to define the amount of worker nodes, the VM size and the AZs per node pool.
module "aks" {
source = "../modules/aks"
...
agent_pool_configuration = [
{
agent_count = 2
vm_size = "Standard_D2_v3"
zones = null
},
{
agent_count = 2
vm_size = "Standard_D2s_v3"
zones = ["1", "2"]
}
]
}
The example above shows the definition of two node pools. One of the node pools does not use AZs to demonstrate the capabilities of the multiple node pool feature.
Finally, we can deploy the AKS cluster with two node pools in Azure.
terraform plan terraform apply -auto-approve
As seen in the screenshots we got two node pools with two nodes each. One does not leverage AZs, but the other one does.
> kubectl describe nodes | grep -e "Name:" -e "failure-domain.beta.kubernetes.io/zone"
Name: aks-nodepool1-33037761-vmss000000
failure-domain.beta.kubernetes.io/zone=0
Name: aks-nodepool1-33037761-vmss000001
failure-domain.beta.kubernetes.io/zone=1
Name: aks-nodepool2-33037761-vmss000000
failure-domain.beta.kubernetes.io/zone=northeurope-1
Name: aks-nodepool2-33037761-vmss000001
failure-domain.beta.kubernetes.io/zone=northeurope-2
You can find my AKS Terraform module beside other modules in my GitHub repository.
-> https://github.com/neumanndaniel/terraform/tree/master/modules