Daniel's Tech Blog

Cloud Computing, Cloud Native & Kubernetes

Terraform – Deploy an Azure Kubernetes Service cluster with multiple node pools

UPDATE: Checkout the new blog post covering AKS multiple node pools in Terraform Azure provider version 1.37.

-> https://www.danielstechblog.io/terraform-working-with-aks-multiple-node-pools-in-tf-azure-provider-version-1-37/


The multiple node pool feature for AKS is currently in preview.

-> https://docs.microsoft.com/en-us/azure/aks/use-multiple-node-pools

Multiple node pools enable us to define a different amount of worker nodes, a different VM size or even that the nodes in the node pool should run across different AZs (Availability Zones) in an Azure region per node pool.

Today we walk through the steps to deploy multiple AKS node pools with Terraform.

Looking at the Terraform documentation for AKS, a node pool gets defined in the azurerm_kubernetes_cluster resource with an agent_pool_profile argument block.

...
  agent_pool_profile {
    name               = "nodepool1"
    count              = 2
    vm_size            = "Standard_D2s_v3"
    type               = "VirtualMachineScaleSets"
    availability_zones = ["1", "2"]
    max_pods           = 250
    os_type            = "Linux"
    os_disk_size_gb    = 128
    vnet_subnet_id     = var.vnet_subnet_id
  }
...

The easiest, but not the most efficient way is to add a second agent_pool_profile argument block.

...
  agent_pool_profile {
    name               = "nodepool1"
    count              = 2
    vm_size            = "Standard_D2s_v3"
    type               = "VirtualMachineScaleSets"
    availability_zones = ["1", "2"]
    max_pods           = 250
    os_type            = "Linux"
    os_disk_size_gb    = 128
    vnet_subnet_id     = var.vnet_subnet_id
  }

  agent_pool_profile {
    name               = "nodepool2"
    count              = 3
    vm_size            = "Standard_D4s_v3"
    type               = "VirtualMachineScaleSets"
    availability_zones = ["1", "2", "3"]
    max_pods           = 250
    os_type            = "Linux"
    os_disk_size_gb    = 128
    vnet_subnet_id     = var.vnet_subnet_id
  }
...

As you do not want to repeat yourself, there is another better way to define multiple node pools in your AKS TF file.

Since Terraform 0.12 we got the dynamic block functionality that is the perfect solution to keep the agent_pool_profile argument block DRY.

-> https://www.terraform.io/docs/configuration/expressions.html#dynamic-blocks

Using the dynamic block construct changes our node pool definition in the AKS TF file slightly.

...
  dynamic "agent_pool_profile" {
    for_each = var.agent_pool_configuration
    content {
      name               = format("nodepool%d", agent_pool_profile.key + 1)
      count              = agent_pool_profile.value["agent_count"]
      vm_size            = agent_pool_profile.value["vm_size"]
      type               = "VirtualMachineScaleSets"
      availability_zones = agent_pool_profile.value["zones"]
      max_pods           = 250
      os_type            = "Linux"
      os_disk_size_gb    = 128
      vnet_subnet_id     = var.vnet_subnet_id
    }
  }
...

The variable for the node pool is defined by a list object.

...
variable "agent_pool_configuration" {
  description = "The list object to configure one or several node pools with number of worker nodes, worker node VM size and Availability Zones."
  type = list(object({
    agent_count = number
    vm_size     = string
    zones       = list(string)
  }))
}
...

In our case the list object contains options to define the amount of worker nodes, the VM size and the AZs per node pool.

module "aks" {
  source = "../modules/aks"

  ...

  agent_pool_configuration = [
    {
      agent_count = 2
      vm_size     = "Standard_D2_v3"
      zones       = null
    },
    {
      agent_count = 2
      vm_size     = "Standard_D2s_v3"
      zones       = ["1", "2"]
    }
  ]
}

The example above shows the definition of two node pools. One of the node pools does not use AZs to demonstrate the capabilities of the multiple node pool feature.

Finally, we can deploy the AKS cluster with two node pools in Azure.

terraform plan
terraform apply -auto-approve

As seen in the screenshots we got two node pools with two nodes each. One does not leverage AZs, but the other one does.

> kubectl describe nodes | grep -e "Name:" -e "failure-domain.beta.kubernetes.io/zone"
Name:               aks-nodepool1-33037761-vmss000000
                    failure-domain.beta.kubernetes.io/zone=0
Name:               aks-nodepool1-33037761-vmss000001
                    failure-domain.beta.kubernetes.io/zone=1
Name:               aks-nodepool2-33037761-vmss000000
                    failure-domain.beta.kubernetes.io/zone=northeurope-1
Name:               aks-nodepool2-33037761-vmss000001
                    failure-domain.beta.kubernetes.io/zone=northeurope-2

You can find my AKS Terraform module beside other modules in my GitHub repository.

-> https://github.com/neumanndaniel/terraform/tree/master/modules

WordPress Cookie Notice by Real Cookie Banner