Terraform – Working with AKS multiple node pools in TF Azure provider version 1.37

On Tuesday this week the Terraform Azure provider version 1.37 was released and brings some changes regarding AKS multiple node pools support. Multiple node pools are now fully supported in Terraform with their own resource azurerm_kubernetes_cluster_node_pool.

First, let us talk about the changes to the resource azurerm_kubernetes_cluster before getting to the azurerm_kubernetes_cluster_node_pool resource. In the azurerm_kubernetes_cluster resource the argument block agent_pool_profile is replaced with the default_node_pool block. Since the first node pool or also called the default node pool in an AKS cluster can only be a Linux one, the argument os_type is removed too.

...
resource "azurerm_kubernetes_cluster" "aks" {
  ...
  default_node_pool {
    name                = substr(var.default_node_pool.name, 0, 12)
    node_count          = var.default_node_pool.node_count
    vm_size             = var.default_node_pool.vm_size
    type                = "VirtualMachineScaleSets"
    availability_zones  = var.default_node_pool.zones
    max_pods            = 250
    os_disk_size_gb     = 128
    vnet_subnet_id      = var.vnet_subnet_id
    node_taints         = var.default_node_pool.taints
    enable_auto_scaling = var.default_node_pool.cluster_auto_scaling
    min_count           = var.default_node_pool.cluster_auto_scaling_min_count
    max_count           = var.default_node_pool.cluster_auto_scaling_max_count
  }
  ...
}
...

The new azurerm_kubernetes_cluster_node_pool resource looks like the former agent_pool_profile block with one exception. You do not define the node pool type which can be from type AvailabilitySets or VirtualMachineScaleSets. The type for the whole AKS cluster is set within the default_node_pool block. Since multiple node pools are only supported, when the AKS cluster uses VMSS. So, this specific argument does not need to be defined in the azurerm_kubernetes_cluster_node_pool resource. The other difference to the newly introduced default_node_pool block is the argument os_type to specify whether the node pool is a Linux or Windows one.

...
resource "azurerm_kubernetes_cluster_node_pool" "aks" {
  lifecycle {
    ignore_changes = [
      node_count
    ]
  }

  for_each = var.additional_node_pools

  kubernetes_cluster_id = azurerm_kubernetes_cluster.aks.id
  name                  = each.value.node_os == "Windows" ? substr(each.key, 0, 6) : substr(each.key, 0, 12)
  node_count            = each.value.node_count
  vm_size               = each.value.vm_size
  availability_zones    = each.value.zones
  max_pods              = 250
  os_disk_size_gb       = 128
  os_type               = each.value.node_os
  vnet_subnet_id        = var.vnet_subnet_id
  node_taints           = each.value.taints
  enable_auto_scaling   = each.value.cluster_auto_scaling
  min_count             = each.value.cluster_auto_scaling_min_count
  max_count             = each.value.cluster_auto_scaling_max_count
}
...

For the deployment of multiple node pools with the new azurerm_kubernetes_cluster_node_pool resource we are using the for_each expression in combination with a map object.

...
variable "additional_node_pools" {
  type = map(object({
    node_count                     = number
    vm_size                        = string
    zones                          = list(string)
    node_os                        = string
    taints                         = list(string)
    cluster_auto_scaling           = bool
    cluster_auto_scaling_min_count = number
    cluster_auto_scaling_max_count = number
  }))
}
...

We could also use count and a list object, but with that the node pools get an index number assigned. Having an index number assigned blocks the removal of a node pool between two other node pools in the index list without destroying and recreating the ones following the one that should be removed.

Here is an example how to define the default node pool / first node pool and two additional node pools for an Azure Kubernetes Service cluster.

module "aks" {
  source = "../modules/aks-windows"
  ...
  default_node_pool = {
    name                           = "nodepool1"
    node_count                     = 2
    vm_size                        = "Standard_D2_v3"
    zones                          = ["1", "2"]
    taints                         = null
    cluster_auto_scaling           = false
    cluster_auto_scaling_min_count = null
    cluster_auto_scaling_max_count = null
  }

  additional_node_pools = {
    pool2 = {
      node_count = 1
      vm_size    = "Standard_D4_v3"
      zones      = ["1", "2"]
      node_os    = "Windows"
      taints = [
        "kubernetes.io/os=windows:NoSchedule"
      ]
      cluster_auto_scaling           = false
      cluster_auto_scaling_min_count = null
      cluster_auto_scaling_max_count = null
    }
    pool3 = {
      node_count                     = 4
      vm_size                        = "Standard_E4_v3"
      zones                          = ["1", "2", "3"]
      node_os                        = "Linux"
      taints                         = null
      cluster_auto_scaling           = true
      cluster_auto_scaling_min_count = 4
      cluster_auto_scaling_max_count = 12
    }
  }
}

The name you specify in the map object for each node pool is used as the node pool’s name. In the example above pool2 and pool3. The name is automatically adjusted with the substr function, if it contains more than 6 characters for Windows node pools or more than 12 characters for Linux node pools.

Beside that I also added support for the cluster autoscaler functionality. This includes a lifecycle block as well to ignore changes to the node_count argument that gets adjusted during scaling operations outside of Terraform.

You can find the modules in the following GitHub repository.

-> https://github.com/neumanndaniel/terraform/tree/master/modules