Running Kubernetes at a cloud provider especially managed Kubernetes like AKS or GKE provides you with a solid foundation and applied security best practices for the managed master control plane.
But there is one downside, the public accessible API endpoint to control the Kubernetes cluster. Even the API endpoint is only exposed via HTTPS and access is secured via the Azure Active Directory integration and certificates, you may want a bit more.
Currently, you can achieve this with explicit IP whitelisting for accessing the API endpoint or using the newly announced public preview of having a private AKS cluster. The cluster is then only accessible via the Virtual Network without having a public API endpoint.
Today we talk about the explicit IP whitelisting feature for the API endpoint that is already GA.
The API endpoint whitelisting feature requires the Standard Load Balancer to be used with AKS. Except if you are matching a special condition.
-> https://docs.microsoft.com/en-us/azure/aks/api-server-authorized-ip-ranges
So, when you are using AKS with a Standard Load Balancer you do not need to redeploy the cluster to use the whitelisting feature. You can configure and use it via the Azure CLI, Azure Resource Manager templates or Terraform.
module "aks" {
source = "../modules/aks"
resource_group_name = "azst-aks1"
location = "North Europe"
...
name = "azst-aks1"
kubernetes_version = "1.14.8"
vnet_subnet_id = module.virtual_network.subnet_id
aad_group_name = "AKS-Admins"
api_auth_ips = [
x.x.x.x/32
]
...
}
In the above Terraform template I just whitelisted a single IP address. For instance, the IP address of my home office. There is no need to whitelist the outbound IP address of the AKS SLB to ensure communication between the kubelets and the API server. This is done automatically by Azure, when you enable the whitelisting.
But whitelisting the home office or office IP address is only one step. Do not forget your CI/CD system that you use for your deployments onto the AKS cluster. Especially when you are using hosted build agent that are not running in the AKS cluster itself.
Staying in the Azure universe we take a look into whitelisting Azure DevOps.
Depending in which Azure region / geography you created the Azure DevOps organization you whitelist a few or a couple of Azure regions with their respective IP address ranges.
Look into the organizations settings to identify the Azure region you are in.
As seen in the screenshot my Azure DevOps organization is in West Europe. Therefore, we need at least to whitelist all IP address ranges Azure uses for the West Europe region.
Microsoft publishes every week a new JSON file that includes the current IP address ranges for a specific Azure service or region.
-> https://www.microsoft.com/en-us/download/details.aspx?id=56519
Since the JSON file has thousand lines and more, we use the following shell command to get the ranges for West Europe.
jq '.values[] | select(.name | contains ("AzureCloud.westeurope")) | .properties.addressPrefixes' ServiceTags_Public_20191209.json
Whitelisting them enables us at least to successfully verify the Kubernetes service connection in Azure DevOps.
Without the whitelisting the verification fails as seen above.
But only whitelisting West Europe does not guarantee successful build or release pipelines that need to talk to the AKS API server.
For Azure DevOps we must whitelist also the North Europe region to cover the Azure geography Europe in our case.
Microsoft itself documented it, but not highlighting it directly.
Your hosted agents run in the same Azure geography as your organization. Each geography contains one or more regions, and while your agent may run in the same region as your organization, it is not guaranteed to do so. To obtain the complete list of possible IP ranges for your agent, you must use the IP ranges from all of the regions that are contained in your geography. For example, if your organization is located in the United States geography, you must use the IP ranges for all of the regions in that geography.
In the end Azure DevOps does not guarantee in which region in the specific Azure geography it is using the hosted build agents. There for all regions in an Azure geography must be whitelisted.
Let us now take a look at the complete Terraform template whitelisting West Europe and North Europe.
module "aks" {
source = "../modules/aks"
resource_group_name = "azst-aks1"
location = "North Europe"
...
name = "azst-aks1"
kubernetes_version = "1.14.8"
vnet_subnet_id = module.virtual_network.subnet_id
aad_group_name = "AKS-Admins"
api_auth_ips = concat(
local.azure_northeurope_ip_ranges,
local.azure_westeurope_ip_ranges
)
...
}
locals {
azure_westeurope_ip_ranges = [
"13.69.0.0/17",
"13.73.128.0/18",
...
]
azure_northeurope_ip_ranges = [
"13.69.128.0/17",
"13.70.192.0/18",
...
]
}
I am using local values to specify the IP address ranges for both regions at the end of my Terraform template to keep it clear. The variable api_auth_ips expects a list of strings as input. Therefore, we concatenate the two lists specified as local values.
After applying the new whitelist our build and release pipelines in Azure DevOps continuing to work.
A nice side-effect of whitelisting the whole regional IP address ranges is the whitelisting of Azure Cloud Shell as well.
If you want to disable the whitelisting on your AKS cluster, just provide an empty list in the Terraform template.
module "aks" {
source = "../modules/aks"
resource_group_name = "azst-aks1"
location = "North Europe"
...
name = "azst-aks1"
kubernetes_version = "1.14.8"
vnet_subnet_id = module.virtual_network.subnet_id
aad_group_name = "AKS-Admins"
api_auth_ips = []
...
}
Alternatively run the following Azure CLI command.
az aks update --resource-group azst-aks1 --name azst-aks1 --api-server-authorized-ip-ranges "" --verbose
The Terraform modules I used in the above code snippets can be found on GitHub.
-> https://github.com/neumanndaniel/terraform/tree/master/modules