Earlier this year I had written a blog post about updating the VMSS base image of an aks-engine cluster.
-> https://www.danielstechblog.io/updating-the-base-image-of-an-vmss-aks-engine-cluster/
Today, we focus on Azure Kubernetes Service running with VMSS.
Microsoft releases updates regularly to the AKS service and refreshes also the base image for the AKS worker nodes.
-> https://github.com/azure/aks/releases
This week I deployed a new VMSS-based AKS cluster. The worker nodes using the base image microsoft-aks / aks / aks-ubuntu-1604-201908 / 2019.08.09. But as of writing this blog post the most recent base image is microsoft-aks / aks / aks-ubuntu-1604-201908 / 2019.08.21.
So, how do you get the latest base image configured for your AKS cluster?
The official and supported way is the AKS cluster upgrade to the latest offered Kubernetes version in Azure.
Another way is this shell script I have written.
-> https://github.com/neumanndaniel/kubernetes/blob/master/aks/aksBaseImageUpdate.sh
Frankly, I do not know if this is supported by the AKS engineering team. But that is the way how it is intended to work, when using Virtual Machine Scale Sets in Azure. From the VMSS perspective it is a supported way to update the VMSS configuration. Therefore, it works fine for VMSS-based AKS clusters and does not break anything.
Now back to the shell script.
#!/bin/bash
set -e
set -o pipefail
echo "[$(date +"%Y-%m-%d %H:%M:%S")] Gathering information about the Kubernetes cluster and the latest base image..."
VMSS=$(kubectl get nodes|grep vmss --max-count=1|cut -d ' ' -f1|rev|cut -c 7-|rev)
RESOURCE_GROUP=$(kubectl get node $(kubectl get nodes|grep vmss --max-count=1|cut -d ' ' -f 1) -o json|jq '.metadata.labels'|grep kubernetes.azure.com/cluster|cut -d '"' -f4)
VMSS_PROPERTIES=$(az vmss show --resource-group $RESOURCE_GROUP --name $VMSS)
OFFER=$(echo $VMSS_PROPERTIES|jq '.virtualMachineProfile.storageProfile.imageReference.offer'|cut -d '"' -f 2)
PUBLISHER=$(echo $VMSS_PROPERTIES|jq '.virtualMachineProfile.storageProfile.imageReference.publisher'|cut -d '"' -f 2)
SKU_TEMP=$(echo $VMSS_PROPERTIES|jq '.virtualMachineProfile.storageProfile.imageReference.sku'|cut -d '"' -f 2|rev|cut -c 7-|rev)
SKU=$SKU_TEMP$(date +"%Y%m")
BASE_IMAGES=$(az vm image list --offer $OFFER --publisher $PUBLISHER --sku $SKU --all)
if [[ $(echo $BASE_IMAGES|jq length) -eq 0 ]]; then
SKU=$SKU_TEMP$(date +"%Y%m" --date="last month")
BASE_IMAGES=$(az vm image list --offer $OFFER --publisher $PUBLISHER --sku $SKU --all)
fi
BASE_IMAGE_COUNT=$(echo $BASE_IMAGES|jq length)
LATEST_BASE_IMAGE=$(echo $BASE_IMAGES|jq ".[$BASE_IMAGE_COUNT-1]")
echo "[$(date +"%Y-%m-%d %H:%M:%S")] Updating base image..."
az vmss update --resource-group $RESOURCE_GROUP --name $VMSS \
--set virtualMachineProfile.storageProfile.imageReference.sku=$(echo $LATEST_BASE_IMAGE|jq '.sku'|cut -d '"' -f 2) \
virtualMachineProfile.storageProfile.imageReference.version=$(echo $LATEST_BASE_IMAGE|jq '.version'|cut -d '"' -f 2)|jq '.virtualMachineProfile.storageProfile.imageReference'
echo "[$(date +"%Y-%m-%d %H:%M:%S")] Updating VMSS instances..."
VMSS_INSTANCES=$(kubectl get nodes|grep vmss|cut -d ' ' -f1)
for ITEM in $VMSS_INSTANCES; do
TEMP_INSTANCE_ID=$(kubectl get nodes $ITEM -o yaml|grep providerID)
INSTANCE_ID=$(echo $TEMP_INSTANCE_ID|cut -d '/' -f13)
echo "[$(date +"%Y-%m-%d %H:%M:%S")] Draining node $ITEM..."
kubectl drain $ITEM --ignore-daemonsets --delete-local-data --force
echo "[$(date +"%Y-%m-%d %H:%M:%S")] Updating VMSS instance $ITEM..."
az vmss update-instances --instance-ids $INSTANCE_ID --name $VMSS --resource-group $RESOURCE_GROUP
echo "[$(date +"%Y-%m-%d %H:%M:%S")] Uncordon node $ITEM..."
kubectl uncordon $ITEM
done
echo "[$(date +"%Y-%m-%d %H:%M:%S")] Base image update finished..."
Start an Azure Cloud Shell session and ensure that the current kubeconfig context is the AKS cluster you want to run the script against.
The script does the following steps:
- Gather AKS cluster and base image information
- Set new base image for VMSS
- For each VMSS instance
- Run kubectl drain
- Update VMSS instance to latest version
- Run kubectl uncordon
./aksBaseImageUpdate.sh
[2019-09-06 20:15:42] Gathering information about the Kubernetes cluster and the latest base image...
[2019-09-06 20:15:53] Updating base image...
{
"id": null,
"offer": "aks",
"publisher": "microsoft-aks",
"sku": "aks-ubuntu-1604-201908",
"version": "2019.08.21"
}
[2019-09-06 20:16:06] Updating VMSS instances...
[2019-09-06 20:16:06] Draining node aks-agentpool1-14987876-vmss000000...
[2019-09-06 20:16:52] Updating VMSS instance aks-agentpool1-14987876-vmss000000...
[2019-09-06 20:19:24] Uncordon node aks-agentpool1-14987876-vmss000000...
[2019-09-06 20:19:24] Draining node aks-agentpool1-14987876-vmss000001...
[2019-09-06 20:20:12] Updating VMSS instance aks-agentpool1-14987876-vmss000001...
[2019-09-06 20:22:44] Uncordon node aks-agentpool1-14987876-vmss000001...
[2019-09-06 20:22:45] Draining node aks-agentpool1-14987876-vmss000002...
[2019-09-06 20:23:26] Updating VMSS instance aks-agentpool1-14987876-vmss000002...
[2019-09-06 20:25:29] Uncordon node aks-agentpool1-14987876-vmss000002...
[2019-09-06 20:25:29] Base image update finished...
After the script finished your AKS cluster runs on the latest base image.
New AKS worker nodes will also use the new base image.