On September 1st Microsoft announced the public preview of the custom policy support for Azure Policy for AKS.
-> https://azure.microsoft.com/en-us/updates/custom-aks-policy-support-now-public-preview/
I am already using the public preview on my AKS cluster and was curious about if this would work as well with Azure Arc enabled Kubernetes.
The short answer is yes, but with some minor adjustments.
Configuration
First step is the deployment of the Azure Policy for Kubernetes on the Azure Arc enabled Kubernetes cluster. The default values of the Helm chart install Gatekeeper and the Azure Policy components with outdated versions. Therefore, I checked my AKS cluster with the enabled Azure Policy add-on for the recent version.
Below is the command for the Helm chart installing the recent version of the Azure Policy add-on.
> helm upgrade azure-policy-addon azure-policy/azure-policy-addon-arc-clusters --install --namespace kube-system \ --set azurepolicy.env.resourceid=/subscriptions/{REDACTED}/resourceGroups/KinD/providers/Microsoft.Kubernetes/connectedClusters/KinD \ --set azurepolicy.env.clientid={REDACTED} \ --set azurepolicy.env.clientsecret='{REDACTED}' \ --set azurepolicy.env.tenantid={REDACTED} \ --set azurepolicy.image.tag="prod_20210719.1" \ --set azurepolicywebhook.image.tag="prod_20210713.1" \ --set gatekeeper.image.tag="v3.4.1"
I am using a single node KinD cluster on my MacBook for Azure Arc enabled Kubernetes.
> kubectl get nodes NAME STATUS ROLES AGE VERSION kind-control-plane Ready control-plane,master 5d23h v1.21.1 > kubectl get pods -n azure-arc NAME READY STATUS RESTARTS AGE cluster-metadata-operator-77d878d65c-5rxvz 2/2 Running 4 5d23h clusterconnect-agent-6d894d44b-tp6gz 3/3 Running 6 5d23h clusteridentityoperator-578c88fb78-wsxsq 2/2 Running 4 5d23h config-agent-78974cb85b-9wp27 2/2 Running 4 5d23h controller-manager-5b99f7b9df-4l5fs 2/2 Running 4 5d23h extension-manager-58fd78456c-pltm5 2/2 Running 4 5d23h flux-logs-agent-bd5659f94-xmhxs 1/1 Running 2 5d23h kube-aad-proxy-78954db59b-9lh6q 2/2 Running 4 5d23h metrics-agent-675566f58f-h652h 2/2 Running 4 5d23h resource-sync-agent-5c547cd6-dvckp 2/2 Running 4 5d23h > kubectl get pods -n gatekeeper-system NAME READY STATUS RESTARTS AGE gatekeeper-audit-56f4664568-9hjhj 1/1 Running 1 2d11h gatekeeper-controller-manager-6597bd98fc-6fcw2 1/1 Running 1 2d11h gatekeeper-controller-manager-6597bd98fc-hv2sm 1/1 Running 1 2d11h > kubectl get pods -n kube-system | grep azure-policy azure-policy-7cdb6d4d9b-88b4x 1/1 Running 1 2d11h azure-policy-webhook-fc78c646d-w9dt4 1/1 Running 1 2d11h
When you follow the instructions for the custom policy setup take a look into the generated JSON document.
Especially at the field policyRule. The type of resource the custom policy applies to only includes Azure Kubernetes Service cluster.
{ "properties": { "policyType": "Custom", "mode": "Microsoft.Kubernetes.Data", ... "policyRule": { "if": { "field": "type", "in": [ "Microsoft.ContainerService/managedClusters" ] }, "then": { "effect": "[parameters('effect')]", ...
So, how we get the custom policy on an Azure Arc enabled Kubernetes cluster? We add the resource type.
{ "properties": { "policyType": "Custom", "mode": "Microsoft.Kubernetes.Data", ... "policyRule": { "if": { "field": "type", "in": [ "Microsoft.Kubernetes/connectedClusters", "Microsoft.ContainerService/managedClusters" ] }, "then": { "effect": "[parameters('effect')]", ...
After that we continue with the setup steps in the documentation.
The custom policy shows up in the Azure portal.
Azure Policy for Kubernetes synchronizes every 15 minutes with Azure for new policies, submitting audit results, or configuration changes.
Checking the Kubernetes log of the Azure Policy pod it shows the successful download and creation of the constraint template from the Azure Storage account.
> kubectl logs azure-policy-7cdb6d4d9b-88b4x azure-policy | grep azurekubernetes {"level":"info","ts":"2021-09-18T19:38:49.354457715Z","msg":"Retrieving file from url","log-id":"c99951d9-d-72","method":"github.com/Azure/azure-policy-kubernetes/pkg/formatter.(*PolicyFormatter).GetFileFromURL","url":"https://azurekubernetespolicy.blob.core.windows.net/constraint-templates/disable-automount-default-service-account-token.yaml"} {"level":"info","ts":"2021-09-18T19:38:49.763245049Z","msg":"Constraint template created from URL","log-id":"c99951d9-d-73","method":"github.com/Azure/azure-policy-kubernetes/pkg/resourceutils/templateutils.(*Utils).CreateTemplateFromPolicy","Source":"https://azurekubernetespolicy.blob.core.windows.net/constraint-templates/disable-automount-default-service-account-token.yaml"}
Receiving the first audit results of the new policy takes between 15-30 minutes. The screenshots below showing a non-compliant state for the Azure Arc enabled Kubernetes cluster.
Summary
Even Microsoft currently says that the public preview is only for Azure Policy for AKS it also works for Azure Arc enabled Kubernetes.
Only two adjustments are needed to get the custom policy working on Azure Arc enabled Kubernetes. First, using the same version for the Azure Policy add-on as AKS. Second, adding the resource type to the policyRule field in the generated JSON document.