Azure Policy for Kubernetes – custom policies on Azure Arc enabled Kubernetes

On September 1st Microsoft announced the public preview of the custom policy support for Azure Policy for AKS.

-> https://azure.microsoft.com/en-us/updates/custom-aks-policy-support-now-public-preview/

I am already using the public preview on my AKS cluster and was curious about if this would work as well with Azure Arc enabled Kubernetes.

The short answer is yes, but with some minor adjustments.

Configuration

First step is the deployment of the Azure Policy for Kubernetes on the Azure Arc enabled Kubernetes cluster. The default values of the Helm chart install Gatekeeper and the Azure Policy components with outdated versions. Therefore, I checked my AKS cluster with the enabled Azure Policy add-on for the recent version.

Below is the command for the Helm chart installing the recent version of the Azure Policy add-on.

> helm upgrade azure-policy-addon azure-policy/azure-policy-addon-arc-clusters --install --namespace kube-system \
    --set azurepolicy.env.resourceid=/subscriptions/{REDACTED}/resourceGroups/KinD/providers/Microsoft.Kubernetes/connectedClusters/KinD \
    --set azurepolicy.env.clientid={REDACTED} \
    --set azurepolicy.env.clientsecret='{REDACTED}' \
    --set azurepolicy.env.tenantid={REDACTED} \
    --set azurepolicy.image.tag="prod_20210719.1" \
    --set azurepolicywebhook.image.tag="prod_20210713.1" \
    --set gatekeeper.image.tag="v3.4.1"

I am using a single node KinD cluster on my MacBook for Azure Arc enabled Kubernetes.

Azure portal Azure Arc Kubernetes

> kubectl get nodes
NAME                 STATUS   ROLES                  AGE     VERSION
kind-control-plane   Ready    control-plane,master   5d23h   v1.21.1
> kubectl get pods -n azure-arc
NAME                                         READY   STATUS    RESTARTS   AGE
cluster-metadata-operator-77d878d65c-5rxvz   2/2     Running   4          5d23h
clusterconnect-agent-6d894d44b-tp6gz         3/3     Running   6          5d23h
clusteridentityoperator-578c88fb78-wsxsq     2/2     Running   4          5d23h
config-agent-78974cb85b-9wp27                2/2     Running   4          5d23h
controller-manager-5b99f7b9df-4l5fs          2/2     Running   4          5d23h
extension-manager-58fd78456c-pltm5           2/2     Running   4          5d23h
flux-logs-agent-bd5659f94-xmhxs              1/1     Running   2          5d23h
kube-aad-proxy-78954db59b-9lh6q              2/2     Running   4          5d23h
metrics-agent-675566f58f-h652h               2/2     Running   4          5d23h
resource-sync-agent-5c547cd6-dvckp           2/2     Running   4          5d23h
> kubectl get pods -n gatekeeper-system
NAME                                             READY   STATUS    RESTARTS   AGE
gatekeeper-audit-56f4664568-9hjhj                1/1     Running   1          2d11h
gatekeeper-controller-manager-6597bd98fc-6fcw2   1/1     Running   1          2d11h
gatekeeper-controller-manager-6597bd98fc-hv2sm   1/1     Running   1          2d11h
> kubectl get pods -n kube-system | grep azure-policy
azure-policy-7cdb6d4d9b-88b4x                1/1     Running   1          2d11h
azure-policy-webhook-fc78c646d-w9dt4         1/1     Running   1          2d11h

When you follow the instructions for the custom policy setup take a look into the generated JSON document.

-> https://techcommunity.microsoft.com/t5/azure-governance-and-management/azure-policy-for-kubernetes-releases-support-for-custom-policy/ba-p/2699466

Especially at the field policyRule. The type of resource the custom policy applies to only includes Azure Kubernetes Service cluster.

{
  "properties": {
    "policyType": "Custom",
    "mode": "Microsoft.Kubernetes.Data",
    ...
    "policyRule": {
      "if": {
        "field": "type",
        "in": [
          "Microsoft.ContainerService/managedClusters"
        ]
      },
      "then": {
        "effect": "[parameters('effect')]",
...

So, how we get the custom policy on an Azure Arc enabled Kubernetes cluster? We add the resource type.

{
  "properties": {
    "policyType": "Custom",
    "mode": "Microsoft.Kubernetes.Data",
    ...
    "policyRule": {
      "if": {
        "field": "type",
        "in": [
          "Microsoft.Kubernetes/connectedClusters",
          "Microsoft.ContainerService/managedClusters"
        ]
      },
      "then": {
        "effect": "[parameters('effect')]",
...

After that we continue with the setup steps in the documentation.

The custom policy shows up in the Azure portal.

Azure portal custom policy Azure portal custom policy

Azure Policy for Kubernetes synchronizes every 15 minutes with Azure for new policies, submitting audit results, or configuration changes.

Checking the Kubernetes log of the Azure Policy pod it shows the successful download and creation of the constraint template from the Azure Storage account.

> kubectl logs azure-policy-7cdb6d4d9b-88b4x azure-policy | grep azurekubernetes
{"level":"info","ts":"2021-09-18T19:38:49.354457715Z","msg":"Retrieving file from url","log-id":"c99951d9-d-72","method":"github.com/Azure/azure-policy-kubernetes/pkg/formatter.(*PolicyFormatter).GetFileFromURL","url":"https://azurekubernetespolicy.blob.core.windows.net/constraint-templates/disable-automount-default-service-account-token.yaml"}
{"level":"info","ts":"2021-09-18T19:38:49.763245049Z","msg":"Constraint template created from URL","log-id":"c99951d9-d-73","method":"github.com/Azure/azure-policy-kubernetes/pkg/resourceutils/templateutils.(*Utils).CreateTemplateFromPolicy","Source":"https://azurekubernetespolicy.blob.core.windows.net/constraint-templates/disable-automount-default-service-account-token.yaml"}

Azure Storage account custom policy

Receiving the first audit results of the new policy takes between 15-30 minutes. The screenshots below showing a non-compliant state for the Azure Arc enabled Kubernetes cluster.

Azure portal custom policy audit results Azure portal custom policy audit results Azure portal custom policy audit results

Summary

Even Microsoft currently says that the public preview is only for Azure Policy for AKS it also works for Azure Arc enabled Kubernetes.

Only two adjustments are needed to get the custom policy working on Azure Arc enabled Kubernetes. First, using the same version for the Azure Policy add-on as AKS. Second, adding the resource type to the policyRule field in the generated JSON document.

Facebooktwitterlinkedinmail