Daniel's Tech Blog

Cloud Computing, Cloud Native & Kubernetes

Using ACS Engine to build private Kubernetes clusters with bring your own Virtual Network on Azure

Looking at Azure Container Service (AKS) – Managed Kubernetes you may have recognized that AKS currently does not support bring your own VNET and private Kubernetes masters. If you need both capabilities and one of them today, you must use ACS Engine to create the necessary Azure Resource Manager templates for the Kubernetes cluster deployment.

-> https://github.com/Azure/acs-engine

Beside that ACS Engine has advantages and disadvantages. Some of the ACS Engine advantages are RBAC, Managed Service Identity, private Kubernetes master, bring your own VNET and jump box deployment support. You even can choose between your favorite CNI plugin for network policies like the Azure CNI plugin, Calico or Cilium. If you want, you can specify none. But the default option is the Azure CNI plugin.

The main disadvantage of ACS Engine is that it creates non-managed Kubernetes cluster. You are responsible for nearly everything to keep the cluster operational.

So, you have a greater flexibility with ACS Engine, but you are responsible for more things compared to a managed solution.

After setting the context let us now start with the two scenarios I would like to talk about.

  1. Private Kubernetes cluster with bring your own VNET and jump box deployment
  2. Private Kubernetes cluster with bring your own VNET, custom Kubernetes service CIDR, custom Kubernetes DNS server IP address and jump box deployment

For each scenario, a VNET with address space 172.16.0.0/16 and one subnet with address space 172.16.0.0/20 will be the foundation to deploy the Kubernetes cluster in.

Starting with the first scenario and its config.json, have a look at the following lines.

{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorRelease": "1.10",
      "kubernetesConfig": {
        "useManagedIdentity": true,
        "networkPolicy": "azure",
        "containerRuntime": "docker",
        "enableRbac": true,
        "maxPods":30,
        "useInstanceMetadata": true,
        "addons": [
          {
            "name": "tiller",
            "enabled": true
          },
          {
            "name": "kubernetes-dashboard",
            "enabled": true
          }
        ],
        "privateCluster": {
          "enabled": true,
          "jumpboxProfile": {
            "name": "azst-acse1-jb",
            "vmSize": "Standard_A2_v2",
            "osDiskSizeGB": 32,
            "storageProfile": "ManagedDisks",
            "username": "azureuser",
            "publicKey": "REDACTED"
          }
        }
      }
    },
    "masterProfile": {
      "count": 1,
      "dnsPrefix": "azst-acse1",
      "vmSize": "Standard_A2_v2",
      "osDiskSizeGB": 32,
      "distro": "ubuntu",
      "vnetSubnetId": "/subscriptions/REDACTED/resourceGroups/acs-engine/providers/Microsoft.Network/virtualNetworks/acs-engine/subnets/k8s",
      "firstConsecutiveStaticIP": "172.16.15.239",
      "vnetCIDR": "172.16.0.0/16"
    },
    "agentPoolProfiles": [
      {
        "name": "agentpool",
        "count": 3,
        "vmSize": "Standard_A2_v2",
        "osDiskSizeGB": 32,
        "distro": "ubuntu",
        "storageProfile": "ManagedDisks",
        "diskSizesGB": [
          32
        ],
        "availabilityProfile": "AvailabilitySet",
        "vnetSubnetId": "/subscriptions/REDACTED/resourceGroups/acs-engine/providers/Microsoft.Network/virtualNetworks/acs-engine/subnets/k8s"
      }
    ],
    "linuxProfile": {
      "adminUsername": "azureuser",
      "ssh": {
        "publicKeys": [
          {
            "keyData": "REDACTED"
          }
        ]
      }
    }
  }
}

-> https://github.com/neumanndaniel/kubernetes/blob/master/acs-engine/kubernetes_custom_vnet_private_master.json

If your network does not overlap with the Kubernetes service CIDR, you only need to specify the VNET subnet id for the master and agent nodes.

{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
    ...
    },
    "masterProfile": {
      ...
      "vnetSubnetId": "/subscriptions/REDACTED/resourceGroups/acs-engine/providers/Microsoft.Network/virtualNetworks/acs-engine/subnets/k8s",
      "firstConsecutiveStaticIP": "172.16.15.239",
      "vnetCIDR": "172.16.0.0/16"
    },
    "agentPoolProfiles": [
      {
        ...
        "vnetSubnetId": "/subscriptions/REDACTED/resourceGroups/acs-engine/providers/Microsoft.Network/virtualNetworks/acs-engine/subnets/k8s"
      }
    ],
    "linuxProfile": {
    ...
    }
  }
}

Additionally, you should set the first consecutive IP and VNET CIDR in the master node section. With the first consecutive IP you set the IP of the first master node. For details check the ACS Engine documentation. The VNET CIDR should be set to prevent source address NAT’ing in the VNET.

-> https://github.com/Azure/acs-engine/blob/master/docs/kubernetes/features.md#feat-custom-vnet

Our second case covers the necessary configuration steps, if your network overlaps with the Kubernetes service CIDR. So, you would like to change the Kubernetes service CIDR, Kubernetes DNS server IP address and pod CIDR. Have a look at the following lines.

{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorRelease": "1.10",
      "kubernetesConfig": {
        "useManagedIdentity": true,
        "kubeletConfig": {
          "--non-masquerade-cidr": "172.16.0.0/20"
        },
        "clusterSubnet": "172.16.0.0/20",
        "dnsServiceIP": "172.16.16.10",
        "serviceCidr": "172.16.16.0/20",
        "networkPolicy": "azure",
        "containerRuntime": "docker",
        "enableRbac": true,
        "maxPods":30,
        "useInstanceMetadata": true,
        "addons": [
          {
            "name": "tiller",
            "enabled": true
          },
          {
            "name": "kubernetes-dashboard",
            "enabled": true
          }
        ],
        "privateCluster": {
          "enabled": true,
          "jumpboxProfile": {
            "name": "azst-acse1-jb",
            "vmSize": "Standard_A2_v2",
            "osDiskSizeGB": 32,
            "storageProfile": "ManagedDisks",
            "username": "azureuser",
            "publicKey": "REDACTED"
          }
        }
      }
    },
    "masterProfile": {
      "count": 1,
      "dnsPrefix": "azst-acse1",
      "vmSize": "Standard_A2_v2",
      "osDiskSizeGB": 32,
      "distro": "ubuntu",
      "vnetSubnetId": "/subscriptions/REDACTED/resourceGroups/acs-engine/providers/Microsoft.Network/virtualNetworks/acs-engine/subnets/k8s",
      "firstConsecutiveStaticIP": "172.16.15.239",
      "vnetCIDR": "172.16.0.0/16"
    },
    "agentPoolProfiles": [
      {
        "name": "agentpool",
        "count": 3,
        "vmSize": "Standard_A2_v2",
        "osDiskSizeGB": 32,
        "distro": "ubuntu",
        "storageProfile": "ManagedDisks",
        "availabilityProfile": "AvailabilitySet",
        "vnetSubnetId": "/subscriptions/REDACTED/resourceGroups/acs-engine/providers/Microsoft.Network/virtualNetworks/acs-engine/subnets/k8s"
      }
    ],
    "linuxProfile": {
      "adminUsername": "azureuser",
      "ssh": {
        "publicKeys": [
          {
            "keyData": "REDACTED"
          }
        ]
      }
    }
  }
}

-> https://github.com/neumanndaniel/kubernetes/blob/master/acs-engine/kubernetes_custom_network_config_private_master.json

In this case the service CIDR is part of the VNET CIDR to ensure no overlapping with existing networks. The DNS server IP address must be in the service CIDR space and the pod CIDR equals the VNET subnet address space, because we are using the Azure CNI plugin. So, the pods are receiving an IP address directly from the VNET subnet. If you are using another CNI plugin or none, then make sure to use an address space that is part of the VNET CIDR in this case. The final parameter is the – -non-masquerade-cidr that must be set to the VNET subnet CIDR. Have a brief overview of all necessary settings.

{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      ...
      "kubernetesConfig": {
        ...
        "kubeletConfig": {
          "--non-masquerade-cidr": "172.16.0.0/20" //VNET subnet CIDR address space
        },
        "clusterSubnet": "172.16.0.0/20", //VNET subnet CIDR address space
        "dnsServiceIP": "172.16.16.10", //IP address in serviceCidr address space
        "serviceCidr": "172.16.16.0/20", //CIDR address space in VNET CIDR - no overlapping in VNET
        ...
      }
    },
    "masterProfile": {
      ...
      "vnetSubnetId": "/subscriptions/REDACTED/resourceGroups/acs-engine/providers/Microsoft.Network/virtualNetworks/acs-engine/subnets/k8s",
      "firstConsecutiveStaticIP": "172.16.15.239",
      "vnetCIDR": "172.16.0.0/16" //VNET CIDR address space
    },
    "agentPoolProfiles": [
      {
        ...
        "vnetSubnetId": "/subscriptions/REDACTED/resourceGroups/acs-engine/providers/Microsoft.Network/virtualNetworks/acs-engine/subnets/k8s"
      }
    ],
    "linuxProfile": {
    ...
    }
  }
}

The next step on our way to the private Kubernetes cluster with bring your own VNET is the generation of the Azure Resource Manager templates by using the ACS Engine. Assuming you have downloaded the necessary ACS Engine bits and placed the kubernetes.json config into the same folder.

./acs-engine.exe generate ./kubernetes.json

The command generates the ARM template and places them in the folder _output per default.

Now, start the Azure Cloud Shell (https://shell.azure.com) and jump into the _output folder. Per drag and drop upload the azuredeploy.json and azuredeploy.parameters.json to the Cloud Shell. Afterwards kick off the deployment using the Azure CLI.

az group create --name acs-engine --location westeurope
az network vnet create --name acs-engine --resource-group acs-engine --address-prefixes 172.16.0.0/16 --subnet-name K8s --subnet-prefix 172.16.0.0/20

az group deployment create --resource-group acs-engine --template-file ./azuredeploy.json --parameters ./azuredeploy.parameters.json --verbose

After the successful deployment we can connect to the jump box and check, if all settings were taken effectively.

acsengineprivatek8s01acsengineprivatek8s02acsengineprivatek8s03

azcdmdn@azst-acse1-jb:~$ kubectl get services --all-namespaces -o wide
NAMESPACE     NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)         AGE       SELECTOR
default       azure-vote-back        ClusterIP      172.16.29.115   <none>          6379/TCP        13d       app=azure-vote-back
default       azure-vote-front       LoadBalancer   172.16.30.243   51.144.43.142   80:32627/TCP    13d       app=azure-vote-front
default       kubernetes             ClusterIP      172.16.16.1     <none>          443/TCP         13d       <none>
kube-system   heapster               ClusterIP      172.16.18.168   <none>          80/TCP          13d       k8s-app=heapster
kube-system   kube-dns               ClusterIP      172.16.16.10    <none>          53/UDP,53/TCP   13d       k8s-app=kube-dns
kube-system   kubernetes-dashboard   NodePort       172.16.28.177   <none>          80:32634/TCP    13d       k8s-app=kubernetes-dashboard
kube-system   metrics-server         ClusterIP      172.16.26.104   <none>          443/TCP         13d       k8s-app=metrics-server
kube-system   tiller-deploy          ClusterIP      172.16.29.150   <none>          44134/TCP       13d       app=helm,name=tiller
azcdmdn@azst-acse1-jb:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                            READY     STATUS    RESTARTS   AGE       IP              NODE
default       azure-vote-back-68d6c68dcc-xcfgk                1/1       Running   0          2h        172.16.0.46     k8s-agentpool-35404701-2
default       azure-vote-front-7976b7dcd9-mt2hh               1/1       Running   0          2h        172.16.0.51     k8s-agentpool-35404701-2
default       kured-2hrwh                                     1/1       Running   5          7d        172.16.0.22     k8s-agentpool-35404701-0
default       kured-5sgdn                                     1/1       Running   4          7d        172.16.0.45     k8s-agentpool-35404701-2
default       kured-qgbsx                                     1/1       Running   4          7d        172.16.0.99     k8s-agentpool-35404701-1
default       omsagent-2d2kf                                  1/1       Running   10         8d        172.16.0.41     k8s-agentpool-35404701-2
default       omsagent-ql9xv                                  1/1       Running   18         13d       172.16.0.84     k8s-master-35404701-0
default       omsagent-sc2xm                                  1/1       Running   10         8d        172.16.0.110    k8s-agentpool-35404701-1
default       omsagent-szj6j                                  1/1       Running   9          8d        172.16.0.16     k8s-agentpool-35404701-0
default       vsts-agent-qg5rz                                1/1       Running   1          2h        172.16.0.52     k8s-agentpool-35404701-2
kube-system   heapster-568476f785-c46r9                       2/2       Running   0          2h        172.16.0.39     k8s-agentpool-35404701-2
kube-system   kube-addon-manager-k8s-master-35404701-0        1/1       Running   6          13d       172.16.15.239   k8s-master-35404701-0
kube-system   kube-apiserver-k8s-master-35404701-0            1/1       Running   9          13d       172.16.15.239   k8s-master-35404701-0
kube-system   kube-controller-manager-k8s-master-35404701-0   1/1       Running   6          13d       172.16.15.239   k8s-master-35404701-0
kube-system   kube-dns-v20-59b4f7dc55-wtv6h                   3/3       Running   0          2h        172.16.0.44     k8s-agentpool-35404701-2
kube-system   kube-dns-v20-59b4f7dc55-xxgdd                   3/3       Running   0          2h        172.16.0.48     k8s-agentpool-35404701-2
kube-system   kube-proxy-hf467                                1/1       Running   6          13d       172.16.15.239   k8s-master-35404701-0
kube-system   kube-proxy-n8sj6                                1/1       Running   7          8d        172.16.0.5      k8s-agentpool-35404701-0
kube-system   kube-proxy-nb4gx                                1/1       Running   6          8d        172.16.0.36     k8s-agentpool-35404701-2
kube-system   kube-proxy-r8wdz                                1/1       Running   6          8d        172.16.0.97     k8s-agentpool-35404701-1
kube-system   kube-scheduler-k8s-master-35404701-0            1/1       Running   6          13d       172.16.15.239   k8s-master-35404701-0
kube-system   kubernetes-dashboard-64dcf5784f-gxtqv           1/1       Running   0          2h        172.16.0.109    k8s-agentpool-35404701-1
kube-system   metrics-server-7fcdc5dbb9-vs26l                 1/1       Running   0          2h        172.16.0.55     k8s-agentpool-35404701-2
kube-system   tiller-deploy-d85ccb55c-6nncg                   1/1       Running   0          2h        172.16.0.57     k8s-agentpool-35404701-2
azcdmdn@azst-acse1-jb:~$ kubectl get nodes --all-namespaces -o wide
NAME                       STATUS    ROLES     AGE       VERSION   EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION      CONTAINER-RUNTIME
k8s-agentpool-35404701-0   Ready     agent     13d       v1.10.0   <none>        Debian GNU/Linux 9 (stretch)   4.13.0-1014-azure   docker://1.13.1
k8s-agentpool-35404701-1   Ready     agent     13d       v1.10.0   <none>        Debian GNU/Linux 9 (stretch)   4.13.0-1014-azure   docker://1.13.1
k8s-agentpool-35404701-2   Ready     agent     13d       v1.10.0   <none>        Debian GNU/Linux 9 (stretch)   4.13.0-1014-azure   docker://1.13.1
k8s-master-35404701-0      Ready     master    13d       v1.10.0   <none>        Debian GNU/Linux 9 (stretch)   4.13.0-1014-azure   docker://1.13.1

The private Kubernetes cluster with bring your own VNET and custom network configuration is now fully operational and ready for some container deployments.

Last but not least I highly recommend that you secure your jump box with the Azure Security Center just in time capability.

-> https://docs.microsoft.com/en-us/azure/security-center/security-center-just-in-time

WordPress Cookie Notice by Real Cookie Banner