High available control plane with Istio 1.5 on Azure Kubernetes Service

Back in January I have written a blog post about installing the Istio control plane on AKS in HA mode.

-> https://www.danielstechblog.io/install-a-high-available-istio-control-plane-on-azure-kubernetes-service/

Since the release of Istio 1.5 this month the overall architecture of the Istio control plane has changed.

-> https://istio.io/news/releases/1.5.x/announcing-1.5/#introducing-istiod

With the introduction of Istiod the number of deployed pods get dramatically reduced, when running a high available Istio control plane to at least four pods. Those four pods are two Istiod pods, representing the core of the control plane, and two Istio ingress gateway pods.

Beside that you might have some more pods representing add-on components like Grafana, Prometheus and Kiali.

Instead of using istioctl with --set option, I transferred those settings into a YAML template utilizing the IstioOperator API more easily.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  addonComponents:
    grafana:
      enabled: true
    tracing:
      enabled: true
    kiali:
      enabled: true
    prometheus:
      enabled: true
  components:
    ingressGateways:
    - enabled: true
      k8s:
        hpaSpec:
          minReplicas: 2
        overlays:
        - apiVersion: v1
          kind: Service
          name: istio-ingressgateway
          patches:
          - path: spec.ports
            value:
            - name: status-port
              port: 15020
              targetPort: 15020
            - name: http2
              port: 80
              targetPort: 80
            - name: https
              port: 443
              targetPort: 443
    pilot:
      enabled: true
      k8s:
        hpaSpec:
          minReplicas: 2
        overlays:
        - apiVersion: policy/v1beta1
          kind: PodDisruptionBudget
          name: istiod
          patches:
          - path: spec.selector.matchLabels
            value:
              app: istiod
              istio: pilot
  values:
    global:
      mtls:
        enabled: true
      controlPlaneSecurityEnabled: true
      proxy:
        accessLogFile: "/dev/stdout"
        accessLogEncoding: "JSON"
    gateways:
      istio-ingressgateway:
        sds:
          enabled: true
    sidecarInjectorWebhook:
      rewriteAppHTTPProbe: true

As you see in the template, I am setting the HPA minReplicas to 2 and thus ensure having a minimal HA setup for the Istio control plane.

Furthermore, I am using the overlays option to overwrite some specific settings.

Istio ingress gateway – Reduce number of exposed ports

My initial intention here came from a cost perspective regarding the costs of Standard Load Balancer rules in Azure. But it is also a good approach from the security perspective to only expose necessary ports.

In my case port 80, 443 and 15020 for the Envoy proxy health check endpoint.

...
        overlays:
        - apiVersion: v1
          kind: Service
          name: istio-ingressgateway
          patches:
          - path: spec.ports
            value:
            - name: status-port
              port: 15020
              targetPort: 15020
            - name: http2
              port: 80
              targetPort: 80
            - name: https
              port: 443
              targetPort: 443
...

Istiod PodDisruptionBudget issue

Same issue as described in my previous blog post, but now just for another component. The PDB for Istiod has three labels for the selector defined.

❯ kubectl describe poddisruptionbudgets.policy istiod
Name:           istiod
Namespace:      istio-system
Min available:  1
Selector:       app=istiod,istio=pilot,release=istio
Status:
    Allowed disruptions:  0
    Current:              0
    Desired:              1
    Total:                0
Events:
  Type    Reason  Age                From               Message
  ----    ------  ----               ----               -------
  Normal  NoPods  27s (x4 over 58s)  controllermanager  No matching pods found

Unfortunately, the Istiod Kubernetes Deployment template has only two of them defined.

When using selector labels, you need a match of all of them not only a partial match. Therefore, I am overwriting the PDB selector labels to only include the ones specified in the Kubernetes Deployment template.

...
        overlays:
        - apiVersion: policy/v1beta1
          kind: PodDisruptionBudget
          name: istiod
          patches:
          - path: spec.selector.matchLabels
            value:
              app: istiod
              istio: pilot
...

This ensures a working PDB for Istiod.

❯ kubectl describe poddisruptionbudgets.policy istiod
Name:           istiod
Namespace:      istio-system
Min available:  1
Selector:       app=istiod,istio=pilot
Status:
    Allowed disruptions:  1
    Current:              2
    Desired:              1
    Total:                2
Events:
  Type    Reason  Age                  From               Message
  ----    ------  ----                 ----               -------
  Normal  NoPods  70s (x6 over 2m41s)  controllermanager  No matching pods found

Appendix A – IstioOperator API templates

You can find the IstioOperator API template for Istio 1.5 under the following link.

-> https://github.com/neumanndaniel/kubernetes/blob/master/istio/istio-1.5.yaml

I also transferred the istioctl command from my previous blog post targeting Istio 1.4 to an IstioOperator API template.

-> https://github.com/neumanndaniel/kubernetes/blob/master/istio/istio-1.4.yaml