Cilium’s new Hubble flow policy log field

Cilium in version 1.18 introduced a new useful feature called “policy log field” for Hubble flows.

-> https://isovalent.com/blog/post/cilium-1-18/#hubble-flow-policy-log-field

This feature provides additional possibilities for further insights/checks on which network policy was applied to a network flow.

We have a look at how to configure the policy log field and what a Hubble flow looks like with and without it.

Hubble flow policy log field

Based on my previous blog post about how to restrict access to Azure’s IMDS endpoint, we now add the policy log field to the IMDS network policies.

-> https://www.danielstechblog.io/restrict-access-to-the-imds-endpoint-on-azure-kubernetes-service-with-cilium/

Let us first have a look at the Hubble flow logs for denied and allowed network traffic to the IMDS endpoint.

❯ hubble observe -P --to-ip 169.254.169.254 --not --from-identity host -f -o json

{
  "flow": {
    ...
    "event_type": {
      "type": 5
    },
    "traffic_direction": "EGRESS",
    "policy_match_type": 1,
    "drop_reason_desc": "POLICY_DENY",
    "Summary": "TCP Flags: SYN",
    "egress_denied_by": [
      {
        "name": "azure-imds-deny",
        "labels": [
          "k8s:io.cilium.k8s.policy.derived-from=CiliumClusterwideNetworkPolicy",
          "k8s:io.cilium.k8s.policy.name=azure-imds-deny",
          "k8s:io.cilium.k8s.policy.uid=244e124e-5fd8-4b3d-be6a-9371c750ccd0"
        ],
        "revision": "9",
        "kind": "CiliumClusterwideNetworkPolicy"
      }
    ],
    "policy_log": [
      ""
    ]
  },
  ...
}

{
  "flow": {
    ...
    "event_type": {
      "type": 5
    },
    "traffic_direction": "EGRESS",
    "policy_match_type": 2,
    "is_reply": false,
    "Summary": "TCP Flags: SYN",
    "egress_allowed_by": [
      {
        "name": "azure-imds-allow",
        "labels": [
          "k8s:io.cilium.k8s.policy.derived-from=CiliumClusterwideNetworkPolicy",
          "k8s:io.cilium.k8s.policy.name=azure-imds-allow",
          "k8s:io.cilium.k8s.policy.uid=cd4ff6bf-3daf-4dd0-8715-0b85905cf9fe"
        ],
        "revision": "9",
        "kind": "CiliumClusterwideNetworkPolicy"
      }
    ],
    "policy_log": [
      ""
    ]
  },
  ...
}

Now, let us add additional information via the policy log field that helps us to identify certain things in one view. In this example, we use the following annotation:

• Cilium policy type: cnp/ccnp
• Traffic direction: ingress/egress/both
• Verdict: allowed/denied
• Service: azure_imds

For the egress deny, we add ccnp_egress_denied_azure_imds to the policy log field, and for the egress allow, ccnp_egress_allowed_azure_imds.

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: azure-imds-deny
  annotations:
    description: "Deny traffic to Azure IMDS"
  labels:
    app.kubernetes.io/part-of: cilium
    area: network-security
spec:
  endpointSelector:
    matchExpressions:
      - key: k8s:io.kubernetes.pod.namespace
        operator: NotIn
        values:
          - kube-system
          - logging
          - grafana-alloy
  enableDefaultDeny:
    egress: false
    ingress: false
  egressDeny:
    - toCIDRSet:
        - cidrGroupRef: azure-imds
  log:
    value: "ccnp_egress_denied_azure_imds"
---
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: azure-imds-allow
  annotations:
    description: "Allow traffic to Azure IMDS"
  labels:
    app.kubernetes.io/part-of: cilium
    area: network-security
spec:
  endpointSelector:
    matchExpressions:
      - key: k8s:io.kubernetes.pod.namespace
        operator: In
        values:
          - kube-system
          - logging
          - grafana-alloy
  enableDefaultDeny:
    egress: false
    ingress: false
  egress:
    - toCIDRSet:
        - cidrGroupRef: azure-imds
      toPorts:
        - ports:
            - port: "80"
              protocol: TCP
          rules:
            http:
              - method: "GET"
                path: "/metadata"
  log:
    value: "ccnp_egress_allowed_azure_imds"

A well-defined pattern or annotation design helps you in your log queries to easily search for the Hubble flow logs you are interested in.

❯ hubble observe -P --to-ip 169.254.169.254 --not --from-identity host -f -o json

{
  "flow": {
    ...
    "event_type": {
      "type": 5
    },
    "traffic_direction": "EGRESS",
    "policy_match_type": 1,
    "drop_reason_desc": "POLICY_DENY",
    "Summary": "TCP Flags: SYN",
    "egress_denied_by": [
      {
        "name": "azure-imds-deny",
        "labels": [
          "k8s:io.cilium.k8s.policy.derived-from=CiliumClusterwideNetworkPolicy",
          "k8s:io.cilium.k8s.policy.name=azure-imds-deny",
          "k8s:io.cilium.k8s.policy.uid=244e124e-5fd8-4b3d-be6a-9371c750ccd0"
        ],
        "revision": "9",
        "kind": "CiliumClusterwideNetworkPolicy"
      }
    ],
    "policy_log": [
      "ccnp_egress_denied_azure_imds"
    ]
  },
  ...
}

{
  "flow": {
    ...
    "event_type": {
      "type": 5
    },
    "traffic_direction": "EGRESS",
    "policy_match_type": 2,
    "is_reply": false,
    "Summary": "TCP Flags: SYN",
    "egress_allowed_by": [
      {
        "name": "azure-imds-allow",
        "labels": [
          "k8s:io.cilium.k8s.policy.derived-from=CiliumClusterwideNetworkPolicy",
          "k8s:io.cilium.k8s.policy.name=azure-imds-allow",
          "k8s:io.cilium.k8s.policy.uid=cd4ff6bf-3daf-4dd0-8715-0b85905cf9fe"
        ],
        "revision": "9",
        "kind": "CiliumClusterwideNetworkPolicy"
      }
    ],
    "policy_log": [
      "ccnp_egress_allowed_azure_imds"
    ]
  },
  ...
}

Summary

The Hubble flow policy log field feature is a useful addition to the Hubble flows that can provide additional information.

You can find the example configurations on my GitHub repository.

-> https://github.com/neumanndaniel/kubernetes/tree/master/cilium/azure-imds