Istio has a feature that allows you to configure it to recognize your gateway network topology correctly.
-> https://istio.io/latest/docs/ops/configuration/traffic-management/network-topologies/
The reason you want to do this is that, when using the Istio ingress gateway as the entry point to your services, you want to determine the trusted client address from the x-forwarded-for HTTP header.
Envoy populates the trusted client address into the x-envoy-external-address HTTP header.
Running Istio in its default configuration, the x-envoy-external-address header is filled with the address of the immediate downstream node, and the x-forwarded-for header values are ignored.
Furthermore, Istio sets Envoy’s use_remote_address to true, and this determines how Envoy calculates, together with the xff_num_trusted_hops setting the trusted client address.
$ istioctl proxy-config listener istio-ingressgateway-5cf59789b9-r4pdj -o json | grep -E 'useRemoteAddress|xffNumTrustedHops'
"useRemoteAddress": true,
"useRemoteAddress": true,
| use_remote_address | xff_num_trusted_hops > 0 | Calculation | Example Result |
|---|---|---|---|
| false | 1 | N+1 | 2 |
| true | 1 | N | 1 |
The trusted client configuration is always determined from the right end of the x-forwarded-for header and is, depending on the settings, either the (N+1)th address or Nth address. When the number of addresses in the x-forwarded-for header is lower than the (N+1)th or Nth, Envoy falls back using the address of the immediate downstream node.
-> https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#x-forwarded-for
Example scenarios for Azure Kubernetes Service behind an Azure Load Balancer
Let us dive into two example scenarios for Istio running on Azure Kubernetes Service.
The first scenario is that the Istio ingress gateway is directly exposed via an Azure Load Balancer to serve traffic. An Azure Load Balancer operates on layer 4 and not on layer 7. Hence, we have zero trusted hops in front of our Istio ingress gateway, and we can specify the following gateway network topology configuration or leave Istio at its default configuration to determine the trusted client address.
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
name: istiocontrolplane
spec:
meshConfig:
gatewayTopology:
numTrustedProxies: 0
A quick test against the endpoint, which I have, shows that the x-envoy-external-address is filled correctly.
$ curl -I https://aks.danielstechblog.de
# Istio ingress gateway logs
{
"authority": "aks.danielstechblog.de",
...
"downstream_direct_remote_address": "92.117.27.189:59975",
"downstream_local_address": "100.64.2.164:8443",
"downstream_remote_address": "92.117.27.189:59975",
...
"upstream_cluster": "outbound|80||go-webapp.go-webapp.svc.cluster.local;",
"upstream_host": "100.64.2.158:8080",
"upstream_local_address": "100.64.2.164:49526",
...
"user_agent": "curl/8.7.1",
"x_envoy_external_address": "92.117.27.189",
"x_forwarded_for": "92.117.27.189"
}
Now imagine the second scenario in which we have a reverse proxy sitting in front of the Istio ingress gateway, in addition to the Azure Load Balancer. The previous gateway network topology configuration would provide incorrect results. We simulate that by adding an x-forwarded-for header to the curl command earlier, stating that 1.1.1.1 is our client address.
$ curl -I https://aks.danielstechblog.de -H "X-Forwarded-For: 1.1.1.1"
# Istio ingress gateway logs
{
"authority": "aks.danielstechblog.de",
...
"downstream_direct_remote_address": "92.117.27.189:63673",
"downstream_local_address": "100.64.2.164:8443",
"downstream_remote_address": "92.117.27.189:63673",
...
"upstream_cluster": "outbound|80||go-webapp.go-webapp.svc.cluster.local;",
"upstream_host": "100.64.2.158:8080",
"upstream_local_address": "100.64.2.164:49526",
...
"user_agent": "curl/8.7.1",
"x_envoy_external_address": "92.117.27.189",
"x_forwarded_for": "1.1.1.1,92.117.27.189"
}
Instead of setting 1.1.1.1 as the trusted client address, it is the address of the reverse proxy. So, for this scenario, we need a different gateway network topology configuration.
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
name: istiocontrolplane
spec:
meshConfig:
gatewayTopology:
numTrustedProxies: 1
With this config, 1.1.1.1 is correctly determined as the trusted client address.
$ curl -I https://aks.danielstechblog.de -H "X-Forwarded-For: 1.1.1.1"
# Istio ingress gateway logs
{
"authority": "aks.danielstechblog.de",
...
"downstream_direct_remote_address": "92.117.27.189:62248",
"downstream_local_address": "100.64.2.157:8443",
"downstream_remote_address": "1.1.1.1:0",
...
"upstream_cluster": "outbound|80||go-webapp.go-webapp.svc.cluster.local;",
"upstream_host": "100.64.2.89:8080",
"upstream_local_address": "100.64.2.157:55724",
...
"user_agent": "curl/8.7.1",
"x_envoy_external_address": "1.1.1.1",
"x_forwarded_for": "1.1.1.1,92.117.27.189"
}
Pitfalls in determining the correct number of trusted hops
When determining the correct number of trusted hops, you cannot trust the Istio ingress gateway’s x-forwarded-for header logs, because they show the header after Istio has processed it.
Knowing that you can use the rule of thumb: the number of entries in the x-forwarded-for header minus one to determine the number of trusted hops.
An alternative is to set the HTTP logging of the Istio ingress gateway to debug mode. This logs the x-forwarded-for header as it arrives at the Istio ingress gateway before the header is processed.
$ POD="istio-ingressgateway-54c56d567c-v44kp" $ kubectl -n istio-system exec "$POD" -c istio-proxy -- pilot-agent request POST 'logging?http=debug' $ curl -I https://aks.danielstechblog.de $ kubectl -n istio-system logs -f "$POD" -c istio-proxy | grep -iE "x-forwarded-for" empty results $ curl -I https://aks.danielstechblog.de -H "X-Forwarded-For: 1.1.1.1" $ kubectl -n istio-system logs -f "$POD" -c istio-proxy | grep -iE "x-forwarded-for" 'x-forwarded-for', '1.1.1.1' $ kubectl -n istio-system exec "$POD" -c istio-proxy -- pilot-agent request POST 'logging?http=info'
Summary
Once understood, it is straightforward to configure the gateway network topology for Istio. Depending on your environment the number of the numTrustedProxies might be different. However, it is worth configuring this correctly to determine the correct trusted client address.