Recently Microsoft introduced silently some configurations options for the Azure Monitor for containers solution.
As you might know the Azure Monitor for containers solution collects stdout, stderr and environment variables from AKS and AKS-engine clusters except from containers running in the kube-system namespace.
If you want to use the new feature the minimum agent version that is required is ciprod06142019 or later. Run the following command to check the agent version in your AKS cluster.
kubectl get pods -l component=oms-agent -o yaml | grep image:
When the output shows an older version, you can upgrade the agent manually following the guide on Azure docs.
We can now focus on the data collection configuration. I am providing two scenarios. First, the activation of the log collection for the kube-system namespace and second how to disable the environment variable collection on the whole AKS cluster or individual containers.
The log collection activation for the kube-system namespace is done via a ConfigMap named container-azm-ms-agentconfig and must be deployed to the kube-system namespace.
The following ConfigMap does the job.
kind: ConfigMap
apiVersion: v1
metadata:
name: container-azm-ms-agentconfig
namespace: kube-system
data:
schema-version: v1
config-version: 1.0.0
log-data-collection-settings: |-
[log_collection_settings]
[log_collection_settings.stdout]
enabled = true
exclude_namespaces = []
[log_collection_settings.stderr]
enabled = true
exclude_namespaces = []
[log_collection_settings.env_var]
enabled = true
Apply the ConfigMap with kubectl apply -f to your AKS cluster. Afterwards run kubectl logs for each omsagent pod to check the correct loading of the ConfigMap.
... AZMON_AGENT_CFG_SCHEMA_VERSION:v1 AZMON_AGENT_CFG_FILE_VERSION:1.0.0 ****************Start Config Processing******************** config::configmap container-azm-ms-agentconfig for settings mounted, parsing values config::Successfully parsed mounted config map config::Using config map setting for stdout log collection config::Using config map setting for stderr log collection config::Using config map setting for cluster level environment variable collection Both stdout & stderr log collection are turned off for namespaces: '*.csv2' ****************End Config Processing******************** ...
The container logs from the kube-system namespace are logged from now on to Azure Monitor for containers and stored in Azure Log Analytics.
Moving forward to the second scenario to disable the environment variable collection.
If you want to disable it overall in the AKS cluster, you just need to change the configuration parameter [log_collection_settings.env_var] in the ConfigMap from true to false.
...
[log_collection_settings.env_var]
enabled = false
Afterwards apply the ConfigMap to your cluster and you are done.
The configuration option for individual containers to deactivate or activate the environment variable collection is available since the agent version ciprod11292018.
Add AZMON_COLLECT_ENV to the env section in your templates and set the value to False. It works for new containers or updating existing ones.
...
spec:
containers:
- name: azure-vote-front
image: REDACTED.azurecr.io/azure-vote-front:latest
imagePullPolicy: Always
ports:
- containerPort: 80
resources:
requests:
cpu: 250m
limits:
cpu: 500m
env:
- name: REDIS
value: "azure-vote-back"
- name: AZMON_COLLECT_ENV
value: "False"
Check Azure Monitor for containers for the successful configuration. In my example I deployed an application with two pods. On one, the azure-vote-front, I have disabled the environment variable collection.
For further details and what settings are available to configure the data collection follow this section in Azure docs.