Tweaking data collection for Azure Monitor for containers

Recently Microsoft introduced silently some configurations options for the Azure Monitor for containers solution.

As you might know the Azure Monitor for containers solution collects stdout, stderr and environment variables from AKS and AKS-engine clusters except from containers running in the kube-system namespace.

If you want to use the new feature the minimum agent version that is required is ciprod06142019 or later. Run the following command to check the agent version in your AKS cluster.

kubectl get pods -l component=oms-agent -o yaml | grep image:

When the output shows an older version, you can upgrade the agent manually following the guide on Azure docs.

-> https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-manage-agent#how-to-upgrade-the-azure-monitor-for-containers-agent

We can now focus on the data collection configuration. I am providing two scenarios. First, the activation of the log collection for the kube-system namespace and second how to disable the environment variable collection on the whole AKS cluster or individual containers.

The log collection activation for the kube-system namespace is done via a ConfigMap named container-azm-ms-agentconfig and must be deployed to the kube-system namespace.

The following ConfigMap does the job.

kind: ConfigMap
apiVersion: v1
metadata:
  name: container-azm-ms-agentconfig
  namespace: kube-system
data:
  schema-version: v1
  config-version: 1.0.0
  log-data-collection-settings: |-
    [log_collection_settings]
      [log_collection_settings.stdout]
        enabled = true
        exclude_namespaces = []
      [log_collection_settings.stderr]
        enabled = true
        exclude_namespaces = []
      [log_collection_settings.env_var]
        enabled = true

-> https://github.com/neumanndaniel/kubernetes/blob/master/azure-monitor/container-azm-ms-agentconfig.yaml

Apply the ConfigMap with kubectl apply -f to your AKS cluster. Afterwards run kubectl logs for each omsagent pod to check the correct loading of the ConfigMap.

...
AZMON_AGENT_CFG_SCHEMA_VERSION:v1
AZMON_AGENT_CFG_FILE_VERSION:1.0.0
****************Start Config Processing********************
config::configmap container-azm-ms-agentconfig for settings mounted, parsing values
config::Successfully parsed mounted config map
config::Using config map setting for stdout log collection
config::Using config map setting for stderr log collection
config::Using config map setting for cluster level environment variable collection
Both stdout & stderr log collection are turned off for namespaces: '*.csv2'
****************End Config Processing********************
...

The container logs from the kube-system namespace are logged from now on to Azure Monitor for containers and stored in Azure Log Analytics.

azuremonitor01azuremonitor02

Moving forward to the second scenario to disable the environment variable collection.

If you want to disable it overall in the AKS cluster, you just need to change the configuration parameter [log_collection_settings.env_var] in the ConfigMap from true to false.

...
      [log_collection_settings.env_var]
        enabled = false

Afterwards apply the ConfigMap to your cluster and you are done.

The configuration option for individual containers to deactivate or activate the environment variable collection is available since the agent version ciprod11292018.

Add AZMON_COLLECT_ENV to the env section in your templates and set the value to False. It works for new containers or updating existing ones.

...
    spec:
      containers:
      - name: azure-vote-front
        image: REDACTED.azurecr.io/azure-vote-front:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 250m
          limits:
            cpu: 500m
        env:
        - name: REDIS
          value: "azure-vote-back"
        - name: AZMON_COLLECT_ENV
          value: "False"

Check Azure Monitor for containers for the successful configuration. In my example I deployed an application with two pods. On one, the azure-vote-front, I have disabled the environment variable collection.

azuremonitor03azuremonitor04azuremonitor05

For further details and what settings are available to configure the data collection follow this section in Azure docs.

-> https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-agent-config#overview-of-configurable-data-collection-settings

Facebooktwittergoogle_pluslinkedinmail