Daniel's Tech Blog

Cloud Computing, Cloud Native & Kubernetes

Remove dangling container manifests from Azure Container Registry

The Azure Container Registry offers three different SKUs which differentiate from each other not only from the feature set. Each SKU comes with included storage starting at 10 GB up to 500 GB.

ACR overview

Depending on the usage pattern the included storage fills up quickly. This can be due to a lot of different container manifests from successful build pipelines or from failed pipelines pushing the container manifests each time with the same tag to the ACR.

The latter one is problematic and is the major reason for a full ACR which then runs in storage overage usage. Such container manifests are called dangling container manifests or untagged manifests.

When you push an image to the ACR with the same tag repeatedly the ACR overwrites the previous manifest. It does it by untagging the existing container manifest to accommodate the new version. The already existing container manifest becomes an untagged manifest and still consumes storage space.

Using the ACR Premium SKU you prevent your registry from filling up by activating the retention feature. After X days, the default is seven days, untagged manifests get deleted automatically.

-> https://docs.microsoft.com/en-us/azure/container-registry/container-registry-retention-policy

For the Basic and Standard SKU, you need a simple PowerShell script achieving the same result. Such PowerShell script is executed by an Azure Function or a GitHub Action on a regular schedule.

$ACRS = Get-AzContainerRegistry
foreach ($ACR in $ACRS) {
  $REPOS = Get-AzContainerRegistryRepository -RegistryName $ACR.Name
  foreach ($REPO in $REPOS) {
    $MANIFESTS = (Get-AzContainerRegistryManifest -RegistryName $ACR.Name -RepositoryName $REPO).ManifestsAttributes | Where-Object { $_.Tags -eq $null } | Sort-Object -Property LastUpdateTime -Descending
    foreach ($ITEM in $MANIFESTS) {
      $TAG = $ITEM.digest
      Write-OutPut "------------------------"
      Write-Output "Delete dangling image $REPO@$TAG"
      Remove-AzContainerRegistryManifest -RegistryName $ACR.Name -RepositoryName $REPO -Manifest $TAG
    }
  }
}

-> https://github.com/neumanndaniel/kubernetes/blob/master/acr/Remove-UntaggedManifests.ps1

The script simply iterates over every ACR in an Azure subscription and then over every repository of an ACR. Each untagged manifest is deleted. As untagged manifests are independent from the tagged manifest the deletion does not affect the production container manifest.

------------------------
Delete dangling image akscnicalc@sha256:af59dd54b997e61081efbfc7ed63647b01e44c002d9513a8db349a3a6b75f1a0
True
------------------------
Delete dangling image azp@sha256:82a0acbc2b10af684dd3379b7383e31446029ad8b5d0f9ce2de2b733f907dd9f
True

Summary

The ACR Premium SKU provides the mechanism for the clean-up of untagged manifests out-of-the-box. The other SKUs require a script for that.

What all SKUs have in common is that old manifests need to be untagged or cleaned up again by some automation mechanism. Currently, the Azure Container Registry does not provide a built-in mechanism to automatically untag container manifests that are older than X days.

So, you need to implement such mechanism yourself via a script.


Posted

in

WordPress Cookie Notice by Real Cookie Banner