Last year I wrote a blog post about removing dangling container manifests from ACR.
-> https://www.danielstechblog.io/remove-dangling-container-manifests-from-azure-container-registry/
I did not cover an edge case when it comes to multi-arch container manifests. So, here we are, and I walk you through that topic today.
First, do not be afraid the PowerShell script from last year works perfectly with multi-arch images created by the default method.
Normally, multi-arch container manifests get created doing multiple steps after another. Let us assume we want to have a multi-arch container manifest containing an amd64 and an arm64 manifest. In our first step we build an amd64 and an arm64 image and push those images to our ACR. Then we create our final multi-arch container manifest which references the amd64 and arm64 one.
In the end we have three container manifests in our ACR with three different tags. When we push new versions with the same tags the previous ones get untagged and will be removed during the next script execution. So, no big deal.
Now let us take a look into the edge case building multi-arch containers manifests with docker buildx.
docker buildx
Using docker buildx simplifies the multi-arch container manifest build by reducing a multi-step process into one. The only downside is that the referenced amd64 and arm64 container manifests are pushed to the ACR as untagged manifests.
> (Get-AzContainerRegistryManifest -RegistryName "azstcr" -RepositoryName "ubuntu").ManifestsAttributes | Sort-Object -Property LastUpdateTime -Descending
Digest : sha256:99175bbbd182bded8342c0ef2bba93a02b7d864f4746624c70749d39a5cdbe58
ImageSize : 0
CreatedTime : 2022-02-17T21:19:24.7867985Z
LastUpdateTime : 2022-02-17T21:19:24.7867985Z
Architecture :
Os :
MediaType : application/vnd.docker.distribution.manifest.list.v2+json
ConfigMediaType :
Tags : {latest}
ChangeableAttributes : Microsoft.Azure.Commands.ContainerRegistry.Models.PSChangeableAttribute
Digest : sha256:685665a5c8d85acb5974bd2c552f1a1155b525086cbe5320cc0f56a856d92f45
ImageSize : 27169640
CreatedTime : 2022-02-17T21:19:24.4302273Z
LastUpdateTime : 2022-02-17T21:19:24.4302273Z
Architecture : arm64
Os : linux
MediaType : application/vnd.docker.distribution.manifest.v2+json
ConfigMediaType : application/vnd.docker.container.image.v1+json
Tags :
ChangeableAttributes : Microsoft.Azure.Commands.ContainerRegistry.Models.PSChangeableAttribute
Digest : sha256:e50152615c84948a44845d5cf7be7463dcc4d38b5c0718f7dd3c243c606cce0c
ImageSize : 28564099
CreatedTime : 2022-02-17T21:19:24.0868396Z
LastUpdateTime : 2022-02-17T21:19:24.0868396Z
Architecture : amd64
Os : linux
MediaType : application/vnd.docker.distribution.manifest.v2+json
ConfigMediaType : application/vnd.docker.container.image.v1+json
Tags :
ChangeableAttributes : Microsoft.Azure.Commands.ContainerRegistry.Models.PSChangeableAttribute
Let us assume we already built our multi-arch container manifest and pushed it into the ACR by running the following command.
> docker buildx build --push --platform linux/arm64,linux/amd64 --tag azstcr.azurecr.io/ubuntu:latest .
We pull the image to our workstation which in the end succeeds.
> docker pull azstcr.azurecr.io/ubuntu:latest latest: Pulling from ubuntu 08c01a0ec47e: Pull complete Digest: sha256:44959906cfef41a65e4019acf6e4944059100a8682d45f45d4cfaa0a07c166a5 Status: Downloaded newer image for azstcr.azurecr.io/ubuntu:latest azstcr.azurecr.io/ubuntu:latest
Now we execute our PowerShell script and try to pull the image on another workstation.
> pwsh Remove-UntaggedManifests.ps1 ------------------------ Delete dangling image ubuntu@sha256:df52b11a7a3512d98117f3277363d03b5d0dc89be9777893366f01f59c1dae45 True ------------------------ Delete dangling image ubuntu@sha256:6668c313d6c9395b1132d214aa716916f86be9754fac58186e44ada94866c7e9 True > docker pull azstcr.azurecr.io/ubuntu:latest latest: Pulling from ubuntu manifest for azstcr.azurecr.io/ubuntu:latest not found: manifest unknown: manifest sha256:df52b11a7a3512d98117f3277363d03b5d0dc89be9777893366f01f59c1dae45 is not found
The image pull fails as the image cannot be found which is true. As earlier mentioned docker buildx does not tag the referenced amd64 and arm64 container manifests. Hence, they get deleted by the PowerShell script. We have only the tagged multi-arch container manifest in our ACR with the empty manifest references.
Solution
The solution for this problem is an additional step in the script checking for referenced manifests and putting them onto an exclusion list.
...
foreach ($ITEM in $MANIFESTS) {
$TAG = $ITEM.digest
$ITEM_DETAILS = Invoke-RestMethod -Uri https://$ACR_URL/v2/$REPO/manifests/$TAG -Authentication Basic -Method Get -Credential $CREDENTIAL -Headers $HEADERS
if ($ITEM_DETAILS.manifests -ne $null) {
$EXCLUDE_LIST += $ITEM_DETAILS.manifests.digest
}
if ($ITEM.Tags -eq $null -and $ITEM.digest -notin $EXCLUDE_LIST) {
Write-OutPut "------------------------"
Write-Output "Delete dangling image $REPO@$TAG"
Remove-AzContainerRegistryManifest -RegistryName $ACR.Name -RepositoryName $REPO -Manifest $TAG
}
}
...
Unfortunately, Azure PowerShell has no cmdlet for doing this and we must fall back to the ACR REST API here.
The additional check overall increases the runtime of the script itself. That is the reason for having two different versions of the script in my GitHub repository now. One for the default method of creating multi-arch container manifests or using only amd64 container manifests for instance and one for creating multi-arch container manifests with docker buildx.
-> https://github.com/neumanndaniel/kubernetes/blob/master/acr/Remove-UntaggedManifestsDockerBuildx.ps1