Immutable Container Image Tags
Learn the benefits of immutable tags for container images and how to fix your GitHub Actions.
This week’s post is an excuse to write a note for my future self. Using immutable container registries and the docker build and push action always trips me up. My pipelines break because the action likes to push a latest tag.
This is a post in 3 parts. In the first part I will try to convince you that immutable tags are a good thing. Then I will show you how to configure popular registries. Finally we will review an example of a GitHub Actions workflow that won’t build a latest tag. The last one is the bit I know I will need in 6 months.
Immutable Images for Consistency
If you’re always deploying :latest you have no control over which image you’re using. Your engineers will struggle to debug issues if they don’t know which version is running in production. Tagging every release with a unique version, be it semver, calver, or a simple date time string allows you to track them properly. If there is a problem with a release, just create a new one. Don’t overwrite the existing tag.
If you’re always deploying :latest and you ship a bad image, you can’t rollback. The rollback will pull in the same bad latest image.
Knowing exactly which version you’re running helps improve security too. If there is a vulnerability, you can know if your environments are running the :202510100101 release or later, they’re patched. If you’re using :latest, who knows.
At any point you should be able to answer the question, what version of the code was running on Tuesday afternoon 3 months ago? You can check the release history and pull the artefacts, including container image based on the referenced tag.
Using immutable tags prevents accidents. Someone can’t accidentally overwrite an existing tag with a new release.
Image hashes provide truly immutable references, but they have one serious drawback. Like all strong hashes, SHA256 represented as a hexadecimal string isn’t very human friendly. Let’s take @sha256:83cba1e9751dbafcc63fb6f241adb6ab4c969dc665ce10732d98a2bbd2d3aeaa as an example. Do you want to read that out on a call during an incident? When was that image built? Immutable tags based on predictable versions are easier for humans to manage and process.
If you need to keep track of the “latest” release, you can store the tag in SSM Parameters or some other persistent store offered by your preferred cloud provider. You can then pull in the reference into your deployment pipeline. This way you keep the traceability of your versions without needing to resort to mutable tags like “latest” or “dev”. Using unique tags allows you to promote images through your various environments rather than rebuilding them each time. This reduces the amount of storage space your images consume.
Storing lots of old images can consume a lot of space. Some registries support lifecycle rules that allow you to purge old images. Others force you to resort to scripting to implement this. Either way you can remove old non production builds after a short period of time, while you will probably want to retain production images for an extended period.
Configuring Immutable Repositories
Many of the popular registries support immutable tags.
RedHat’s Quay has supported immutable tags since last year. Harbor registry, a CNCF graduated project, offers rules to control tag immutability. GitLab has the feature available in beta. GitHub’s Container Registry, JFrog’s Artifactory, and Sonatype’s Nexus are notable for lacking immutable tag configuration.
The big 3 cloud providers and Docker Hub all support immutable tags. Let’s look at how to configure it in each service.
Docker Hub
The immutable tags feature is in public beta on Docker Hub.. Users can turn this feature on via the UI. You can select to turn it on for all tags, no tags or tags matching a regular expression.
Alternatively we can use the official Docker provider for Terraform to manage our repository. Here is a quick example that makes all tags immutable.
resource "docker_hub_repository" "example" {
namespace = "my-organization"
name = "my-image"
description = "Short description of the image"
full_description = file("${path.module}/README.md")
private = true
immutable_tags_settings {
enabled = true # Enabled for all tags
}
}
Amazon ECR
AWS ECR makes it easy to make your tags immutable. Refer to the documentation if you use ClickOps or the AWS CLI for creating your ECR repositories.
If you prefer CloudFormation to create an ECR repo, you can configure tag immutability with the following YAML.
ECRMyImage:
Type: AWS::ECR::Repository
Properties:
RepositoryName: "my-image"
ImageTagMutability: "IMMUTABLE"
# ...
If you’re managing your ECR repositories with Terraform, the configuration is very similar. Here is an example.
resource "aws_ecr_repository" "my_image" {
name = "my-image"
image_tag_mutability = "IMMUTABLE"
# ...
}
Azure Container Registry
Of all the platforms I reviewed for this post, Azure has the most convoluted system for managing immutable tags. Rather than managing it at the repository level like other platforms, they force you to configure it for each individual tag. Once you push a tag you need to make an API call to mark it as immutable. Good luck if you want to enforce this across all of your repos.
GCP Artifact Registry
Google Cloud Platform’s Artifact Registry supports immutable tags. Like all the other sensible platforms they support repo level configuration. Let’s look at a quick Terraform example using the google_artifact_registry_repository resource.
resource "google_artifact_registry_repository" "example" {
location = "us-central1"
repository_id = "my-image"
description = "Short description of the image"
format = "DOCKER"
docker_config {
immutable_tags = true
}
}
Avoiding Latest Tag with GitHub Actions
By default Docker’s meta data action adds the latest tag. We need to explicitly remove it from the list of tags it will generate. We add type=raw,value=latest,enable=false as one of our tags. Let’s break it down. type=raw tells the action this is a fixed value tag. value=latest sets the value of the tag to “latest”. Finally we use enable=false to specify we don’t want the tag included. We also need to add latest=false to the flavor argument.
Let’s look at this as part of a GitHub Action release workflow. The workflow will run once a week or whenever a date time string based tag is pushed.
name: Release
on:
push:
tags:
- '202*' # YYYYMMDDHHmm TODO Update in January 2030
schedule:
- cron: '6 6 * * 6' # Every Saturday
env:
GHCR_IMAGE: "ghcr.io/$</span><span class="nv"> </span><span class="s">github.repository</span><span class="nv"> </span><span class="s">"
permissions:
packages: write # Needed to write to GHCR
contents: read
# id-token: write # Needed if using OIDC to publish to Cloud provider
jobs:
release:
runs-on: ubuntu-24.04
steps:
- name: Checkout
id: checkout
uses: actions/checkout@v5
- name: Extract metadata
id: meta
uses: docker/metadata-action@v5.8.0
with:
images: |
$
flavor: |
latest=false
prefix=
suffix=
tags: |
type=schedule,pattern=date 'YYYYMMDDHHmm'
type=ref,event=tag
type=raw,value=latest,enable=false # Don't create latest tag
# Swap out GHCR for your container registry of choice
- name: Log in to GHCR
id: ghcr-login
uses: docker/login-action@v3.6.0
with:
registry: ghcr.io
username: $
password: $
- name: Build and push image
id: image
uses: docker/build-push-action@v6.18.0
with:
push: true
tags: $
labels: $
I used GitHub Container Registry as the target in order to keep things simple. The workflow can be adapted to work with any container registry. Let’s run through the key steps.
We’re using date time strings with the YYYYMMDDHHmm format for tagging our images. This removes the need to figure out what the next semver tag should be. It also avoids the one release per day limitation of using calver.
As we’re using GHCR, we need to enable write permissions for packages. If we’re only deploying to a cloud provider, then we could remove the packages permission and allow writes to id-token so we can use OIDC.
We tell the docker/metadata-action to only create tags for our images using the git tag value or the current date and time for scheduled builds.
Next we authenticate to GHCR. We always authenticate as late as possible to minimise the other actions that have access to our session.
Finally we pass the tags from the metadata action to the build and push action. This will build the images and push them to GHCR. If we got the configuration right, the workflow won’t build a latest tag.
Wrap Up
Immutable tags help trace images lineage. Many container registries support immutable tags. Some make it easier to configure than others. Even if the platform doesn’t support immutable tags, you can configure your pipeline to not create images with the latest tag. This will make it easier to promote images through your environments. Consider enabling this in your pipelines.🌊
Need Help?
Do you need some help implementing the ideas in this post? Get in touch! I am happy to help.
Like and Subscribe
Did you like this post? Please subscribe so you don't miss the next one. Do you know someone who would benefit from this article? Please share it with them.
Proactive Ops is produced on the unceeded territory of the Ngunnawal people. We acknowledge the Traditional Owners and pay respect to Elders past and present.