In this article, I’d like to talk about the purpose and usages of a little-known feature of the Docker ecosystem, the Docker manifests, and about the circumstances through which I discovered them.
What problem brought me to Docker manifests ?
I recently received my first M1 MacBook (meaning it uses an Apple built M1 chip), and started a new project, on which I had to develop a classic application relying on a Mysql database. The database needed to be runnable as a container to facilitate integration tests, both on my machine and in the CI pipeline (running on Linux instances).
So, as usual when needing a new Docker image, I tried getting it from DockerHub
. But unlike every other time for the past 5 years, it did not work! And instead I saw this message:
What does this message mean exactly? And why is it the first time I see it?
This message tells me that there is no existing Docker image called mysql with a tag 5.7 available on DockerHub, and that I could run on my M1 Macbook. Note that the notion of manifest was already mentioned here, but I didn’t notice it at first.
As for why I never saw this before, it’s because up to that point I could usually use any image I downloaded from DockerHub, having always worked with Linux/Mac distributions that relied on an amd64 platform/architecture. But the Mac M1 (and now M2) is still new enough that not all of the most popular tools have provided a compatible Docker image on DockerHub for arm64v8 (the architecture coming with the M1 chip), and especially not for older versions of their tools (Mysql 5.7 is more or less deprecated in favor of Mysql 8.0).
When you get this message, you have 2 options:
- find another source for your image (another DockerHub account/image or from another repository)
- build your own image, assuming that the Dockerfile and required resources are open sourced
In my case, I managed to find a compatible image on another repository: mysql/myqsl-server.
All well and good then? Not exactly. I still had to address the DockerHub rate limitations if I wanted to have a reliable pipeline that wouldn’t fail randomly when hitting those limits.
The next part explains what impact those limitations have and what you can do to get around it.
DockerHub rate limitations and multi-platform support
Since the “recent” implementation of the DockerHub rate limits for free and anonymous users, most of the companies which do not want to pay for a DockerHub professional account have had to set up their own Docker registries. You obviously don’t want your CI or even CD (Continous Deployment) pipelines to randomly fail to download a Docker image because someone else in the company is using the same IP and doing a lot of downloads themselves (anonymous users are identified by IPs and other similar means).
Replacing DockerHub with your own registry is not that complex or costly if you already use one of the main cloud providers, which have their own solutions (i.e. AWS ECR, Azure CR). Additionally, it often simplifies security measures and speeds up images downloads, since your containers and hosts are in the same network and provider.
In my case, we were using AWS and had thus access to our own registry via AWS ECR.
So all I needed to do was push the Docker image that works for me to ECR and the problem was solved, right? Well, not quite. I had now 2 places where I wanted to run my tests (my Mac M1, and the CI pipeline running on Linux), which each needed a different image compatible with their own platform (amd64 and arm64v8).
The obvious solution was to push two images. If I pushed an image “mysql:5.7-amd64v8” and another “mysql:5.7-arm64” to ECR, then I just had to provide a different tag when running the tests in different places.
But then what if new colleagues joining the team are using older Macs, or even Windows machines? I would then need to introduce a way to set the right image depending on some local setup. This sounded like it could get quickly complicated to configure and maintain.
But wait a second! Why was I facing this problem pulling from a private docker registry, when I never faced it pulling from DockerHub before? There must be a better way to do this, right?
I had a couple of hints to help my search:
- when pulling from DockerHub the mysql/mysql-server:5.7 image, I didn’t need to specify a platform dependent tag, nor say which platform I wanted
- the error message above had an interesting line, “no matching manifest for linux/arm64/v8 in the manifest list entries”
The solution to this problem is something called a Docker manifest. One thing a Docker manifest can do, is to provide a list of images, each built for a specific platform. This is in fact how DockerHub automatically selects the right image for us.
The next sections describe what the manifests are, and how we can use them to solve this problem.
What are Docker manifests ?
Depending on their type, Docker manifests are either designed to contain an image metadata (list of layers), an image list indexed by platform, or a completely different type of data. They are available in 2 versions.
The first and deprecated version is called “manifest version 2, schema 1” and was introduced as a temporary transitional format between manifests version 1 (integral part of the old images) and version 2.
The second and current version is called “manifest version 2, schema 2”:
- introduced in 2017 during the final migration out of the old image V1 format
- now the only version you can create
- can contain a variety of information, depending on its mediaType field. For this article, we are only interested in the following two types:
- Image Manifest (application/vnd.docker.distribution.manifest.v2+json)
- Manifest List (application/vnd.docker.distribution.manifest.list.v2+json)
The image manifest is created automatically every time you push an image into a registry and is stored alongside the image. It describes a single image and its layers, similarly to the first manifest version. It will also be used by the Docker client to detect if the image you try to pull is compatible with your platform (arm64, amd64…).
The manifest list describes a list of image manifests, aka fat manifest. It has to be created separately from the image manifests it references, and uploaded on its own. From the perspective of the user, it looks like a Docker image with a tag in the registry, while in fact it acts more like a router for the pull request. This is the one we need.
The following command shows the Mysql (version 8) DockerHub manifest:
You will notice here that the manifest mentions two images, one for Amd64 platforms (Linux/Older Macs) and one for Arm64v8 (Mac M1/M2). This means that there are two images available for the “8” tag. Which one we get when pulling using the “8” tag will depend on our platform.
Note that usually you will never pull a manifest from a registry, but simply read it directly there. For this reason, you still need to have internet access to check the manifest of images that you have already pulled from the registry.
Create and use a manifest list
To understand how to create and manipulate manifests, we will build and push a manifest list, based on the mysql:8 Docker image. As we saw in the previous section, this image is available on DockerHub, both for Amd64 and Arm64V8 architectures. The following commands will allow us to make both images available in our private repository under a single Docker tag.
We start by downloading the image compatible with Mac M1. We then need to rename and retag it (the tag must be unique to each image), then push it to our private repository (in our case an AWS ECR repository):
Let’s do the same with the other platform:
We now have both images in our private repository. The next step is to create the manifest that will allow us to use a single tag “8”, instead of a special tag (“8-arm64” or “8-amd64”) every time we change platform:
We can check the content of that list manifest, which should contain 2 image manifests, one for each platform:
The last step is to push that manifest and check that we can pull the mysql image by using the platform agnostic tag:
And we are done!
Conclusion
Docker manifests are a convenient and transparent way to provide multi-architecture Docker images, without the need for the user to know what is really happening under the hood. The only restriction being that the private Docker registry you are using must be able to use those manifests - for AWS ECR, it has been possible since 2020.
Finally, it’s worth noting that Docker seems to have planned another usage for the Docker manifest, the creation of “content-addressable images”. This feature does not appear to be very widely used yet, and its actual purpose and potential are still unclear. But this will certainly change in the future evolutions of the Docker ecosystem, so you should keep an eye out for any manifest related updates in the future Docker versions.