Multi-CPU Architecture builds with Docker — Understanding Docker manifest lists
Docker brought a massive change and practical movement in the virtualization world. And it couldn’t just be supported towards one operating system or one CPU architecture.
So, throughout the lifecycle of docker development, the docker platform expanded to multiple architectures like in 2013, docker spanned across x86_64 architecture, in 2014 docker spanned to ARM architectures etc.
So, this brought the fantastic capability to build, ship and run containerized workloads on almost any underlying architecture.
So, what was the problem?
The problem was that for every individual architecture, the docker image was supposed to be built corresponding to that architecture and run only on that architecture.
For example,
Let’s say, I want to create and run a golang docker container on one host backed by amd64 (x86–64, linux hosts) and on another host backed by ARM architecture.
So, this involved the following tasks:
- For the one backed by amd64 — I would have to find and download the docker image of golang specifically built and made for amd64. And then, run it.
docker run amd64/golang- For the one backed by ARM — I would have to manually find and download the docker image of golang specifically build and made for ARM. And then, run it.
docker run arm64v8/golang
So, the problem was that there wasn’t any capability of creating one single unified parent docker image which could be downloaded on any operating system (or over any architecture) and just be run, and docker behind the scenes will automatically make sure that it runs the image made for the corresponding arch. or operating system.
So, basically, that’s what we wanted,
Same command/image name producing desired result over all the architectures.
docker run super-golang#### docker, behind the scenes, automatically pulls and runs the golang image built for arm64 or amd64 depending on the host arch.
So, what was the exact solution from docker’s perspective?
Docker expects the developer to build and push docker images corresponding to each architecture.
Extending the above example,
docker expects the image of arm64v8/golang and amd64/golang to be individually built and pushed to docker hub (or any other registry :P)
Now, what we expect from docker is that when we run “docker run super-golang” then, docker determines the arch. of the host, downloads the single rightful image for that architecture (arm64v8/golang for ARM , amd64/golang for AMD) and runs it.
So, how do we make docker do this? Enter “Docker manifest lists”
So, here we create a manifest list with the name “super-golang” and it contains a list of platform/architecture segregated list of entries consisting to all the different images for different architectures.
But what is a docker manifest, in the first place?
So, every docker image is represented by a manifest. A manifest is a JSON file containing all the information about that docker image like information about it’s sub-layers, the size allocation, the platform on which individual layer is capable of being run, etc.
For dealing with manifests, enable experimental mode in docker on your OS by adding “experimental”: “enabled” in your ~/.docker/config.json file
Here is a pic of the docker manifest of nginx image on my laptop.
Back to the manifest lists
Hence, through the power of “manifest” docker can know, without the developer telling it, that which image corresponds to be run on which architecture, by inspecting the “architecture” in that image’s manifest.
So, when we create that manifest list and push it to docker hub, it pushes all those “child” images to docker hub, along with the manifest named “super-golang”.
And when finally, a user or client having a Linux computer (having underlying architecture of amd64) runs the command “docker run super-golang”, docker performs the following steps behind the scenes:
- It searches dockerhub for “super-golang” and finds that it’s a manifest list object.
- It downloads the manifest list corresponding to “super-golang”.
- It then looks at all the entries of that manifest list and determines which is the rightful entry (child image) to run on the user’s computer on the basis of their computer’s underlying architecture (that is, the one corresponding to amd64 for that user)
- And finally, it pulls rightfully determined child image made for amd64 architecture, on the user’s computer and runs it.
Time for Hands-on!
Let’s take the same example of two golang docker images (amd64v8 and arm64) to be pushed under a single manifest-file named “super-golang”
- So, say my laptop has two dockerfiles for building each of those docker images.
ls> Dockerfile.amd64
Dockerfile.arm64v8
Step 1). Build those dockerfiles to produce the respective docker images.
### Creates the "arm64v8/golang" image by building Dockerfile.arm64v8
docker build -t arm64v8/golang -f Dockerfile.arm64v8 .### Creates the "amd64/golang" image by building Dockerfile.amd64
docker build -t amd64/golang -f Dockerfile.amd64 .
Step 2). Now, write the manifest list with the name “super-golang” consisting of details about the child images with their platform details
Step 3). Install manifest-tool from this link.
Step 4). Run the following command to push the manifest file. (I created my manifest.yml file here -> /home/yash1300/docker_manifest/manifest.yml)
manifest-tool push --from-spec /home/yash1300/docker_manifest/manifest.ymlOR you can do the same thing without dealing with this “manifest-tool”. But with that you would have to do the entire manifest-file thing with commands.
Here are the steps,
Step 2). Push the individual images to docker hub
docker push arm64v8/golang
docker push amd64/golangStep 3). Create the manifest lists.
docker manifest create super-golang:latest \
-amend arm64v8/golang \
-amend amd64/golangStep 4). Push the entire manifest file to docker hub.
docker manifest push super-golang:latest
And that’s it! You are ready with your multi-architecture supported and ready-to-use docker image.
Just run
docker run super-golangand Enjoy!
PS: I found this image depicting a sample flow of multi-arch. build with docker providing a capability to create and run an image on windows or linux depending on client’s OS.
End note
Thanks a lot of reaching till here :D
I hope you understood this article and found it informative. Don’t worry if something went over your head through this article, this topic is pretty complicated but again, it is gorgeous in it’s own essence, so read it again, you’ll surely would understand it further :)
If you liked it, do give it some claps :)
Feel free to connect with me on my handles
LinkedIn — https://www.linkedin.com/in/yashvardhan-kukreja-607b24142/
GitHub — https://www.github.com/yashvardhan-kukreja
Until next time,
Adios!
