Multi-architecture container images allow you to have images for other operating systems and
system architectures that are referenced by the same tag. An example would be if you want to
pull the alpine:3.12 image and you have a 64 bit Intel linux server, and a 64 bit ARM EC2 instance.
Because the alpine image supports multiple architectures, you can pull it on both systems and get
the right image for your platform. There is no need to try to guess a unique namespace, repository
name, or tag for either image.
Building these multi-architecture images can be fairly daunting. Unless you are using BuildKit you need to build each image separately, stitch each of them together, and then push all of the pieces along with a manifest list to a container registry. This requires a lot of highly customized build infrastructure, and can be very complex to set up.
Architectures:
| architecture | bits | notes |
|---|---|---|
| linux/amd64 | 64 bit | Intel/AMD 64 bit |
| linux/386 | 32 bit | Intel 32 bit |
| linux/arm/v6 | 32 bit | RPi |
| linux/arm/v7 | 32 bit | RPi 2/RPi 3 |
| linux/arm/v8 | 32 bit | |
| linux/arm64 | 64 bit | 64 bit ARM |
| linux/ppc64le | 64 bit | IBM Power Architecture |
| linux/s390x | 64 bit | IBM z/Architecture |
| windows/amd64 | 64 bit | Windows |
BuildKit supports several different ways of building container images of different architectures and platforms, each with different strengths and weaknesses. At present we only support building in cross-compilation mode, however, the other modes will be supported in the future.
- Cross-compilation mode (supported)
Cross-compilation is the fastest way to build a multi-architecture image, however it can be difficult to set up. It depends upon whether your compiler supports cross-compilation, as well as the complexity of your project. For something such as a simple statically compiled Go build, this can work quite well, but if you're building a C or C++ application, and you're using dynamically compiled binaries, it may be quite a bit trickier.
Speed of the build really comes down to the number of nodes in the cluster, as well as the speed of those nodes. It's also easier to set up the cluster, since all of your nodes can be of the same type.
- Mixed cluster mode (not yet supported)
Mixed cluster mode will pick the correct node for a given architecture and will natively compile images for that architecture. You will need to have nodes of each architecture for this to work, which can be difficult with more rare architectures.
Speed of the build is comparable to doing cross-compilation, and it's typically easier to get binaries to compile correctly.
- QEMU "Emulation" mode (not yet supported)
QEMU emulates each of the architectures on a single architecture type (linux/amd64). Since this is full emulation, it can be quite slow to build everything. This mode is useful if it's too difficult to get your build to cross-compile and you don't have access to machines to build natively.
The Docker runtime doesn't support multi-architecture images natively. It's only possible to pull the part of a multi-architecture image which is native to your platform, and you cannot hold images for other platforms in your local image cache. To handle a multi-architecture image after you have built it, you need to push it to a container registry.
To push to a registry, you're going to first need to create a registry secret. Do this with the command:
kubectl create secret docker-registry reg-secret --docker-server=my_registry --docker-username=my_user --docker-password=$MY_PASSWORD
You will need to set the docker-server to the hostname of the registry that you are using, and the
docker-username and docker-password settings to your login credentials for that registry. If possible,
try to use an API key for the registry so that you don't expose your plain text credentials. When you
are doing the build, you can pass the flag --registry-secret=reg-secret which will tell the builder
to use that secret to push the image.
As a convenience, if you are only ever pushing to a single registry, it's possible to skip specifying the
--registry-secret flag to kubectl build by naming the secret the same name as the builder. The default builder is
called buildkit, so if you create a secret named the same thing, it will save you a few extra
keystrokes when building your image.
Configuring the Dockerfile can be somewhat difficult to set up, and will be different depending on how
you compile your application. BuildKit uses multi-stage Dockerfiles to create the build, and expects
certain stages to exist to create images for a specific platform. Here's an example Dockerfile.cross
from our example repository.
FROM --platform=$BUILDPLATFORM golang:1.11-alpine AS builder
RUN apk add --no-cache git
RUN go get github.com/pdevine/go-asciisprite
WORKDIR /project
COPY *.go ./
ARG TARGETOS
ARG TARGETARCH
ENV GOOS=$TARGETOS GOARCH=$TARGETARCH
RUN CGO_ENABLED=0 go build -a -ldflags '-extldflags "-static"' -o pants pants.go
FROM scratch AS release-linux
COPY --from=builder /project/pants /pants
ENTRYPOINT ["/pants"]
FROM mcr.microsoft.com/windows/nanoserver:1809 AS release-windows
COPY --from=builder /project/pants /pants.exe
ENTRYPOINT ["\\pants.exe"]
FROM release-$TARGETOS
The initial builder stage is used to cross-compile the pants.go binary for each
platform, and then the uses the release-windows and release-linux to create images for
every windows and linux platform, regardless of the architecture. To build the image, you
will use the command:
$ kubectl build ./ -t <server>/<namespace>/<repositor>:<tag> -f Dockerfile.cross --registry-secret my-secret --platform=linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64,windows/amd64 --push
Once this has built all of the images, it will push everything to the container registry and image tag that you specified in
-t <server>/<namespace>/<repositor>:<tag>. To make this work with your own images, you will need
to adapt the Dockerfile to allow you to cross-compile correctly.