Just got done compiling a buildx multi-arch docker container that includes amd64, arm64 and armv7. It took about two hours to build -- with the armv7 portion being particularly slow. It only takes about 15 minutes to build natively on a RaspberryPi3.
I've setup a Docker swarm with 4 nodes, including each of the architectures I want to build for, but I haven't been able to work out how to setup the buildx cluster to use these native nodes.
I'd appreciate some guidance from anyone that's been down this particular road!
Edit:
I've made some progress with this, and here's what I know so far:
buildx clusters have nothing to do with docker swarm -- it's a separate thing
ssh public key authentication needs to be setup between the ssh "client" and "server" with your platform-specific target build machines being the servers in this case, and here's a good link for doing that:
https://www.ssh.com/academy/ssh/copy-id
test that ssh authentication is working from the machine initiating your cluster, with:
docker -H ssh://username@hostname info
you should see docker client and server info returned
create a new builder on the machine you'll be running the buildx from in the form:
docker buildx create --name <your_cluster_name> --node <your_node_name> --platform <your_desired_build_platforms> --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=10000000 --driver-opt env.BUILDKIT_STEP_LOG_MAX_SPEED=10000000
create additional target build platforms just like the above, but add --append ssh://username@hostname to the options
docker buildx ls
is used to confirm the builder, nodes and platforms (the ones with an * should be what you specified and will override defaults.
docker buildx use <your_cluster_name>
will change the default builder
docker buildx inspect --bootstrap
will initiate your cluster multi-arch multi-platform builder, including downloading the buildkit image on each platform.
I had issues here, with one of my platforms requiring multiple attempts with docker buildx inspect --bootstrap. Ultimately, for my modest armv7 machine, I downloaded the:
moby/buildkit:buildx-stable-1
image on that machine first, and then ran buildx inspect from the machine I'm using to initiate the build. With the image already present, the container was spun-up without further issue.
It all looks to be setup correctly:
bnhf_cluster * docker-container
bnhf_cluster0 unix:///var/run/docker.sock running linux/amd64*, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
rpi6 ssh://pi@raspberrypi6 running linux/armv7*, linux/arm/v7
rpi5 ssh://pi@raspberrypi5 running linux/arm64*, linux/arm/v7, linux/arm/v6
Next, I'll find something new to try out a mutli-arch multi-platform build, as my primary project is all cached by buildx now -- so wouldn't be a good test.
I found this post to be the most helpful in getting multi-arch multi-platform buildx configured:
https://medium.com/@spurin/using-docker-and-multiple-buildx-nodes-for-simultaneous-cross-platform-builds-cee0f797d939
If you use portainer, you should see something like this in your container list on each of your platform specific targets, after the cluster is created:
Edit #2:
Wow, did that work well! I decided to build my primary application with the --no-cache option. The same build that took over 2 hours (with some layers already cached) on an 8th Gen i7 (4 cores, 8 threads, 32GB RAM) running Debian 11 in a WSL2 instance, took 742 seconds.
Sending the arm64 and arm/v7 jobs even to lowly Raspberry Pis is a huge improvement over the emulation offered with Docker Desktop.