Is there any way to speed up the yarn linking step in a docker multi-arch build?

Question

So I am building a repository, leveraging Yarn 2 with offline dependencies and the node-modules linker. Linking on amd64 takes about 1.5 minutes, which is ok, but on arm64 it takes almost 20 minutes.

I understand that ARM is only emulated and therefore expected to be slower, but can this be sped up by any chance? As far as I understand it, yarn just extracts the correct zip folders into node_modules, so theoretically it should be very predictable.

A zip folder with a name of, e.g. react-npm-18.2.0-1eae08fee2-88e38092da.zip should always result in the same output folder inside node_modules.

Is there any way of caching the modules in docker/build-push-action?

Note: Just linking/building on the host and then copying in the result is not an option, as some dependencies provide different versions for different cpu architectures, which means that the final dependencies in node_modules do not match accross archs.

Also, layers are already cached, but it doesn't help much, as a single dependency change triggers a whole new run for that layer.