I am using docker with buildkit and multi-stage builds to create a number of conda environments in parallel then copying them into my final stage. Many of the layers have large complex dependencies and copying each of the environments into the final stage taking a really long time.
An example:
FROM custom.docker.repo:4567/custom/conda/env:latest as conda
FROM conda as foo
COPY lib/foo/environment.yml /tmp/foo.yml
RUN conda env update -f /tmp/foo.yml && conda clean -afy
FROM conda as bar
COPY lib/bar/environment.yml /tmp/bar.yml
RUN conda env update -f /tmp/bar.yml && conda clean -afy
FROM conda as baz
COPY lib/baz/environment.yml /tmp/baz.yml
RUN conda env update -f /tmp/baz.yml && conda clean -afy
FROM conda as final
COPY --from=foo /opt/conda/envs/foo /opt/conda/envs/foo
COPY --from=bar /opt/conda/envs/bar /opt/conda/envs/bar
COPY --from=baz /opt/conda/envs/baz /opt/conda/envs/baz
....
I know that the intermediate layers do not have overlapping files and would like to "cherry pick" them directly onto the final stage's history without having to use COPY --from=... commands. Is this type of low-level overlay2 manipulation possible with docker or the buildkit tooling?