Dockerfile can't copy specified local directory & file

Question

Tried a lot to make it work since days but since I'm unable to had to drop a query on SO to get it clarified.

So, I've following project structre for which I want to build dockerfile.

As you can see from above project structure, I've 2 files about which I'm concerned about. The jars inside lib and the file .databricks-connect

These are essentially the files generated after I configure databricks-connect using command databricks-connect on my local system.Since its an interactive process, we can't simulate that on docker container so I want to copy across my configurations as ready to use inside docker.

The bellow is my Dockerfile. (I'm pretty new to dockers hence I suspect my issue might be trivial)

FROM ubuntu:18.04

# System packages
RUN apt-get update && apt-get install -y curl

# Install miniconda to /miniconda
RUN curl -LO http://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
RUN bash Miniconda3-latest-Linux-x86_64.sh -p /miniconda -b
RUN rm Miniconda3-latest-Linux-x86_64.sh
ENV PATH=/miniconda/bin:${PATH}
RUN conda update -y conda

RUN apt-get update && \
apt-get install -y openjdk-8-jdk openjdk-8-jre

# Define working directory.
WORKDIR /app

# Define commonly used JAVA_HOME variable
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64
ENV JRE_HOME /usr/lib/jvm/java-8-openjdk-amd64/jre

COPY . /app

RUN conda create -n snowflake-graphdb python=3.7

ENV PATH /miniconda/bin:$PATH

# Activate the environment, and make sure it's activated:
RUN ["conda", "run", "-n", "snowflake-graphdb", "pip", "install", "-r", "requirements.txt"]

#COPY .databricks-connect /root
RUN ls -a
RUN "mkdir /miniconda/envs/snowflake-graphdb/lib/python3.7/site-packages/pyspark/jars"
COPY libs/  /miniconda/envs/snowflake-graphdb/lib/python3.7/site-packages/pyspark/jars/
RUN "touch /root/.databricks-connect"
RUN "cat /app/databricks-connect.txt > /root/.databricks-connect"

WORKDIR /app
ENV PYTHONPATH "${PYTHONPATH}://app/src/main/python"
ENTRYPOINT ["conda", "run", "-n", "snowflake-graphdb", "databricks-connect", "test"]
#ENTRYPOINT ["conda", "run", "-n", "snowflake-graphdb", "python", "/app/src/main/python/server/GraphTransformerServer.py"]
EXPOSE 1020

The following is error when I run docker built -t graphtransformer:latest . --no-cache

=> ERROR [14/17] COPY libs/  /miniconda/envs/snowflake-graphdb/lib/python3.7/site-packages/pyspark/jars/                                                                                                                         
------
> [14/17] COPY libs/  /miniconda/envs/snowflake-graphdb/lib/python3.7/site-packages/pyspark/jars/:
------
failed to solve with frontend dockerfile.v0: failed to build LLB: failed to compute cache key: "/libs" not found: not found

I tried other directories like docs etc but looks like can't read anything except src. So, I moved /libs to src/main/libs to fool the system to pick it up as well (Added entry in .gitignore to ignore the file) but it didn't pick from src/main/libs as well. Also, the command COPY .databricks-connect /root also failed with same error as above, i.e. unable to find the file.

Another thing I tried is, since I do COPY . /app I expected all files including libs & .databricks-connect are already inside docker context under /app so I tried doing

CMD "cp /app/libs/* /miniconda/envs/snowflake-graphdb/lib/python3.7/site-packages/pyspark/jars && cat /app/databricks-connect.txt > /root/.databricks-connect"

But that also failed with different error. The above command was getting appended to ENTRYPOINT. Don't know why.

What would be best way to mount the bellow files into docker container during build time so that the container as whole can be used with compose. These files are needed during RUNTIME, so without these files, the ENTRYPOINT will fails.

Any help will be really grateful.

Thanks in advance.

Answer 1

This is hard to me to give such answer (I think) as the dockerfile involves two different entry points that is another python build that I have no idea about (i.e. which entry point should be executed first). So just take this as a reference anyway.

It's a common practice that building an image from a java project be done after the entire project itself getting packed into one unit (.war/jar) where .class files are residing. As to my understanding docker doesn't rebuild your project again (more like just deploying or run the .war/jar itself) within the docker container. I might be wrong though. So it might better to pack the java project first.

There a guy here devops stuff took an unusual(maybe its common) approach that basically he extracted those lib, class, and META-INF folder along with their contents to a separate folder which is AProject/target/dependecy. In case you wonder the target folder is the default location for the generated .war/jar in Eclipse IDE so what he got basically :

 target/dependency/lib
  target/dependency/classes
  target/dependency/META-INF 

Then his docker build :

ARG DEPENDENCY=target/dependency
    COPY ${DEPENDENCY}/lib /app/lib
    COPY ${DEPENDENCY}/META-INF /app/META-INF
    COPY ${DEPENDENCY}/classes /app
  ENTRYPOINT ["java", "-cp", "app:app/lib/*","com.javapointers.Application"]

The dockerfile location is this case is directly within the project (at a the same level with the pom.xml)and finaly image is built.