How to build a docker image with tensorflow-nightly and the tensorflow object detection research models

Question

Since GPU support with tensorflow-nightly is currently broken on Google Colab I'm trying to build my own docker image for development. However, when I install the object_detection package from tensorflow/models my nightly tensorflow package is overwritten by the version pulled in as a dependency from the object_detection setup.py.

I'm following essentially the same steps in Google Colab but my tensorflow nightly isn't overwritten there, so I'm not sure what I'm missing...

Here's my Dockerfile:

FROM tensorflow/tensorflow:nightly-gpu-jupyter

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

RUN apt-get install -y \
        curl \
        git \
        less \
        zip

RUN curl -L -O https://github.com/protocolbuffers/protobuf/releases/download/v3.11.4/protoc-3.11.4-linux-x86_64.zip && unzip protoc-3.11.4-linux-x86_64.zip

RUN cp bin/protoc /usr/local/bin

RUN git clone --depth 1 https://github.com/tensorflow/models
RUN cd models/research && \
        protoc object_detection/protos/*.proto --python_out=. && \
        cp object_detection/packages/tf2/setup.py . && \
        python -m pip install .

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

which I'm building with:

docker pull tensorflow/tensorflow:nightly-gpu-jupyter
docker build --no-cache . -f models-tf-nightly.Dockerfile -t tf-nightly-models

The first print() shows:

Tensorflow version: 2.5.0-dev20201129

but the second one shows:

Tensorflow version: 2.3.1

---

In Google Colab I'm doing essentially the same steps:

# Install the Object Detection API
%%bash
pip install tf-nightly-gpu
[[ -d models ]] || git clone --depth 1 https://github.com/tensorflow/models
cd models/research/
protoc object_detection/protos/*.proto --python_out=.
cp object_detection/packages/tf2/setup.py .
python -m pip install .

After which

import tensorflow as tf
print(tf.__version__)

prints 2.5.0-dev20201201

So somehow my Google Colab steps are preserving my nightly Tensorflow install, whereas on Docker it gets overwritten with 2.3.0.

Answer 1

If you look at pip list before installing the object detection package, you will see that tf-nightly-gpu is installed but tensorflow is not. When you install the object detection package, the tensorflow package is pulled in as a dependency. pip thinks it is not installed, so it installs it.

One way around this is to trick pip install thinking that the tensorflow package is installed. One can do this by symlinking the tf_nightly_gpu-VERSION.dist-info directory in dist-packages. I have added the lines to do this in the Dockerfile below. At the bottom of this post, I have also included a Dockerfile which implements some best practices to minimize image size.

FROM tensorflow/tensorflow:nightly-gpu-jupyter

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

RUN apt-get install -y \
        curl \
        git \
        less \
        zip

# Trick pip into thinking that the 'tensorflow' package is installed.
# Installing `object_detection` attempts to install the 'tensorflow' package.
# Name the symlink with the suffix from tf_nightly_gpu.
WORKDIR /usr/local/lib/python3.6/dist-packages
RUN ln -s tf_nightly_gpu-* tensorflow-$(ls -d1 tf_nightly_gpu* | sed 's/tf_nightly_gpu-\(.*\)/\1/')

WORKDIR /tf
RUN curl -L -O https://github.com/protocolbuffers/protobuf/releases/download/v3.11.4/protoc-3.11.4-linux-x86_64.zip && unzip protoc-3.11.4-linux-x86_64.zip

RUN cp bin/protoc /usr/local/bin

RUN git clone --depth 1 https://github.com/tensorflow/models
RUN cd models/research && \
        protoc object_detection/protos/*.proto --python_out=. && \
        cp object_detection/packages/tf2/setup.py . && \
        python -m pip install .

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

---

Here is a Dockerfile that leads to a slightly smaller image (0.22 GB uncompressed). Notable changes are clearing the apt lists and using --no-cache-dir in pip install.

FROM tensorflow/tensorflow:nightly-gpu-jupyter

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

RUN apt-get install -y --no-install-recommends \
        ca-certificates \
        curl \
        git \
        less \
        zip && \
    rm -rf /var/lib/apt/lists/*

# Trick pip into thinking that the 'tensorflow' package is installed.
# Installing `object_detection` attempts to install the 'tensorflow' package.
# Name the symlink with the suffix from tf_nightly_gpu.
WORKDIR /usr/local/lib/python3.6/dist-packages
RUN ln -s tf_nightly_gpu-* tensorflow-$(ls -d1 tf_nightly_gpu* | sed 's/tf_nightly_gpu-\(.*\)/\1/')

WORKDIR /tf
RUN curl -L -O https://github.com/protocolbuffers/protobuf/releases/download/v3.11.4/protoc-3.11.4-linux-x86_64.zip && \
    unzip protoc-3.11.4-linux-x86_64.zip && \
    cp bin/protoc /usr/local/bin && \
    rm -r protoc-3.11.4-linux-x86_64.zip bin/

# Upgrade pip.
RUN python -m pip install --no-cache-dir --upgrade pip

RUN git clone --depth 1 https://github.com/tensorflow/models
WORKDIR models/research
RUN protoc object_detection/protos/*.proto --python_out=. && \
    cp object_detection/packages/tf2/setup.py . && \
    python -m pip install  --no-cache-dir .

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"