exit code: 139 on dockerfile build of from distroless image

Question

I'm trying to build a python docker image based on a distroless image in order to comply with security requirements. I previously had followed this article with great success. However, I came back after a vacation to find that the same working dockerfile was throwing the following error:

executor failed running [/bin/sh -c echo "monty:x:1000:monty" >> /etc/group]: exit code: 139

Exit code 139 is clearly a "segmentation fault"? Some sort of memory access issue? Regardless, I have no clue how to debug such a thing. I'm on a mac with docker --version = Docker version 20.10.12, build e91ed57.

Simplest dockerfile to reproduce:

## -------------- layer to give access to newer python + its dependencies ------------- ##

FROM python:3.8-slim as python-base

## ------------------------------- distroless base image ------------------------------ ##

# build from distroless C or cc:debug, because lots of Python depends on C
FROM gcr.io/distroless/cc:debug

ARG CHIPSET_ARCH=x86_64-linux-gnu

## ------------------------- copy python itself from builder -------------------------- ##

# this carries more risk than installing it fully, but makes the image a lot smaller
COPY --from=python-base /usr/local/lib/ /usr/local/lib/
COPY --from=python-base /usr/local/bin/python /usr/local/bin/python
COPY --from=python-base /etc/ld.so.cache /etc/ld.so.cache

## -------------------------- add common compiled libraries --------------------------- ##

# If seeing ImportErrors, check if in the python-base already and copy as below

# required by lots of packages - e.g. six, numpy, wsgi
COPY --from=python-base /lib/${CHIPSET_ARCH}/libz.so.1 /lib/${CHIPSET_ARCH}/
COPY --from=python-base /lib/${CHIPSET_ARCH}/libexpat* /lib/${CHIPSET_ARCH}/
COPY --from=python-base /lib/${CHIPSET_ARCH}/libc.so.6 /lib/${CHIPSET_ARCH}/
# required by google-cloud/grpcio
COPY --from=python-base /usr/lib/${CHIPSET_ARCH}/libffi* /usr/lib/${CHIPSET_ARCH}/

## -------------------------------- non-root user setup ------------------------------- ##

COPY --from=python-base /bin/echo /bin/echo
COPY --from=python-base /bin/rm /bin/rm
COPY --from=python-base /bin/sh /bin/sh

RUN echo "monty:x:1000:monty" >> /etc/group
RUN echo "monty:x:1001:" >> /etc/group
RUN echo "monty:x:1000:1001::/home/monty:" >> /etc/passwd

# quick validation that python still works whilst we have a shell
RUN python --version

RUN rm /bin/sh /bin/echo /bin/rm

## --------------------------- standardise execution env ----------------------------- ##

# default to running as non-root, uid=1000
USER monty

# standardise on locale, don't generate .pyc, enable tracebacks on seg faults
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONFAULTHANDLER 1

ENTRYPOINT ["/usr/local/bin/python"]

I've tried various combinations of chmod and I've tried to cut back and/or increase the number of files being copied from the python-base image with no change in behavior.

Answer 1

I never really solved this problem technically, but what I learned is that the google distroless python images are nightmare to work with. Chainguard provides a more stable and easy to use set of python distroless images that I got up and running within minutes. Way better than trying to build from a distroless version of C and doesn't have ANY security vulnerabilities out of the box. Also way better than learning Nix or anything like it for something so simple.