How to embed secrets into a file that the Dockerfile accesses via GitHub Actions

Question

Need:

• I am trying to create a Docker image that is needed for Google's Dataflow resource.
• The Docker file needs to, among other things, perform a pdm install (pdm is a Python build tool similar to poetry)
• Some of what it is installing are private packages on our GCP Artifact Registry.
• I am using the Buildx docker build and push GitHub action.
• So I need to inject secrets into the build and push action, that can then be read by the Dockerfile.

Search:

I have looked at the following possible solutions, and I cannot get anything to work for me:

• SO: Docker passing secrets
• SO: Accessing Docker secrets
• Docker Docs: using GHA secrets with Dockerfile
• Docker Docs: examples of secrets with Dockerfile

Representative Example of my case:

There are many tasks that I am trying to do, so I will just create a simplified set of files that are representative of my issue.

docker-build.yaml GitHub Action:

env:
  PYTHON_REPO_PASSWORD: ${{ secrets.PYTHON_REPO_PASSWORD }}
.
.
.
      - name: mytest
        run: |
           echo ${{ env.PYTHON_REPO_PASSWORD }}
           cat ${{ inputs.working-directory }}/pyproject.toml
           cat ${{ inputs.working-directory }}/pyproject.toml > ${{ inputs.working-directory }}/pyproject.toml
      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          push: true
          tags: my/app:latest
          file: ${{ inputs.working-directory }}/${{ inputs.dockerfile }}
          context: ${{ inputs.working-directory }}
        secrets: inherit # I have also tried by explicitly setting the secrets

The mytest step is in there to verify if the secrets are even in there. Yes, the secret is "shown" (with only ***, of course), including in the pyproject.toml file, described below, so that the URL for the artifact registry with the password embedded is properly rendered.

Dockerfile:

FROM apache/beam_python3.11_sdk:2.51.0 AS foundation
WORKDIR /project
COPY pyproject.toml pdm.lock README.md ./
COPY src/ ./src
RUN --mount=type=secret,id=PYTHON_REPO_PASSWORD \
    pip install -U pip setuptools wheel pdm && \
    pdm install --prod --no-lock --no-editable

The final step here, the pdm install with only flags after it, is the step that builds what is in the pyproject.toml file below.

pyproject.toml Python build file:

[project]
name = "aggregatedata"
dependencies = [
    "apache-beam[gcp]>=2.51",
    "phxdataflow", # This is the private package that I need to access
    "google-cloud-logging>=3.8.0",
]

[[tool.pdm.source]]
name = "insights_python_packages"
url = "https://_json_key_base64:[email protected]/prj-phalanx-mgmt-infra/insights-python-packages/simple/"

Again, in the mytest step, the pyproject.toml file above is being rendered properly so that the URL has the secret embedded into it.

But when the Dockerfile RUN pdm install command runs, the secret is not being injected. I cannot get a detailed log of the Dockerfile, except that pdm is unable to find phxdataflow when I use the secret, and it works fine if I just paste the actual password. So I know the problem is somehow with the injection into pyproject.toml during the Dockerfile build steps.

Solution

I don't need to solve it in the way described above... I just need for a secret that exists in the GHA build agent environment to make it into the pyproject.toml file when the Dockerfile is reading it.