How can pre-commit checks fail locally but pass on a CI server?

Question

I have a clean working tree of a branch on my development machine. If I run pre-commit run --all-files my formatter hooks fail, reformatting some files. My CI server (Atlassian Bamboo) also runs pre-commit run --all-files on the same branch, but the hooks pass, which is unexpected. What can I check to figure out why?

As an example, I added the following line to a Python file:

unused = 1+1

I have a black hook for formatting and a ruff hook for linting. Locally the output is:

black....................................................................Failed
- hook id: black
- files were modified by this hook

reformatted tasks/tool.py

All done! ✨  ✨
1 file reformatted, 1878 files left unchanged.

ruff.....................................................................Failed
- hook id: ruff
- exit code: 1

tasks/tool.py:76:5: F841 [*] Local variable `unused` is assigned to but never used
Found 1 error.
[*] 1 potentially fixable with the --fix option.

This is as expected -- black will reformat the line as unused = 1 + 1 (spaces around the plus operator) and ruff rightly identifies that unused is unused.

On the CI server, however, the output is:

black....................................................................Passed
ruff.....................................................................Failed
- hook id: ruff
- exit code: 1

tasks/tool.py:76:5: F841 [*] Local variable `unused` is assigned to but never used
Found 1 error.
[*] 1 potentially fixable with the --fix option.

The unexpected pass is not unique to black. I also have hooks for mdformat and clang-format that also fail locally and pass unexpectedly on the CI server.

Things I've tried:

• Added a call to pre-commit --version in both places to be sure they're the same (3.3.3)
• Added a call to pre-commit clean in both places to be sure there's not some weird caching issue for hooks
• Added the identity hook to be sure files aren't being filtered out differently somehow

What other things can I do to track this down?

Here is my (trimmed-down but hopefully still valid/representative) pre-commit-config.yaml:

default_language_version:
    python: python3.10

exclude: >
    (?x)^(
        acceptance/.*|
        build/.*|
        deploy/.*
    )$

repos:

-   repo: https://github.com/PyCQA/docformatter
    rev: v1.1
    hooks:
        -   id: docformatter
            alias: reformat-docs
            types: [python]
            args:
                - --in-place

-   repo: https://github.com/psf/black
    rev: "23.3.0"
    hooks:
        -   id: black
            alias: reformat
            types: [python]

-   repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.0.292
    hooks:
        - id: ruff
          alias: lint
          types: [python]

-   repo: https://github.com/executablebooks/mdformat
    rev: "0.7.16"
    hooks:
    -   id: mdformat
        description: Auto-format Markdown files.
        args: [--wrap=88, --number]
        additional_dependencies:
        - mdformat-gfm==0.3.5
        - mdformat-tables==0.4.1
        - mdformat-black==0.1.1
        - linkify-it-py==2.0.2

-   repo: https://github.com/adrienverge/yamllint.git
    rev: v1.29.0
    hooks:
        -   id: yamllint
            description: This hook runs yamllint.
            entry: yamllint
            language: python
            types: [file, yaml]
            args:
                - -c
                - yamllint-config.yml
                - --no-warnings

-   repo: https://github.com/pre-commit/mirrors-clang-format
    rev: v16.0.6
    hooks:
        -   id: clang-format
            types_or: [c++, c]

-   repo: meta
    hooks:
    -   id: identity
        types: [python]

Answer 1

According to the documentation, a pre-commit hook "must exit nonzero on failure or modify files." If a hook unexpectedly passes, it should relate to one of those two things.

Using black as an example, it exits with 0 unless the --check flag is passed, even when it reformats files. So pre-commit must be relying on detecting that files had changed to fail the hook.

How does pre-commit know whether files changed? It uses git diff before and after and compares the stdout of the two runs. If they are different, a change must have been made.

Adding the same git diff command to my build script, I discovered that it was failing on the CI server and printing an error message. Presumably pre-commit was also getting the error, but the same message before and after, resulting in a "pass" for not detecting a change. It doesn't look like pre-commit checks the exit code of git diff.

In my case, git diff was failing because of a combination of Bamboo's linked repositories feature and a Docker container not mounting Bamboo's cache directory. The error looked like:

error: object directory /opt/bamboo-home/alpha/xml-data/build-dir/_git-repositories-cache/fdae2edae02caea8ea6d761ec2587a93d46f063d/.git/objects does not exist; check .git/objects/info/alternates.
error: unable to find e6ce93b6af518f7e3efd767aa7124aed5d036734
fatal: unable to read e6ce93b6af518f7e3efd767aa7124aed5d036734