git ls-files - undocumented syntax that queries files based on their .gitattributes - git ls-files ':(attr:filter=lfs)'

Question

Is comparing git lfs ls-files with git ls-files ':(attr:filter=lfs)' a reliable way to detect lfs files that are not managed by lfs? uses git ls-files syntax that queries files based on their .gitattributes (in that particular case on filter=lfs)

git ls-files ':(attr:filter=lfs)'

The problem is that though it actually works without issues, it is not something that is explained in documentation - https://git-scm.com/docs/git-ls-files

So, is it something that I miss in docs, or is it some undocumented feature?

Answer 1

It's documented at gitglossary:

After attr: comes a space separated list of "attribute requirements", all of which must be met in order for the path to be considered a match; this is in addition to the usual non-magic pathspec pattern matching.

Answer 2

You can find additional information and example in a fix for "git ls-files '(attr:X)D/'"^(man)": when it triggered the common prefix optimization codepath, it failed to read from "D/.gitattributes", and that has been corrected with Git 2.42 (Q3 2023).

See commit f4a8fde (08 Jul 2023), and commit 7e360bc (07 Jul 2023) by Junio C Hamano (gitster).
^{(Merged by Junio C Hamano -- gitster -- in commit 13ed10e, 17 Jul 2023)}

dir: match "attr" pathspec magic with correct paths

^{Reported-by: Matthew Hughes}

The match_pathspec_item() function takes "prefix" value, allowing a caller to chop off the common leading prefix of pathspec pattern strings from the path and only use the remainder of the path to match the pathspec patterns (after chopping the same leading prefix of them, of course).

This "common leading prefix" optimization has two main features:

• discard the entries in the in-core index that are outside of the common leading prefix; if you are doing "ls-files one/a one/b", we know all matches must be from "one/", so first the code discards all entries outside the "one/" directory from the in-core index.
  This allows us to work on a smaller dataset.

• allow skipping the comparison of the leading bytes when matching pathspec with path.
  When "ls-files" finds the path "one/a/1" in the in-core index given "one/a" and "one/b" as the pathspec, knowing that common leading prefix "one/" was found lets the pathspec matchinery not to bother comparing "one/" part, and allows it to feed "a/1" down, as long as the pathspec element "one/a" gets corresponding adjustment to "a".

When the "attr" pathspec magic is in effect, however, the current code breaks down.

The attributes, other than the ones that are built-in and the ones that come from the $GIT_DIR/info/attributes file and the top-level .gitattributes file, are lazily read from the filesystem on-demand, as we encounter each path and ask if it matches the pathspec.

For example, if you say "git ls-files (attr:label)sub/"^(man)" in a repository with a file "sub/file" that is given the 'label' attribute in "sub/.gitattributes":

• The common prefix optimization finds that "sub/" is the common prefix and prunes the in-core index so that it has only entries inside that directory.
  This is desirable.

• The code then walks the in-core index, finds "sub/file", and eventually asks do_match_pathspec() if it matches the given pathspec.

• do_match_pathspec() calls match_pathspec_item() after stripping the common prefix "sub/" from the path, giving it "file", plus the length of the common prefix (4-bytes), so that the pathspec element "(attr:label)sub/" can be treated as if it were "(attr:label)".

The last one is what breaks the match in the current code, as the pathspec subsystem ends up asking the attribute subsystem to find the attribute attached to the path "file".

We need to ask about the attributes on "sub/file" when calling match_pathspec_attrs(); this can be done by looking at "prefix" bytes before the beginning of "name", which is the same trick already used by another piece of the code in the same match_pathspec_item() function.

Unfortunately this was not discovered so far because the code works with slightly different arguments, e.g.

$ git ls-files "(attr:label)sub" 
$ git ls-files "(attr:label)sub/" "no/such/dir/"  

would have reported "sub/file" as a path with the 'label' attribute just fine, because neither would trigger the common prefix optimization.

Answer 3

Actually, it is documented, but not where you would expect, or perhaps even look. The place where this is documented is in the gitglossary, under the definition for pathspec:

A pathspec that begins with a colon : has special meaning ...

After attr: comes a space separated list of "attribute requirements", all of which must be met in order for the path to be considered a match; ...