How to use git bisect: Using unit tests for determining the errorneous commit?

Question

I'm new to git and just discovered "git bisect". How is it generally used when you can check for the error using a newly written unit test (that wasn't there beforehand)?

Say we have a repository containing unit tests, but - as so often - they weren't complete. In some version we discover that a feature isn't working, but we know it was in some version before. Unfortunately that feature wasn't checked by a unit test. What I'd like to do now is:

1. Write the test case (that fails at first of course)
2. Look for the commit introducing the error using git bisect using this test case
3. Correcting the error and committing the fix (and the test case)

This use case has one problem. As git bisect checks out old versions, the just inserted test case won't be there, so I can't use it to check for the error. I can try to not commit the test, but to keep the it as local change. But imagining that the tests file was changed in some version inbetween, I'm likely to run into merge conflicts.

So what to do? How is git bisect generally used?

Answer 1

I usually use bisect to identify when a bug was introduced, so that I can look at a single commit for the cause of an error. This is useful when I know that something used to work and now doesn't. I first pick or find an old commit that worked, mark it with git bisect good, mark the head with git bisect bad, then work through the bisect until git tells me the commit that introduced the problem. Sometimes I can write the unit test before beginning this process, sometimes I can't write it until I've seen the code that introduced the bug.

So, let's say I have the unit test, or some other bit of code or script that I need to use at each point when going through the git bisect process. The stash is useful for holding that as I go. So,

git stash save blah
git bisect bad (and now I'm on a new commit)
git stash pop
mvn clean test

git stash save blah
git bisect good
git stash pop
mvn clean test

and so forth.

Answer 2

First, before trying this, make sure that your git status is clean - I'm suggesting using a script that would invoke git reset --hard, so any uncommitted changes would be lost.

One way to do this is to first copy the source code for your most recent tests to some location that was never tracked in the history of the repository (or somewhere outside the repository). Then write a script that:

1. Copies the most recent tests into place in your source tree
2. Runs the tests, saving the return code.
3. Does a git reset --hard to wipe out the changes from step 1
4. Exits with the return code saved from 2.

Then, after marking two commits as good and bad to give git bisect somewhere to start from, you can use git bisect run YOUR-SCRIPT to find the first commit where the most recent unit tests would have failed.

This method is essentially what's suggested in the documentation for git bisect run, where it says:

You may often find that during a bisect session you want to have temporary modifications (e.g. s/#define DEBUG 0/#define DEBUG 1/ in a header file, or "revision that does not have this commit needs this patch applied to work around another problem this bisection is not interested in") applied to the revision being tested.

To cope with such a situation, after the inner git bisect finds the next revision to test, the script can apply the patch before compiling, run the real test, and afterwards decide if the revision (possibly with the needed patch) passed the test and then rewind the tree to the pristine state. Finally the script should exit with the status of the real test to let the "git bisect run" command loop determine the eventual outcome of the bisect session.

Answer 3

You haven't mentioned which technology you're using, so I'll use Ruby as an example, which is what I'm most familiar with. The same principle should apply to other technologies as well, though.

1. Place your up to date test file outside of your project directory. Here I'll assume it's on /tmp/my_test.rb

2. Inside your repository, create a small shell script, say, test_runner.sh:

   #!/usr/bin/env sh    
   cp -f /tmp/my_test.rb test/my_test.rb # This will potentially overwrite some old test file
   ruby test/my_test.rb # Change for whatever test runner you use
   test_result=$?
   git checkout test/my_test.rb # Undoes any changes that might have been made
   exit $test_result
   

3. Mark commits as good/bad, as usually is done with bisect.

4. Use git bisect run test_runner.sh and go grab a cup of coffee while git finds your bug for you.