Which version control programs can enforce running & passing of tests before integration of changes?

Question

At my work we currently use Aegis version control/SCM. The way we have it configured, we have a bunch of tests, and it forces the following things to be true before a change can be integrated:

• The full set of tests must have been run.
• All tests must have passed.

With test-driven development (TDD) these seem like sensible requirements. But I haven't heard of any way you can do this with any other version control systems. (We're not currently planning to switch, but I would like to know how to do it in the future without using Aegis.)

I would be interested in any VCS (distributed or not) that can do this, I'm also interested in any plugins/extensions to existing VCS that allow this. Preferably open source software.

ETA: OK, it seems like the usual thing to do is have VCS + continuous integration software, and running the tests is automated as part of the build, instead of as a separate step. If I understand correctly, that still lets you commit code that doesn't pass the tests, just you get notified about it -- is that right? Is there anything that would stop you from being able to integrate/commit it at all?

Answer 1

IMO you're much better off using a continuous integration system like CruiseControl or Hudson if you want to enforce that your tests pass, and make the build rather than the check-in dependent on the tests results. The tools are straightforward to set up, and you get the advantages of built-in notification of the results (via email, RSS or browser plugins) and test results reporting via a Web page.

Regarding the update to the question, you're right - VCS + CI lets you commit code that doesn't pass the tests; with most CI setups, you just won't get a final build of your product unless all the tests pass. If you really want to stop anyone from even committing unless all the tests pass you will have to use hooks in the VCS as others have suggested. However, this looks to me to be hard to deal with - either developers would have to run all of the tests every time they made a checkin, including tests that aren't relevant to the checkin they are making, or you'd have to make some very granular VCS hooks that only run the tests that are relevant to a given checkin. In my experience, it's much more efficient to rely on developers to run the relevant tests locally, and have the CI system pick up on the occasional mistakes.

Answer 2

You can also use pre-commit hooks in Perforce. And, if you're a .NET shop, Visual Studio can be configured to require "gated" checkins.

Answer 3

Most CI implementations have a mechanism to reject check-ins that don't meet all the criteria (most notably pass all the tests). They're called by different names. VCS should do what they do best.. version source code.

• TeamCity - Pre-tested commit
• TFS - Gated check-ins

Answer 4

VSTS with custom Work Items, right? I don't see anything wrong with using this. Built in reporting. The choice to automate. Why not?

Answer 5

We use git and buildbot to do something similar, though not quite the same. We give each developer their own Git repository, and have the buildbot set to build any time pushes to one of those repositories. Then there is someone who acts as the integrator, who can check the buildbot status, review changes, and merge their changes or tell them to fix something as appropriate.

There are plenty of variations of this workflow that you could do with Git. If you didn't want to have someone be the integrator manually, you could probably set the buildbot up to run a script on success, which would automatically merge that person's change into the master repository (though it would have to deal with cases in which automatic merge didn't work, and it would have to test the merge results as well since even code that merges cleanly can sometimes introduce other problems).

Answer 6

With subversion and git you can add pre-commit hooks to do this.

It sounds like you need to look at Continuous Intergration (or a variant of).

Think Git has a hook on apply patch too.

Answer 7

Subversion and git both support this via pre-commit hooks.

Visual Studio Team System supports this natively via checkin policies.

I believe that Rational ClearCase also supports it, though I've never seen that demonstrated so I can't say for certain.

Answer 8

I believe continuous integration software such as team city allow you to do pre-commit build and test. I don't know of any vcs that provides it directly...there may be some like the one you use but I'm not familiar with them.

Answer 9

What I do here is following a branch per task pattern which lets you test the code already submmitted to version control but still keeping the mainline pristine. More on this pattern here.

You can find more information about integration strategies here and also comments about Mark Shuttleworth on version control here.