Best Practices for how to include unit tests in a library

Question

I'm working on a general code library for ActionScript 3.0 called as3lib which includes several extensions to the core API and some useful functions. I've written several unit tests (using FlexUnit) to make sure everything is working correctly.

What is the best way to organize these tests in the library? Currently, I have all my code in src/ and my tests in test/ but I've set up a secondary Flex project to run the unit tests. I am also manually adding and removing the test files from the library when I want to run the tests.

What I'm doing doesn't seem right. Is there a better way? Preferably one where the compiled library doesn't include the test files but I don't need two separate projects to test them.

Answer 1

The way we've done these things at my company is that we actually include both under the source directory, and then we have two application mxml files which we use. One is the testing suite, which includes all of the appropriate links to unit-test classes, the other is the main application. We have also have two package structures inside of the src folder: one package structure com.., and another tests.com... Make sure that ALL source code for unit tests are ALWAYS in the tests package -- this way you can use only one SVN ignore, and you can also make sure that your tests do not create dependent and hard-coded relationships with other projects.

There are two ways which we use to ensure that the test.com source files are not included. The automated build system only references the main application, and since that only imports from com., mxmlc.exe will only include the files for the main application. When building locally, inside of eclipse, you can control how things build by clicking on the little arrow next to Debug and scroll to "Organize favorites." When you click "Add", you should be able to select all root level .mxml files which reference the Application class. Be sure to add the base application and the new unit test application file. When you then click "OK", the arrow now allows you to debug either as the main application, or your unit test framework.

As an aside, we also use FlexUnit as our testing framework. I like it.

Answer 2

I've done it similar to the way you're describing in the past, but it seems like the sort of thing where SpringAS could come in pretty handy for dynamically adding and removing them from the configuration. Have you tried looking into that?

Answer 3

We have separate src and tests directories at the top level in our libraries. Our applications are very thin wrappers around library projects, so they don't need any tests. We also have a FlexUnit application project, to run the tests from FlexBuilder.

We use maven for our main build system, and the Sonatype Flex plugin runs all our unit tests during the build, even on our linux-based Continuum server. Maven defaults to looking for tests in a 'tests' directory, which was a good justification for picking that location.

Answer 4

I'm going to go against the flow here and suggest setting up a different project altogether for your tests. I believe that generally it doesn't really matter where you put the tests, as long as it's consistent and somewhat manageable. However, to me there are three compelling reasons for having a separate project for the tests:

1. Separation of concerns. First, your library has one purpose, the tests have another. While the tests needs the library to function, the library has no real use for the tests. Note that I'm not saying tests are useless, far from it. The tests are there to verify the health of the library but in a production environment the tests serve no purpose.

2. Less bloat, smaller files. Tests aren't always trivial. But even if they all were, they'd still be using disk space. Since the tests aren't used in a production environment anyway, this is pointless. Also, separating tests into a new project makes the file structure a lot cleaner.

3. CI environments are generally cleaner to set up when the tests aren't around.

While it's certainly possibly to at least solve the second problem with compiler directives, it's unnecessary work when it's a lot easier to just separate the two. Testing a library or application that may need you to use the same namespace (internal classes anyone?) is also not a problem, since your tests project may mirror the namespaces. Obviously this makes it necessary to not have name collisions in the namespaces, but that's trivial.

In terms of Flash Builder support it's great to separate things into two projects. All you need to do to create a new test is right click whatever class you whish to test, ask to create a new test and make sure you choose your tests project instead of the current in the dialog that pops up. That's really the main reason why me and my team members had a hard time justifying writing tests when we got into TDD, too much overhead just to get started. With the current state of the IDE, it's ridiculously simple and useful.

As with any technique however, there are caveats. For one thing, it's not obvious that the tests are in a different project unless it's documented. Having the tests in the same project effectively sorts that problem. On the other hand, this is easily solved by configuring maven or other dependency management tools you may have in your environment. Another issue is that if you do have a package structure in your tests project that mirrors that of the library or application, there's some maintenance overhead in having those structures synchronized. While not a huge issue and quite easily solved using scripting, it's still worth mentioning.

Anyhow, that's how I do it.