Is there a right way to implement a continuous improvement (AKA software hardening) process?

Question

Each release it seems that our customers find a few old issues with our software. It makes it look like every release has multiple bugs, when in reality our new code is generally solid.

We have tried to implement some additional testing where we have testers do several hours of monthly regression testing on a single app each month in an effort to stay ahead of small issues. We refer to this process as our Software Hardening process, but it does not seem like we are catching enough of the bugs and it feels like a very backburner process since there is always new code to write.

Is there a trick to this kind of testing? Do I need to target one specific feature at a time?

Answer 1

When you develop your testing procedures, you may want to implement these kind of tests:

• unit testing (testing invididual components of your project to test their functionality), these tests are important because they allow you to pinpoint where in the software the error may come from. Basically in these tests you will test a single functionality and use mock objects to simulate the behavior, return value of other objects/entities.

• regression testing, which you mentioned

• characterization testing, one example could be running automatically the program on automatically generated input (simulating the user input), storing the results and compare the results of every version against these results.

At the beginning this will be very heavy to put in place, but with more releases and more bugs fixes being added to the automated non-regression tests, you should be starting to save time.

It is very important that you do not fall in the trap of designing huge numbers of dumb tests. Testing should make your life easier, if you spend too much time understanding where the tests have broken you should redesign the tests such as they give you better messages/understanding of the problem so you can locate the issue quickly.

Depending of your environment, these tests can be linked to the development process.

In my environment, we use SVN for versioning, a bot runs the tests against every revision and returns the failing tests and messages with the name of the revision which broke it and the contributor (his login).

EDIT:

In my environment, we use a combination of C++ and C# to deliver analytics used in Finance, the code was C++ and is quite old while we are trying to migrate the interfaces toward C# and keep the core of the analytics in C++ (mainly because of speed requirements)

Most of the C++ tests are hand-written unit tests and regression tests.

On the C# side we are using NUnit for unit testing. We have a couple of general tests.

We have a 0 warnings policy, we explicitely forbid people to commit code which is generating warnings unless they can justify why it is useful to bypass the warning for this part of the code. We have as well conventions about exception safety, the use of design patterns and many other aspects.

Setting explicitely conventions and best practices is another way to improve the quality of your code.

Answer 2

As soon as your coding ends, first you should go for the unit testing. THere you will get some bugs which should be fixed and you should perform another round of unit testing to find if new bugs came or not. After you finish Unit testing you should go for functional testing.

YOu mentioned here that your tester are performing regression testing on a monthly basis and still there are old bugs coming out. So it is better to sit with the tester and review the test cases as i feel that they need to be updated regularly. Also during review put stress on which module or functionality the bugs are coming. Stress on those areas and add more test cases in those areas and add those in your rgression test cases so that once new build comes those test cases should be run.

YOu can try one more thing if your project is a long term one then you should talk with the tester to automate the regression test cases. It will help you to run the test cases at off time like night and in the next day you will get the results. Also the regression test cases should be updated as the major problem comes when regression test cases are not updated regularly and by running old regression test cases and new progression test cases you are missing few modules that are not tested.

Answer 3

Is there a trick to this kind of testing?

You said, "we have testers do several hours of monthly regression testing on a single app each month in an effort to stay ahead of small issues."

I guess that by "regression testing" you mean "manually exercising old features".

You ought to decide whether you're looking for old bugs which have never been found before (which means, running tests which you've never run before); or, whether you're repeating previously-run tests to verify that previously-tested functionality is unchanged. These are two opposite things.

"Regression testing" implies to me that you're doing the latter.

If the problem is that "customers find a few old issues with our software" then either your customers are running tests which you've never run before (in which case, to find these problems you need to run new tests of old software), or they're finding bugs which you have previous tested and found, but which you apparently never fixed after you found them.

Do I need to target one specific feature at a time?

What are you trying to do, exactly:

• Find bugs before customers find them?
• Convince customers that there's little wrong with the new development?
• Spend as little time as possible on testing?

Very general advice is that bugs live in families: so when you find a bug, look for its parents and siblings and cousins, for example:

• You might have this exact same bug in other modules
• This module might be buggier than other modules (written by somone on an off day, perhaps), so look for every other kind of bug in this module
• Perhaps this is one of a class of problems (performance problems, or low-memory problems) which suggests a whole area (or whole type of requirement) which needs better test coverage

Other advice is that it's to do with managing customer expectations: you said, "It makes it look like every release has multiple bugs, when in reality our new code is generally solid" as if the real problem is the mistaken perception that the bug is newly-written.

it feels like a very backburner process since there is always new code to write

Software develoment doesn't happen in the background, on a burner: either someone is working on it, or they're not. Management must to decide whether to assign anyone to this task (i.e. look for existing previously-unfound bugs, or fix-previously-found-but-not-yet-reported bugs), or whether they prefer to concentrate on new development and let the customers do the bug-detecting.

---

Edit: It's worth mentioning that testing isn't the only way to find bugs. There's also:

• Informal design reviews (35%)
• Formal design inspections (55%)
• Informal code reviews (25%)
• Formal code inspections (60%)
• Personal desk checking of code (40%)
• Unit test (30%)
• Component test (30%)
• Integration test (35%)
• Regression test (25%)
• System test (40%)
• Low volume beta test (<10 sites) (35%)
• High-volume beta test (>1000 sites) (70%)

The percentage which I put next to each is a measure of the defect-removal rate for each technique (taken from page 243 of McConnel's Software Estimation book). The two most effective techniques seem to be formal code inspection, and high-volume beta tests.

So it might be a good idea to introduce formal code reviews: that might be better at detecting defects than black-box testing is.

Answer 4

Going back and implementing a testing strategy for (all) existing stuff is a pain. It's long, it's difficult, and no one will want to do it. However, I strongly recommend that as a new bug comes in, a test be developed around that bug. If you don't get a bug report on it, then either is (a) works or (b) the user doesn't care that it doesn't work. Either way, a test is a waste of your time.

As soon as its identified, write a test that goes red. Right now. Then fix the bug. Confirm that it is fixed. Confirm that the test is now green. Repeat as new bugs come in.

Answer 5

There is a lot of talk here about unit testing and I couldn't agree more. I hope that Josh understands that unit testing is a mechanized process. I disagree with PJ in that unit tests should be written before coding the app and not after. This is called TDD or Test Driven Development.

Some people write unit tests that exercise the middle tier code but neglect testing the GUI code. That is imprudent. You should write unit tests for all tiers in your application.

Since unit tests are also code, there is the question of QA for your test suite. Is the code coverage good? Are there false positives/negatives errors in the unit tests? Are you testing for the right things? How do you assure the quality of your quality assurance process? Basically, the answer to that comes down to peer review and cultural values. Everyone on the team has to be committed to good testing hygiene.

The earlier a bug is introduced into your system, the longer it stays in the system, the harder and more costly it is to remove it. That is why you should look into what is known as continuous integration. When set up correctly, continuous integration means that the project gets compiled and run with the full suite of unit tests shortly after you check in your changes for the day.

If the build or unit tests fail, then the offending coder and the build master gets a notification. They work with the team lead to determine what the most appropriate course correction should be. Sometimes it is just as simple as fix the problem and check the fix in. A build master and team lead needs to get involved to identify any overarching patterns that require additional intervention. For example, a family crisis can cause a developer's coding quality to bottom out. Without CI and some managerial oversight, it might take six months of bugs before you realize what is going on and take corrective action.

You didn't mention what your development environment is. If yours were a J2EE shop, then I would suggest that you look into the following.

• CruiseControl for continuous integration
• Subversion for the source code versioning control because it integrates well with CruiseControl
• Spring because DI makes it easier to mechanize the unit testing for continuous integration purposes
• JUnit for unit testing the middle tier
• HttpUnit for unit testing the GUI
• Apache JMeter for stress testing

Answer 6

Sorry to say that but maybe you're just not testing enough, or too late, or both.