Regression Testing

The SIGIST “Special Interest Group In Software Testing”, have a glossary of terms located at http://www.testingstandards.co.uk/living_glossary.htm

Here we find regression testing defined as: “Retesting of a previously tested program following modification to ensure that faults have not been introduced or uncovered as a result of the changes made”

Regression Testing is a key component of any project and should be the final test performed prior to release to the production platform.

So what comprises a regression test?

The best way of describing this, is that an application which has been previously released, will be in a known state. Tests will have been previously written and executed against this and a selection of them will have been extracted to form the regression test pack. The tests would look to exercise as a minimum the core functionality of the application, in a positive manner. In this way, whenever a change is made to the application in the future, the regression pack can be run to ensure that faults have not been introduced or uncovered in functionality that was previously deemed to be working correctly. The regression pack should be updated as each new version of the application is released, building in new functionality to the suite of regression tests and maintaining it in line with the applications capabilities.

Some debate occurs as to what a regression test should comprise and it is fair to say that the larger the regression suite, the more confidence will exist in the new version of the application, but there is a balance to be achieved. The more tests, the longer they take to execute and the higher the cost to change. Pragmatism must be applied and rather than testing every permutation, coverage should ensure that the functionality of the application has not been impacted by change. Tests which apply different data to the same function, access the same function from a different point or negatively test the function (trying to break it), do not need to be part of the regression suite.

Experience shows that there is a distinct need for regression testing. Problems with Configuration Management or code control are often identified during regression, where previously fixed problems are re-introduced. Code which is common to different modules within the application is also a primary area of difficulty, as a change for the benefit of one module may adversely impact another.

It is key from a regression point of view, that no test is performed that has not been run successfully on a previous occasion. In a similar manner, no code should be regression tested which has not been previously tested.

A recent experience we had with webMethods regression testing was where, due to the nature of webMethods, several projects were being released together and the webMethods code was being converged and tested for the first time during the regression test. Because the convergence was seen as something over and above the need of the projects, it was passed onto a different team to build and a different test team to regression test. Not only was this an inaccurate use of regression testing, but the testers with the most pertinent knowledge around what was expected, were not involved.

As testers we must protect the regression testing activity. Ensure that it is full, up-to-date and complete. Failure to run regression properly increases the risk of faults in production.