Middleware Regression Testing

I have been working for a client on an interesting problem recently, which I thought I would share on the blog. The problem has been around the regression testing of an item which is intrinsic by nature, to many other projects. The item is classed as middleware, which provides a translation between two different systems, but is being relied upon increasingly, with in excess of 50 systems now interfacing. Most projects now require the middleware to be updated and the volume of change is placing the regression testing under increasing pressure as more releases occur.

The problem is amplified as the projects are working individually and the middleware changes are independent as a result. A group of projects are collated to form a release and once normal testing is completed the middleware changes are converged to form a single drop of code, which is then passed into the regression testing phase.

The first mistake has been made. The projects are not taking accountability for testing the converged code and in some instances have even been disbanded prior to this testing occurring. At this point the release has lost the testers with the understanding of the end to end function and more pressure is applied to the regression test team, to trace the scripts from the project, interpret them and generate new scripts. It is important that The testing of the converged code is now taking up large amounts of the already pressured regression testing window. This needs to be pushed back to the projects and the convergence owned by them.

By removing the testing of the converged code from the regression process, more time is now allowed and full regression can be performed, but this does not provide a future proof solution. The same window is now being fully utilised and as the volume of utilisation increases over time, the regression pack will increase and the window will become too small again.

This is an ideal opportunity to make the most of an automated solution. If we look to industry standards, automation makes a test 7 times faster to execute but 7 times longer to write. This means that the testing duration, once automated can be reduced to 1.5 days. Now we have a regression test that is only performing regression; that can be executed swiftly and provides scope for future growth.

The testing environments were another level of complexity. Whilst the project completed it’s testing with all the required components/systems/applications, the regression team, within their two week window had to acquire their own equivalent. Again this was another drag on time, but more importantly the regression was sometimes being performed on an environment that was incomplete. The result has been to make the regression middleware environment available to the projects for the convergence testing, but to utilise the projects interfacing systems. In order to future proof, the regression testing will be moved onto a pre-production environment.

There were other aspects to this problem, but those listed were particularly pertinent. The following feedback was received from two sources as a result of the work and presentation of the findings:

“I was delighted that your slides prompted so much discussion, even if it made your presentation difficult, and that by all accounts we have got the main individuals on board.”

“I was very appreciative of the clear way you presented where we are and your proposals for going forward, together with a clear capture of issues that are spoken about in lots of quarters but never pulled together into a single picture. Great.”

In summary – regression testing should encompass the entire system.
Projects should retain responsibility past the point of release to production.
Automation is ideally suited to regression testing.
Testing should always occur on as complete an environment as possible.