Software Testing - Defect Turnaround

Within the test execution phase of a project, the testers will be raising defects against the application/s under test. These defects will more often than not refer to the development of the application and will require developers to rectify the defects that have been identified. The speed at which the defects are rectified is referred to as Defect Turnaround.

A regular project issue is around the amount of time it takes to resolve defects once they have been identified. The problem is often caused by a reduction in the size of the development team once test execution begins. The majority of development effort is thought to have been completed in the creation of the item. In an ideal world this would be true. If the quality of the produced application is high and in accordance with requirements, then the need for a continuation of the same development effort is not necessary.

Too often, the quality of the application supplied is below standard or fails to meet requirement. This begins to cause issues when the size of the development team is reduced. Now the testers are raising defects at a faster rate than they can be fixed. This can be particularly true when dealing with small development agencies which are spreading their resources thinly across multiple clients and cannot afford to retain staff on the project past the development phase.

The problem manifests as the volume of outstanding defects increases and the volume of fixed defects fails to match those raised. This gives two problems. The first is that the testers are likely to have areas of code that they have not been able to reach as severe defects are not being rectified and testing paths through scripts cannot be followed. This begins to endanger the project as the execution will have been estimated on the basis of being able to pass through the scripts with only minor issues. The project timescales are now immediately under threat. The second problem is that by plotting the volume of defects against fixes, it is now possible to calculate the volume of defects that will remain open and unfixed at the point of go live. This places the project at risk of delivering a production or application/s which are of sub-standard quality and will therefore reflect badly on IT and the project team. The volume of defects going live is also indicative of the likelihood of calls through to support and if of sufficient volume may cause support to refuse to allow the project to go live, or be taken into support. Projects which are not taken into support require the continuation of the project past the point of go-live to ensure that support is provided and that outstanding issues are rectified before being taken into formal support. This places more stress onto the development agency as their resources are tied up for longer.

So firstly let’s look at how we avoid this. Try to understand the reputation of the development agency. If they have a good reputation and generate high quality code, there is less likely to be an issue. If they are an unknown development agency or have a reputation for poor quality, look closely at the project plans and the level of resource that they are allocating during the test execution. I would suggest a 2 to 1 ratio of tester to developer at this point as a minimum and potentially higher. Make recommendations to the Project Manager and explain the dangers. Bear in mind that this will potentially have a cost impact and that resistance is likely. Opt for a time and materials effort basis at this point and securing commitment to certain resource volumes. If the quality is good and the developers are not needed then there is no penalty to pay for but the protection is there.

A further preventative measure, may be to discuss this as a concern with the development manager and raise their awareness of the potential for a problem and obtain written assurances that they will work to protect against this.

Monitoring for this occurrence is incredibly important. Reporting on defects on a daily basis during test execution is imperative and knowing how many are raised and classed as fixed each day critical. (Notice that I say fixed and not closed – we are expecting development to state a defect is fixed and ready for re-test, but we may delay the re-test waiting for the correct point in time at the end of a test cycle.) Be careful not to hit the panic button, but look at the amount of time that is allocated to test execution and understand what timeline you need to measure over. On a project lasting a couple of weeks, I would be looking at a trend over two or three days, but if the execution only lasts for a week, then as soon as I saw rectified defects failing to match raised defects I would begin ringing warning bells with the Project Manager.

Lastly, consider the severity and priority of the defects and the exit criteria for the test execution. If low priority defects are accumulating but are within tolerance of the exit criteria, it is not a problem and should not be escalated. The opposite also applies and is of high concern, that if high severity or high priority defects are not being fixed quickly, it may be pertinent to ring the alarm bell sooner. I would suggest that defects of high priority or severity need to be turned around in 10% of the execution time.

To summarise, make sure you have enough developers on board during test execution. Monitor defects raised against defects fixed and alert Project Management if trends develop showing an increased in the remaining open defects.