Contractually Demanding Testing from Developers

It is an experience that testers all over the world will instantly understand. The contract with the development agency makes no demand for them to perform unit and integration (in the small) testing.

How does this manifest itself? Firstly, the Test Manager will specify within the Test Strategy or Test Plans a series of entry criteria, from development into testing. This should comprise test plans, a full suite of test scripts and a series of defects that have been raised. As the development nears completion, the Test Manager will begin to ask for the evidence of the testing and look to meet the entry criteria for Integration (in the large) or formal testing.

At this point the response can be that the unit and integration (in the small) testing has not been contractually specified. The supplier often takes one of two stances, stating that this cannot be achieved, or insisting that the testing can be performed, but at additional cost and increased timescale. This immediately places the Project Manager in an awkward position: they will be unable to agree to the additional time and will not want to incur the increased costs. The normal action at this point is for the Project Manager to accept that a mistake has been made, log a risk and accept that the code will be delivered into testing without development testing having occurred (or proven to have occurred).

The quality of code is now likely to be of a lower standard and the volume of defects expected in the formal testing higher. This may result in additional cycles of testing and increases in costs from this perspective (part of the risks that the Project Manager will have taken on board). As the volume of defects found during the black box testing grows, the demand on the development agency increases and care should be taken to ensure that defect fix times are monitored. These are likely to slow as the volume of defects increases and this should be reported to the Project Manager as soon as it is seen. A good contract will have service level agreements(SLAs) in place, to ensure that defect turn-around is at a specified rate depending on severity and priority.

If the development agency has failed to perform their own testing, the demand is placed on the testing team or test supplier and the estimates that they will be working to should be re-visited. If testing is working on a fixed price contract, failure to meet entry criteria should constitute a change request and the ability to re-price.

It may be possible to insert some of the test team with the developers, allowing them to test unit and integration (in the small) in parallel to the developers. Alternatively, testers with a technical bent can be asked to look at the code, perhaps performing dry runs. If the testing is being performed but the deliverables are not being made available, then it may be possible to witness some of the testing that is occurring. It must be said that these last two points are more geared to damage limitation, rather than ensuring that the right level of quality has been built into the product.

We need to avoid this in the first place, so ensure that your organisation's Supply Chain has a standard wording that is inserted into all contracts with development agencies protecting from such events. For smaller organisations that do not have a Supply Chain function, bring the Test Manager on board for up-front consultancy; they should look to ensure that this does not happen. A further measure is for the development agency to be asked to sign off the test strategy, which should detail the entry criteria into the formal (black box) testing phase, giving the buyer some level of recourse. Lastly, as a Test Manager, don't leave it until the end of development to find you have a problem - identify it as early as possible and get written agreement that the appropriate testing will take place and that the test assets produced will form part of a deliverable.