Grading Exit Criteria

For the sake of this article, let us assume that we have 5 different grades of defect that start with insignificant going on to, minor, significant, major and ending in critical. When we come to exit one phase of a project and enter the next, we need to recognise that criteria will be set by the test manager, which need to be achieved. It is normal that the exit criteria from one phase, becomes the entry criteria for the next phase. For instance, a Test Manager for “Integration in the Large” testing will set his entry criteria, which by default becomes the exit criteria for the phase before. This remains true up to the point of release to live.


What I would like to draw attention to, are two key points. The test manager that says I am not going to have any defects remaining open at the point in time I go live, is living in “cloud cuckoo land”. This is the test manager that wants to be able to test forever and release perfect code. This does not happen. The reality of business is that certain levels of risk become acceptable in order to deliver the project to live and meet the requirements of the business and deliver the business benefits that were used to justify the project in the first instance.


So the next question becomes one of levels of risk and what is likely to be acceptable. This will depend on the nature of the project. One that involves safety critical applications is likely to have a far higher level requirement for quality, than one which is being used by 2 people in a small business. One which is exposed to the public or clients is also going to need to be of a significantly higher level of quality than one which is used internally only. The skill of the test manager is to assess the application, its’ use and define the level of defects that are going to be tolerable. As previously defined in http://tcl-india.blogspot.com/2008/06/entry-and-exit-criteria.html the Project Manager should be bought into these levels in order to ensure support later in the project.


Now we need to look at the defects that are going to be acceptable. A common misconception is to identify that the level of critical defects is 0, be that priority or severity, but that there is no restraint on the level of insignificant defects. This could not be further from the truth. Whilst 1 insignificant defect is not going to stop the release, there comes a volume of insignificant defects that makes the release of the application unacceptable.


I would suggest that we concentrate initially on the severity of defects. We need to understand proportionately that a volume of insignificant is equal to 1 minor, a volume of minor is equal to 1 significant and so on. This will change dependent on the application use, so here are some suggestions for different application types.


Safety Critical
Proportions: 1 Critical = 2 Severe : 1 Severe = 3 Significant : 1 Significant = 5 Minor : 1 Minor = 10 Insignificant
Final Exit Criteria: 0 Critical : 0 Severe : 0 Significant : 5 Minor : 10 Insignificant


Public/Client Facing
Proportions: 1 Critical = 3 Severe : 1 Severe = 5 Significant : 1 Significant = 10 Minor : 1 Minor = 20 Insignificant
Final Exit Criteria: 0 Critical : 0 Severe : 3 Significant : 10 Minor : 20 Insignificant


Internal Consumption (20 + Users)
Proportions: 1 Critical = 4 Severe : 1 Severe = 7 Significant : 1 Significant = 15 Minor : 1 Minor = 50 Insignificant
Final Exit Criteria: 0 Critical : 1 Severe : 5 Significant : 10 Minor : 40 Insignificant


Internal Consumption (0 to 20 Users)
Proportions: 1 Critical = 5 Severe : 1 Severe = 10 Significant : 1 Significant = 20 Minor : 1 Minor = 100 Insignificant
Final Exit Criteria: 0 Critical : 2 Severe : 5 Significant : 10 Minor : 50 Insignificant

Please bear in mind that these are indicative and that the best solution is for the test manager and other members of the project team to determine the levels between them.