A Thought on Test Effort Estimation

Introduction

Test estimation always starts with the following common questions.

1. What are the requirements to test software: Resource? Infrastructure? Time? Proper planning and Estimation?
2. How many resources \ time \ infrastructure required: To understand requirements and plan test strategy? To write and review test cases? For test execution? For reporting?
3. Why estimation of the testing process: To deliver quality product on time? To complete process in low budget?
   

Commonly in a software development environment, the focus of the estimation is always on the development activity. The project estimation team’s main focus will be to estimate the effort for the development and underestimate the testing effort. Usually, effort for testing phase is considered to be 30-40% of the total development effort. The effect of this approach results in insufficient testing effort and test coverage.

Test estimation is an iterative process. Usually we create estimations in terms of test cases, early in the test cycle. Here we need to make it clear to the clients; "This estimation is based on current understanding of the requirements and can be changed". It provides flexibility later in the test cycle to ask for more resources and time.

Challenges:

Estimation of effort for the testing project done by the testing team is always considered very high by the other team. The reason being no formal process for estimation exists. The real challenge for the testing team at the organization and project level is to come out with the effective estimation technique for testing activity.
Effort estimation at the time of making proposals is very difficult because at that time requirements are not so much clear and it is very difficult to see end to end scenarios at that time. Therefore at that time software industry standard approach of taking around 40% is followed relative to the development efforts.
Sometimes, even expert managers, especially those who are unfamiliar with software testing project, have real trouble estimating the time and resources to allocate for testing. It’s important to combine good estimation techniques with an understanding of the factors that can influence effort, time, dependencies, and resources.
It is the reality that successful test estimation is a challenge for most organizations, since few can accurately estimate software project development efforts, but much less the testing effort of a project.


Solution?

There is several estimation approaches used in the industries. Followings are some of the different approaches.

Implicit Risk Context Approach: A typical approach to test estimation is for a manager to implicitly use risk context, in combination with past personal experiences in the organization, to choose a level of resources to allocate to testing. This is essentially an intuitive guess based on experience.

Metrics-Based Approach: A useful approach is to track past experience of an organization's various projects and the associated test effort that worked well for projects. Once there is a set of data covering characteristics for a reasonable number of projects, then this 'past experience' information can be used for future test project planning. But with this approach determining and collecting useful project metrics over time can be an extremely difficult task.

Iterative Approach: With this approach a rough testing estimate can be made for large test efforts. When testing starts, a more refined estimate is made after a small percentage (e.g., 5%) of the first estimate's work is done. At this point testers have obtained additional test project knowledge and a better understanding of issues, general software quality, and risk. Test plans and schedules can be re-factored if necessary and a new estimate provided. More refined estimate can be made after a somewhat larger percentage (e.g., 10%) of the new work estimation is done.

Test Work Breakdown Approach: This approach is to separate the expected testing tasks into a collection of small tasks, for which estimates can be made with reasonable accuracy. But in many large projects, this is not the case. If quite large numbers of defects are being found in a project, then it will increase the time required for testing, re-testing, defect analysis and reporting. It will also add to the time required for development, and if development schedules and efforts do not go as planned, this will further impact testing.

Since estimation for testing varies from project to project as the test strategy depends on the project requirement, the testing team should be the part of estimation process for the requirement stage.

The test estimation technique should consist of the following major steps.

1. Knowing the test life cycle: The main focus of this activity is to identify the different stages of test life cycle where testing is to be done. The phases of test life cycles can be:

• Test requirement phase
• Test case design phase
• Test script development phase
• Test execution phase
• Test result analysis and documentation phase
  

The objective of identifying different life cycle is to estimate size and effort for each phase and document the estimation assumption for each phase. Later on the team can work back on the estimation for variation in estimation.

2. Output identification of each test phase: In this activity the product of each phase of test life cycle can be identified. The job of the testing team is to estimate the size of the work products. The following figure shows the different phases of testing and respective products.



3. Estimating size for each phase: The main activity is to estimate the size of the product for each phase. When estimating the size for each phase, all the assumption considered during estimation, should be documented for future analysis if, any variation in estimation exists. Some predefined techniques e.g. Wide Band Size Estimation can be used to do size estimation. Even previous project data can be used as reference depending upon the project Scope and nature.

Size estimation for each phase is detailed below.

I. Test requirement phase: The main goal is to analyze the project requirement and identify the test requirement based on the test strategy. The testing team should brain storm the project requirement and categorize the test requirement.
Test requirement can be categorized mainly as, functional, GUI, performance, security and compatibility.
Here, the testing team can identify the positive and negative test requirement for each category. This will help the testing team in identifying the test data. The purpose of this activity is that the testing team is the part of the estimation from the requirement stage. Also any ambiguity in the requirement can also be addressed at this stage.

Size estimation process:
Input: Project requirement, Use case
Process: Project requirement analysis, test requirements identification and categorization, documentation of estimation assumptions
Output: Number of test requirements, functional requirements, performance requirements, compatibility test requirements and, security test requirements.

Note:
1. Under each category we can have positive and negative conditions. e.g. estimation assumption.
2. Platform for testing identified during the Estimation will not change during the Test Life Cycle.

II. Test case design phase: The aim of this phase is to identify and generate test cases. Both manual and automated test cases are identified in this phase. Also test data will be identified for positive and negative condition. Test strategy can be used as a base to design of test case.

Size estimation process:
Input: Test requirement
Process: Test requirement reanalysis, identification and categorisation of test cases based on test requirements; apply conversion factor, documentation of estimation assumptions
Output: Number of functional, GUI, performance, compatibility, security test cases
Conversion Factor: The best way to identify the conversion factor is clubbing the test requirement, which have similar scenarios.

Number of test cases = Number of test requirements / N
Here N is the conversion factor.
If N = 1 then Number of test cases = Number of test requirements
Also using the conversion factor user can calculate the test coverage against the requirement. For example one test case can test two requirements.

III. Test script development phase (if automation): The aim of this phase is to automate test cases using automated test tools. The test asset generated from this phase is test script. The goal of the estimation team is to identify number of test script and source lines of code (SLOC).

Size estimation process:
Input: Test cases, external Interfaces, third party controls, verification points
Process: Identification of automatable test cases, external interfaces, third party controls, reusable component based on the product feature, and documentation for the estimation assumptions
Output: Number of test scripts, reusable components, and total SLOC.

Note:
Total SLOC = Executable lines + Comment lines
1 Test script = N SLOC (where ‘N’ is average SLOC per Script)
Total SLOC = N x Number of Test scripts

IV. Test Case Execution Phase: Main aim of this phase is to execute the manual and automated test case. The test asset for this phase is test log. The goal of testing team is to identify the number of test cases to be executed.

Size estimation process:
Input: Number of manual test cases and automated test cases
Process: Identification of number of manual test cases and automated test cases, number of defects to be detected (e.g. N test cases will detect one defect) and documentation for the estimation assumptions
Output: Number of manual and automated test cases to be executed, and number of probable defects

Note:
Number of defects will help the user to identify number of test cases to be executed during regression

V. Regression Cycle: The aim of this activity is to re-execute the test cases. It is quite difficult to identify how many iteration of testing needs to be carried out. Depending upon the number of defects, the estimation team can identify the number of test cases to be executed.

Size estimation process:
Input: Number of manual and automated test cases, number of defects uncovered
Process: Identification of number of manual and automated test cases to be executed, document assumptions, Identification of test coverage criteria
Output: Number of regression cycle, effort estimation:

4. Estimating effort: Based upon the size estimated for each stage of testing phase, the total testing effort for the project can be calculated.

Total Effort of each phase = Size of the Product * Productivity
Productivity factor should be considered separately for manual and automated testing. Productivity is specific to organization as it is influenced by knowledge and skill level of the testing team.

e.g. Manual Testing
Total effort for test case generation = Number of manual test cases * Productivity
Where Productivity = Number of test case generated /Time

e.g. Automated Testing
Total effort for test script generation = Total SLOC * Productivity
Where Productivity = Total SLOC / Time
SLOC = executable + non executable line
Total testing effort for first iteration = Effort for test requirement + Effort for test development + Effort for test execution

Note:
Test development comprises of test case and test script development
Effort for regression testing: It consists of two components, manual testing and automated testing.

This effort can be split into two components
• Test case or Test script development/Enhancement
• Test execution

Total Regression Effort for manual testing = Test case development/Enhancement + Test Execution
Total Regression Effort for Automation = Test script development/Enhancement + Test Execution
Where Test Script Development = (Total SLOC * Reuse effectiveness) * Productivity
Based on this, total test project effort will be
Total test project effort = Total testing effort + Regression testing effort + Planning effort + Management effort + Rework effort

Note:
Rework Effort depends on the risk factor specific to the project.

Conclusion:

At the project level, with this estimation technique, the testing team will arrive very near to accurate estimate and provide better ability to understand test cycle to the management or client. Testing team will be able to trace any variation in estimation as all the estimation assumptions will be documented for each test phase. This technique doesn’t introduce any complexity factors for estimation, and it will reduce the estimation variation as all the parameters with respect to requirement can be addressed.
For an organization a standard estimation technique can help the testing team in improving the existing estimation process. With this estimation technique, the testing team can strongly defend and reason the effort required for testing. A standard estimation technique will help build an organization metrics data for testing that can be used for future testing project estimation.
I conclude by saying; there are three factors that decide test estimation: experience, manipulation and intelligent planning. We need to utilize more of negotiation and communication skills than any proven method.

By: Kamaljjeet Sinha
kamaljeet.sinha@tcl-asia.com