Zappers - 18th August

The 5th edition of the ‘Zappers – Eventful Testing’ was yet another successfully organized event by TCL, encouraging and discovering avid testers across the city. The event was wholly organized and sponsored TCL and held at The Grand Magrath Hotel in Bangalore.
As attendees began walking into the venue by 6:30 pm, they were greeted with some appetizing food and post-work beer. It was nice to see new faces  and some familiar ones, socialize with eachother as they discussed the proceedings of the evening. By 7:45 most of the people were well settled and rearing to go. There was the omnipresent connectivity glitch which was rectified soon by the hotel authorities.


The event saw an array of talented testers glued to their screens battling it out, finding bugs and logging it into the bug reporting system. The event was a conducted with a view to firstly encourage peer sharing and enhance the potential of many enthusiastic testers from various companies around Bangalore. HP, SourceEdge, Canary’s, Thomson Reuters, EFI, Fidelity, and Collabera were some of the participating organizations. There were a number of well known testers from the Weekend Testing community, who were looking to establish their dominance and well, did to a certain level. The applications which were given to the participants to test were Irfan View, Google Chrome 6 Beta, Song Bird 1.8 Beta, Firefox 4 Beta and finally the popular gaming application Quake 3 Arena. Teams logged a total of 190 defects in 60 minutes which kept our judges on their toes.

The evening of competition and anticipation was brought to an end by participants from the weekend testers bagging the 1st place and walking away with a cash prize. The team consisted of Pradeep, Sharath and Santhosh. The prize for the Show Stopper bug went to Parimala who almost couldn’t make it to the event but was quite evidently happy she did. The prizes were handed out by the Chairman, Stewart Noakes and the COO of TCL India, Manoj Chandrappa. The event concluded with a dinner and a few photo sessions.
The event is termed as a success purely because of the interest it generated. This event is not aimed at having hundreds of participants but to form a platform and create a community which would aid in doing so. We can just hope there were new connections formed and extend our gratitude to each and every participant alike.


Pictures of the event can be found on our facebook page.

Marcel Hoover
TCL India

Early Testing: Always Consumer Friendly

As a testing professional, it is up to us to deliver a quality product that will satisfy our customers and lead to a more successful and prosperous organization. To do this we must locate and correct defects efficiently, select processes and technology as well as lead others to apply them, and effectively communicate the results of your efforts to management.

A test scenario, a bug report deferred as-designed: According to Associate Press “Toyota is recalling 437,000 Prius and other hybrid vehicles worldwide to fix brake problem.” Toyota is the world’s largest automaker with impeccable quality reputation up to now.

As it turns out, the cause of the problem is a bug in the software that controls the brake system. “There have been about 200 complaints in Japan and the U.S. about a delay when the brakes in the Prius were pressed in cold conditions and on some bumpy roads. The delay doesn’t indicate a brake failure. The company says the problem can be fixed in 40 minutes with new software that oversees the controls of the antilock brakes.”

I don’t mean to wage in on Toyota disastrous situation. I just think that this is an essential software testing lesson and it illustrates some key issues that testers like us have experienced over the years. “After receiving a similar report from the U.S. in October, Toyota’s tests concluded that a glitch in the Prius antilock brake system software could reduce braking force when drivers traveled across bumpy surfaces.
The company wrote that “although this system was operating as intended,” it decided to make a change to its production line in January to address the problem,” Kareyama reported.

If I wrote a bug report, it would look like this:

• Bug Report Summary: Braking force is reduced (the problem) when driving across bumpy surface (the scenario).
• Reproducibility: Intermittent.
  

It seems that the Resolution can be treated as “As-designed” (although it does not work in certain scenarios, that was how Toyota designed it), or “Deferred” (the problem is acknowledged and will be fixed later), or “To-be-fixed” (but in the next release). We already know the outcome. So, what do we learn?

• Scenario-based and exploratory testing during system-test is essential—this requires skill and creativity in test design, not test-driven development (which is all good and an essential element to software development) or process-driven testing such as CMMi or ISO-9000 (which is good for quality control).

• Software testing is not quality assurance—testers can report bugs, but a decision for corrective action lies somewhere else. There is nothing wrong with that. Testing is an important information service provider. We find and report problems.
• Software is everywhere; and software is buggy—our job as testers is to find bug by breaking software in every possible way we can!
  

Last but not least, we, testers don’t create software or the product; we help improve software reliability through our defect finding skills. Buy breaking software, we are saving consumer and our company time and money (even contributing to public safety in this case). So, keep breaking software tester…Your job is much appreciated!

By Shri Vidhya
shri.vidhya@tcl-asia.com

A Thought on Test Effort Estimation

Introduction

Test estimation always starts with the following common questions.

1. What are the requirements to test software: Resource? Infrastructure? Time? Proper planning and Estimation?
2. How many resources \ time \ infrastructure required: To understand requirements and plan test strategy? To write and review test cases? For test execution? For reporting?
3. Why estimation of the testing process: To deliver quality product on time? To complete process in low budget?
   

Commonly in a software development environment, the focus of the estimation is always on the development activity. The project estimation team’s main focus will be to estimate the effort for the development and underestimate the testing effort. Usually, effort for testing phase is considered to be 30-40% of the total development effort. The effect of this approach results in insufficient testing effort and test coverage.

Test estimation is an iterative process. Usually we create estimations in terms of test cases, early in the test cycle. Here we need to make it clear to the clients; "This estimation is based on current understanding of the requirements and can be changed". It provides flexibility later in the test cycle to ask for more resources and time.

Challenges:

Estimation of effort for the testing project done by the testing team is always considered very high by the other team. The reason being no formal process for estimation exists. The real challenge for the testing team at the organization and project level is to come out with the effective estimation technique for testing activity.
Effort estimation at the time of making proposals is very difficult because at that time requirements are not so much clear and it is very difficult to see end to end scenarios at that time. Therefore at that time software industry standard approach of taking around 40% is followed relative to the development efforts.
Sometimes, even expert managers, especially those who are unfamiliar with software testing project, have real trouble estimating the time and resources to allocate for testing. It’s important to combine good estimation techniques with an understanding of the factors that can influence effort, time, dependencies, and resources.
It is the reality that successful test estimation is a challenge for most organizations, since few can accurately estimate software project development efforts, but much less the testing effort of a project.


Solution?

There is several estimation approaches used in the industries. Followings are some of the different approaches.

Implicit Risk Context Approach: A typical approach to test estimation is for a manager to implicitly use risk context, in combination with past personal experiences in the organization, to choose a level of resources to allocate to testing. This is essentially an intuitive guess based on experience.

Metrics-Based Approach: A useful approach is to track past experience of an organization's various projects and the associated test effort that worked well for projects. Once there is a set of data covering characteristics for a reasonable number of projects, then this 'past experience' information can be used for future test project planning. But with this approach determining and collecting useful project metrics over time can be an extremely difficult task.

Iterative Approach: With this approach a rough testing estimate can be made for large test efforts. When testing starts, a more refined estimate is made after a small percentage (e.g., 5%) of the first estimate's work is done. At this point testers have obtained additional test project knowledge and a better understanding of issues, general software quality, and risk. Test plans and schedules can be re-factored if necessary and a new estimate provided. More refined estimate can be made after a somewhat larger percentage (e.g., 10%) of the new work estimation is done.

Test Work Breakdown Approach: This approach is to separate the expected testing tasks into a collection of small tasks, for which estimates can be made with reasonable accuracy. But in many large projects, this is not the case. If quite large numbers of defects are being found in a project, then it will increase the time required for testing, re-testing, defect analysis and reporting. It will also add to the time required for development, and if development schedules and efforts do not go as planned, this will further impact testing.

Since estimation for testing varies from project to project as the test strategy depends on the project requirement, the testing team should be the part of estimation process for the requirement stage.

The test estimation technique should consist of the following major steps.

1. Knowing the test life cycle: The main focus of this activity is to identify the different stages of test life cycle where testing is to be done. The phases of test life cycles can be:

• Test requirement phase
• Test case design phase
• Test script development phase
• Test execution phase
• Test result analysis and documentation phase
  

The objective of identifying different life cycle is to estimate size and effort for each phase and document the estimation assumption for each phase. Later on the team can work back on the estimation for variation in estimation.

2. Output identification of each test phase: In this activity the product of each phase of test life cycle can be identified. The job of the testing team is to estimate the size of the work products. The following figure shows the different phases of testing and respective products.



3. Estimating size for each phase: The main activity is to estimate the size of the product for each phase. When estimating the size for each phase, all the assumption considered during estimation, should be documented for future analysis if, any variation in estimation exists. Some predefined techniques e.g. Wide Band Size Estimation can be used to do size estimation. Even previous project data can be used as reference depending upon the project Scope and nature.

Size estimation for each phase is detailed below.

I. Test requirement phase: The main goal is to analyze the project requirement and identify the test requirement based on the test strategy. The testing team should brain storm the project requirement and categorize the test requirement.
Test requirement can be categorized mainly as, functional, GUI, performance, security and compatibility.
Here, the testing team can identify the positive and negative test requirement for each category. This will help the testing team in identifying the test data. The purpose of this activity is that the testing team is the part of the estimation from the requirement stage. Also any ambiguity in the requirement can also be addressed at this stage.

Size estimation process:
Input: Project requirement, Use case
Process: Project requirement analysis, test requirements identification and categorization, documentation of estimation assumptions
Output: Number of test requirements, functional requirements, performance requirements, compatibility test requirements and, security test requirements.

Note:
1. Under each category we can have positive and negative conditions. e.g. estimation assumption.
2. Platform for testing identified during the Estimation will not change during the Test Life Cycle.

II. Test case design phase: The aim of this phase is to identify and generate test cases. Both manual and automated test cases are identified in this phase. Also test data will be identified for positive and negative condition. Test strategy can be used as a base to design of test case.

Size estimation process:
Input: Test requirement
Process: Test requirement reanalysis, identification and categorisation of test cases based on test requirements; apply conversion factor, documentation of estimation assumptions
Output: Number of functional, GUI, performance, compatibility, security test cases
Conversion Factor: The best way to identify the conversion factor is clubbing the test requirement, which have similar scenarios.

Number of test cases = Number of test requirements / N
Here N is the conversion factor.
If N = 1 then Number of test cases = Number of test requirements
Also using the conversion factor user can calculate the test coverage against the requirement. For example one test case can test two requirements.

III. Test script development phase (if automation): The aim of this phase is to automate test cases using automated test tools. The test asset generated from this phase is test script. The goal of the estimation team is to identify number of test script and source lines of code (SLOC).

Size estimation process:
Input: Test cases, external Interfaces, third party controls, verification points
Process: Identification of automatable test cases, external interfaces, third party controls, reusable component based on the product feature, and documentation for the estimation assumptions
Output: Number of test scripts, reusable components, and total SLOC.

Note:
Total SLOC = Executable lines + Comment lines
1 Test script = N SLOC (where ‘N’ is average SLOC per Script)
Total SLOC = N x Number of Test scripts

IV. Test Case Execution Phase: Main aim of this phase is to execute the manual and automated test case. The test asset for this phase is test log. The goal of testing team is to identify the number of test cases to be executed.

Size estimation process:
Input: Number of manual test cases and automated test cases
Process: Identification of number of manual test cases and automated test cases, number of defects to be detected (e.g. N test cases will detect one defect) and documentation for the estimation assumptions
Output: Number of manual and automated test cases to be executed, and number of probable defects

Note:
Number of defects will help the user to identify number of test cases to be executed during regression

V. Regression Cycle: The aim of this activity is to re-execute the test cases. It is quite difficult to identify how many iteration of testing needs to be carried out. Depending upon the number of defects, the estimation team can identify the number of test cases to be executed.

Size estimation process:
Input: Number of manual and automated test cases, number of defects uncovered
Process: Identification of number of manual and automated test cases to be executed, document assumptions, Identification of test coverage criteria
Output: Number of regression cycle, effort estimation:

4. Estimating effort: Based upon the size estimated for each stage of testing phase, the total testing effort for the project can be calculated.

Total Effort of each phase = Size of the Product * Productivity
Productivity factor should be considered separately for manual and automated testing. Productivity is specific to organization as it is influenced by knowledge and skill level of the testing team.

e.g. Manual Testing
Total effort for test case generation = Number of manual test cases * Productivity
Where Productivity = Number of test case generated /Time

e.g. Automated Testing
Total effort for test script generation = Total SLOC * Productivity
Where Productivity = Total SLOC / Time
SLOC = executable + non executable line
Total testing effort for first iteration = Effort for test requirement + Effort for test development + Effort for test execution

Note:
Test development comprises of test case and test script development
Effort for regression testing: It consists of two components, manual testing and automated testing.

This effort can be split into two components
• Test case or Test script development/Enhancement
• Test execution

Total Regression Effort for manual testing = Test case development/Enhancement + Test Execution
Total Regression Effort for Automation = Test script development/Enhancement + Test Execution
Where Test Script Development = (Total SLOC * Reuse effectiveness) * Productivity
Based on this, total test project effort will be
Total test project effort = Total testing effort + Regression testing effort + Planning effort + Management effort + Rework effort

Note:
Rework Effort depends on the risk factor specific to the project.

Conclusion:

At the project level, with this estimation technique, the testing team will arrive very near to accurate estimate and provide better ability to understand test cycle to the management or client. Testing team will be able to trace any variation in estimation as all the estimation assumptions will be documented for each test phase. This technique doesn’t introduce any complexity factors for estimation, and it will reduce the estimation variation as all the parameters with respect to requirement can be addressed.
For an organization a standard estimation technique can help the testing team in improving the existing estimation process. With this estimation technique, the testing team can strongly defend and reason the effort required for testing. A standard estimation technique will help build an organization metrics data for testing that can be used for future testing project estimation.
I conclude by saying; there are three factors that decide test estimation: experience, manipulation and intelligent planning. We need to utilize more of negotiation and communication skills than any proven method.

By: Kamaljjeet Sinha
kamaljeet.sinha@tcl-asia.com