Transition Consulting Ltd. India

Zappers - 18th August

2010-08-24T17:15:00.001+05:30

The 5th edition of the ‘Zappers – Eventful Testing’ was yet another successfully organized event by TCL, encouraging and discovering avid testers across the city. The event was wholly organized and sponsored TCL and held at The Grand Magrath Hotel in Bangalore.
As attendees began walking into the venue by 6:30 pm, they were greeted with some appetizing food and post-work beer. It was nice to see new faces and some familiar ones, socialize with eachother as they discussed the proceedings of the evening. By 7:45 most of the people were well settled and rearing to go. There was the omnipresent connectivity glitch which was rectified soon by the hotel authorities.

The event saw an array of talented testers glued to their screens battling it out, finding bugs and logging it into the bug reporting system. The event was a conducted with a view to firstly encourage peer sharing and enhance the potential of many enthusiastic testers from various companies around Bangalore. HP, SourceEdge, Canary’s, Thomson Reuters, EFI, Fidelity, and Collabera were some of the participating organizations. There were a number of well known testers from the Weekend Testing community, who were looking to establish their dominance and well, did to a certain level. The applications which were given to the participants to test were Irfan View, Google Chrome 6 Beta, Song Bird 1.8 Beta, Firefox 4 Beta and finally the popular gaming application Quake 3 Arena. Teams logged a total of 190 defects in 60 minutes which kept our judges on their toes.

The evening of competition and anticipation was brought to an end by participants from the weekend testers bagging the 1st place and walking away with a cash prize. The team consisted of Pradeep, Sharath and Santhosh. The prize for the Show Stopper bug went to Parimala who almost couldn’t make it to the event but was quite evidently happy she did. The prizes were handed out by the Chairman, Stewart Noakes and the COO of TCL India, Manoj Chandrappa. The event concluded with a dinner and a few photo sessions.
The event is termed as a success purely because of the interest it generated. This event is not aimed at having hundreds of participants but to form a platform and create a community which would aid in doing so. We can just hope there were new connections formed and extend our gratitude to each and every participant alike.

Pictures of the event can be found on our facebook page.

Marcel Hoover
TCL India

The TCL Blog is up!!

2010-07-14T15:11:00.000+05:30

It has been a hectic two weeks, juggling between working on the blog, getting the one year press release prepared and the usual tasks. With TCL India finishing a year and a big Zappers coming up next month we each have our work cut out for the weeks to come.
Let me first introduce to you the revamped blog. Although most of you must have observed a drastic change from what the blog was earlier on, it actually isn't very different. Cosmetically we have tried to give it a personal touch but have stayed true to the TCL brand colors. We already have an enthusiastic bunch of people waiting to submit all sorts of articles for the blog, and now they can.
The section included are quite basic and interactivity may be minimum at this point, but this WILL change over time. We at TCL look to promote testing to the masses as we have done with the Zappers for crowdsourcing. You as a reader can contribute by sending in articles and commenting on other people work.
Do note, this is not a substitute to our website. Even though we have included the news and jobs section in the blog, the purpose remains the same; promoting knowledge sharing.
Please feel free to go through the blog and post your suggestions.

We hope to have a constant flow of articles from almost anyone, and anything that you would want to share. We will constantly be tweeting so you are kept update on our happenings. If you aren't already following us do so by hitting the twitter button at the top of the page.

I would like to thank Vidaya, Priyanka, Sandhya and Deepak for their initiatives and helping out a novice like me to get this started.

Until next time, happy blogging.

by Siddharth Mukherjee

Early Testing: Always Consumer Friendly

2010-07-09T14:53:00.004+05:30

As a testing professional, it is up to us to deliver a quality product that will satisfy our customers and lead to a more successful and prosperous organization. To do this we must locate and correct defects efficiently, select processes and technology as well as lead others to apply them, and effectively communicate the results of your efforts to management.

A test scenario, a bug report deferred as-designed: According to Associate Press “Toyota is recalling 437,000 Prius and other hybrid vehicles worldwide to fix brake problem.” Toyota is the world’s largest automaker with impeccable quality reputation up to now.

As it turns out, the cause of the problem is a bug in the software that controls the brake system. “There have been about 200 complaints in Japan and the U.S. about a delay when the brakes in the Prius were pressed in cold conditions and on some bumpy roads. The delay doesn’t indicate a brake failure. The company says the problem can be fixed in 40 minutes with new software that oversees the controls of the antilock brakes.”

I don’t mean to wage in on Toyota disastrous situation. I just think that this is an essential software testing lesson and it illustrates some key issues that testers like us have experienced over the years. “After receiving a similar report from the U.S. in October, Toyota’s tests concluded that a glitch in the Prius antilock brake system software could reduce braking force when drivers traveled across bumpy surfaces.
The company wrote that “although this system was operating as intended,” it decided to make a change to its production line in January to address the problem,” Kareyama reported.

If I wrote a bug report, it would look like this:

Bug Report Summary: Braking force is reduced (the problem) when driving across bumpy surface (the scenario).
Reproducibility: Intermittent.

It seems that the Resolution can be treated as “As-designed” (although it does not work in certain scenarios, that was how Toyota designed it), or “Deferred” (the problem is acknowledged and will be fixed later), or “To-be-fixed” (but in the next release). We already know the outcome. So, what do we learn?

Scenario-based and exploratory testing during system-test is essential—this requires skill and creativity in test design, not test-driven development (which is all good and an essential element to software development) or process-driven testing such as CMMi or ISO-9000 (which is good for quality control).

Software testing is not quality assurance—testers can report bugs, but a decision for corrective action lies somewhere else. There is nothing wrong with that. Testing is an important information service provider. We find and report problems.
Software is everywhere; and software is buggy—our job as testers is to find bug by breaking software in every possible way we can!

Last but not least, we, testers don’t create software or the product; we help improve software reliability through our defect finding skills. Buy breaking software, we are saving consumer and our company time and money (even contributing to public safety in this case). So, keep breaking software tester…Your job is much appreciated!

By Shri Vidhya
shri.vidhya@tcl-asia.com

A Thought on Test Effort Estimation

2010-07-09T10:41:00.010+05:30

Introduction

Test estimation always starts with the following common questions.

What are the requirements to test software: Resource? Infrastructure? Time? Proper planning and Estimation?
How many resources \ time \ infrastructure required: To understand requirements and plan test strategy? To write and review test cases? For test execution? For reporting?
Why estimation of the testing process: To deliver quality product on time? To complete process in low budget?

Commonly in a software development environment, the focus of the estimation is always on the development activity. The project estimation team’s main focus will be to estimate the effort for the development and underestimate the testing effort. Usually, effort for testing phase is considered to be 30-40% of the total development effort. The effect of this approach results in insufficient testing effort and test coverage.

Test estimation is an iterative process. Usually we create estimations in terms of test cases, early in the test cycle. Here we need to make it clear to the clients; "This estimation is based on current understanding of the requirements and can be changed". It provides flexibility later in the test cycle to ask for more resources and time.

Challenges:

Estimation of effort for the testing project done by the testing team is always considered very high by the other team. The reason being no formal process for estimation exists. The real challenge for the testing team at the organization and project level is to come out with the effective estimation technique for testing activity.
Effort estimation at the time of making proposals is very difficult because at that time requirements are not so much clear and it is very difficult to see end to end scenarios at that time. Therefore at that time software industry standard approach of taking around 40% is followed relative to the development efforts.
Sometimes, even expert managers, especially those who are unfamiliar with software testing project, have real trouble estimating the time and resources to allocate for testing. It’s important to combine good estimation techniques with an understanding of the factors that can influence effort, time, dependencies, and resources.
It is the reality that successful test estimation is a challenge for most organizations, since few can accurately estimate software project development efforts, but much less the testing effort of a project.

Solution?

There is several estimation approaches used in the industries. Followings are some of the different approaches.

Implicit Risk Context Approach: A typical approach to test estimation is for a manager to implicitly use risk context, in combination with past personal experiences in the organization, to choose a level of resources to allocate to testing. This is essentially an intuitive guess based on experience.

Metrics-Based Approach: A useful approach is to track past experience of an organization's various projects and the associated test effort that worked well for projects. Once there is a set of data covering characteristics for a reasonable number of projects, then this 'past experience' information can be used for future test project planning. But with this approach determining and collecting useful project metrics over time can be an extremely difficult task.

Iterative Approach: With this approach a rough testing estimate can be made for large test efforts. When testing starts, a more refined estimate is made after a small percentage (e.g., 5%) of the first estimate's work is done. At this point testers have obtained additional test project knowledge and a better understanding of issues, general software quality, and risk. Test plans and schedules can be re-factored if necessary and a new estimate provided. More refined estimate can be made after a somewhat larger percentage (e.g., 10%) of the new work estimation is done.

Test Work Breakdown Approach: This approach is to separate the expected testing tasks into a collection of small tasks, for which estimates can be made with reasonable accuracy. But in many large projects, this is not the case. If quite large numbers of defects are being found in a project, then it will increase the time required for testing, re-testing, defect analysis and reporting. It will also add to the time required for development, and if development schedules and efforts do not go as planned, this will further impact testing.

Since estimation for testing varies from project to project as the test strategy depends on the project requirement, the testing team should be the part of estimation process for the requirement stage.

The test estimation technique should consist of the following major steps.

Knowing the test life cycle: The main focus of this activity is to identify the different stages of test life cycle where testing is to be done. The phases of test life cycles can be:

Test requirement phase
Test case design phase
Test script development phase
Test execution phase
Test result analysis and documentation phase

The objective of identifying different life cycle is to estimate size and effort for each phase and document the estimation assumption for each phase. Later on the team can work back on the estimation for variation in estimation.

2. Output identification of each test phase: In this activity the product of each phase of test life cycle can be identified. The job of the testing team is to estimate the size of the work products. The following figure shows the different phases of testing and respective products.

3. Estimating size for each phase: The main activity is to estimate the size of the product for each phase. When estimating the size for each phase, all the assumption considered during estimation, should be documented for future analysis if, any variation in estimation exists. Some predefined techniques e.g. Wide Band Size Estimation can be used to do size estimation. Even previous project data can be used as reference depending upon the project Scope and nature.

Size estimation for each phase is detailed below.

I. Test requirement phase: The main goal is to analyze the project requirement and identify the test requirement based on the test strategy. The testing team should brain storm the project requirement and categorize the test requirement.
Test requirement can be categorized mainly as, functional, GUI, performance, security and compatibility.
Here, the testing team can identify the positive and negative test requirement for each category. This will help the testing team in identifying the test data. The purpose of this activity is that the testing team is the part of the estimation from the requirement stage. Also any ambiguity in the requirement can also be addressed at this stage.

Size estimation process:
Input: Project requirement, Use case
Process: Project requirement analysis, test requirements identification and categorization, documentation of estimation assumptions
Output: Number of test requirements, functional requirements, performance requirements, compatibility test requirements and, security test requirements.

Note:
1. Under each category we can have positive and negative conditions. e.g. estimation assumption.
2. Platform for testing identified during the Estimation will not change during the Test Life Cycle.

II. Test case design phase: The aim of this phase is to identify and generate test cases. Both manual and automated test cases are identified in this phase. Also test data will be identified for positive and negative condition. Test strategy can be used as a base to design of test case.

Size estimation process:
Input: Test requirement
Process: Test requirement reanalysis, identification and categorisation of test cases based on test requirements; apply conversion factor, documentation of estimation assumptions
Output: Number of functional, GUI, performance, compatibility, security test cases
Conversion Factor: The best way to identify the conversion factor is clubbing the test requirement, which have similar scenarios.

Number of test cases = Number of test requirements / N
Here N is the conversion factor.
If N = 1 then Number of test cases = Number of test requirements
Also using the conversion factor user can calculate the test coverage against the requirement. For example one test case can test two requirements.

III. Test script development phase (if automation): The aim of this phase is to automate test cases using automated test tools. The test asset generated from this phase is test script. The goal of the estimation team is to identify number of test script and source lines of code (SLOC).

Size estimation process:
Input: Test cases, external Interfaces, third party controls, verification points
Process: Identification of automatable test cases, external interfaces, third party controls, reusable component based on the product feature, and documentation for the estimation assumptions
Output: Number of test scripts, reusable components, and total SLOC.

Note:
Total SLOC = Executable lines + Comment lines
1 Test script = N SLOC (where ‘N’ is average SLOC per Script)
Total SLOC = N x Number of Test scripts

IV. Test Case Execution Phase: Main aim of this phase is to execute the manual and automated test case. The test asset for this phase is test log. The goal of testing team is to identify the number of test cases to be executed.

Size estimation process:
Input: Number of manual test cases and automated test cases
Process: Identification of number of manual test cases and automated test cases, number of defects to be detected (e.g. N test cases will detect one defect) and documentation for the estimation assumptions
Output: Number of manual and automated test cases to be executed, and number of probable defects

Note:
Number of defects will help the user to identify number of test cases to be executed during regression

V. Regression Cycle: The aim of this activity is to re-execute the test cases. It is quite difficult to identify how many iteration of testing needs to be carried out. Depending upon the number of defects, the estimation team can identify the number of test cases to be executed.

Size estimation process:
Input: Number of manual and automated test cases, number of defects uncovered
Process: Identification of number of manual and automated test cases to be executed, document assumptions, Identification of test coverage criteria
Output: Number of regression cycle, effort estimation:

4. Estimating effort: Based upon the size estimated for each stage of testing phase, the total testing effort for the project can be calculated.

Total Effort of each phase = Size of the Product * Productivity
Productivity factor should be considered separately for manual and automated testing. Productivity is specific to organization as it is influenced by knowledge and skill level of the testing team.

e.g. Manual Testing
Total effort for test case generation = Number of manual test cases * Productivity
Where Productivity = Number of test case generated /Time

e.g. Automated Testing
Total effort for test script generation = Total SLOC * Productivity
Where Productivity = Total SLOC / Time
SLOC = executable + non executable line
Total testing effort for first iteration = Effort for test requirement + Effort for test development + Effort for test execution

Note:
Test development comprises of test case and test script development
Effort for regression testing: It consists of two components, manual testing and automated testing.

This effort can be split into two components
• Test case or Test script development/Enhancement
• Test execution

Total Regression Effort for manual testing = Test case development/Enhancement + Test Execution
Total Regression Effort for Automation = Test script development/Enhancement + Test Execution
Where Test Script Development = (Total SLOC * Reuse effectiveness) * Productivity
Based on this, total test project effort will be
Total test project effort = Total testing effort + Regression testing effort + Planning effort + Management effort + Rework effort

Note:
Rework Effort depends on the risk factor specific to the project.

Conclusion:

At the project level, with this estimation technique, the testing team will arrive very near to accurate estimate and provide better ability to understand test cycle to the management or client. Testing team will be able to trace any variation in estimation as all the estimation assumptions will be documented for each test phase. This technique doesn’t introduce any complexity factors for estimation, and it will reduce the estimation variation as all the parameters with respect to requirement can be addressed.
For an organization a standard estimation technique can help the testing team in improving the existing estimation process. With this estimation technique, the testing team can strongly defend and reason the effort required for testing. A standard estimation technique will help build an organization metrics data for testing that can be used for future testing project estimation.
I conclude by saying; there are three factors that decide test estimation: experience, manipulation and intelligent planning. We need to utilize more of negotiation and communication skills than any proven method.

By: Kamaljjeet Sinha
kamaljeet.sinha@tcl-asia.com

SoftTec 2010 Conference Report

2010-07-09T10:21:00.005+05:30

July 3rd 2010, the day for a team of testers from TCL India, to attend a conference on Software Testing; Softech 2010. The conference had two tracks, one was the Test Professional Track which covered technical sessions on a wide-range of testing topics, combining the best software testing theories with real-world examples. The other was the Test Leadership Track which focused on managing testers and the software test areas of the product lifecycle. Among the team, it was decided that Test Professional Track will be attended by me (Mahesh) and Ms. Priyanka and the Test Leadership Track will be attended by Mr. Vidaya and Mr. Kiran.
The conference was organised at the Nimhans Convention Centre scheduled at 9:00 am. Before entering the auditorium adhering to our company’s vision, we were mentally prepared to take the full benefit of the Conference, both knowledge wise, as well as networking with other software professional.

As the hour hand of the clock struck 9, the session began with the inaugural talk from Mr. T Ashok, founder and CEO of STAG Software. In this session he welcomed all the delegates and ended his speech with a story called “The Ugly Duckling”.
The following were the session planned for the Test Professional Track:

Mobile Application Testing
Globalization Testing
Cloud Testing
Open Source in Performance Testing
Application Security TestingMobile Protocol Testing.

Below is a overview of the above mentioned topics:

Mobile Application Testing: “Each day ends with new and innovative ideas in technology”, a quote to an example; For the computer to reach a target of 1 billion it took almost 50 years, television 20 yrs, the internet 10yrs but mobile device took just 5yrs. Thus most of the companies are concentrating on developing mobile apps. Where there is development there is testing too. Mr. Avinash, Test Architect, L&T InfoTech spoke about the approach for this kind of Testing and also about the wide opportunities which are up for grabs within the mobile application testing domain.

Globalization Testing: “SimShip” as we know, is the term used to specify simultaneous releases of a particular product in different languages. Mr.Anuj, Test Manager, Citrix R&D gave a session on this topic. As we had recently done a project in localisation testing, for Norton, were able to co-relate most of the things with his talk. During the session, he mainly told us about the challenges faced in the globalisation of testing and how to overcome such challenges. Hope his words helped us to set up much more clear approach in Localisation testing.

Cloud Testing: The major cost investment before the test execution process is, setting up the test environment based on the requirements, either at the client’s side, at a developer’s side or within the testing team’s side itself. So if you raise a question asking how we can reduce these costs, the answer is “Cloud Testing”; which is pay per use for the virtual testing environment. The speaker Mr.Mritunjaya, Project Manager, Mphasis spoke about how we can make use of the cloud testing approach in order to reduce the cost in setting up the test environment.

Open Source in Performance Testing: It is established that the commercial tools available for the performance testing are costlier. So if you are thinking of ways to bring open source tools for performance testing then your idea is positive and plausible. The speaker Mr.Ramanath Test Director, Mindtree Spoke about how they made used of an open source tool Called “Jmeter” with some additional plugin tools to achieve success in performance testing.

Security Testing: As technology grows, more and more transactions are done digitally. So the first question while performing any digital transaction is, how secure is my transaction? The assurance of security for the customer transaction is given by a Security Testing team. Speaker, Mr.Chandrashekar Umapathy, Manager Symphony conducted a brainstorming session on the application of security testing. In his words, he primarily spoke about this testing approach and also about how to overcome challenges faced during security testing.

Mobile Protocol Testing: Speaker Mr.Prabha Aithal, CTO, CanvasM spoke about the new 4th generation technologies like LTE, Wi-Max and how the Protocol Testing approach can be adopted for these technologies. Like numerous others, this testing specialization too has a wide range of opportunities within the Telecom domain.

As a team, we were happy in being a part of the conference. We had a lot of fun, shared knowledge with other professionals and returned back with a mind, set to apply those tips and tricks in our day to day professional lives..

Article by: Mahesh Manjunath,
mahesh.manjunath@tcl-asia.com

Test Managers – Balance of Opinions

2008-08-21T00:30:00.000+05:30

I am sure that most of the readers of the blog will instantly know what I mean when I say that different Test Managers arrive at different decisions. There can be many reasons for this, some are harder than others, some are nicer than others, some come from one country and some from another, some are permanent, some are supplied and some are contract, some are new to role, some are new to the organisation.

The reality is that people are different and our experiences drive us to behave in certain ways. To some extent those that know us well will predict our reactions and actions. When running a department full of Test Managers and having a mix of personalities and capabilities, it becomes important to bring some balance or levelling to this. You don’t want Test Managers to be sought because the project will always go live.

This means that we need to turn some of those decision making activities into more of a science and remove the imbalance that can be introduced by emotion or experience. I can’t really supply you with facts and figures to enable you to draw a decision diagram, because it is not that easy. But I will try and point you in what I think is the right general direction.

I would start by breaking down the departments testing functions into business domains. It is more likely that you can assess what is being released in terms of quality at domain level as a starting point. For instance, when releasing to the web you experience a high degree of success and few calls for support, yet when compared to releasing in Business Intelligence, the quality is always an issue and the first run of the month end batch processes always crashes. This gives an indication of where to start defining some criteria. But there are other areas to be considered as well. It could be that one development agency has a far higher success rate than another, where the volume of defects found are always low, or perhaps when a development agency or department develop using a particular language, the results are always much better than others.

These different criteria can be used to form some basic rules around final exit criteria for instance, enabling decisions to be made within a set range, ensuring that all Test Managers reach the same conclusions. Perhaps looking at the ratio of defects to scripts and comparing this to some analysis on the application being developed, the language of development, volume of calls pertaining to the application in support and the domain of release would provide good statistical evidence for decision making criteria.

Basically I am suggesting that you use historical data to form basic guidelines that the Test Managers can use. It does not necessarily eradicate the problem, but at least it will enable easy identification of projects that are outside of the norm. The Test Manager must still have the power to state what they think, even if this flies in the face of the statistical evidence, but when doing so they should be able to substantiate their view.

For projects running with inexperienced Test Managers or those that are inclined to shy away from awkward decisions, the decision becomes one of science which in most cases is going to be correct.

Empowered Test Managers

2008-08-19T00:30:00.003+05:30

The test function plays an important part within an organisation, yet too often they are asked to work in a manner which is restrictive and fails to give them the authority or accountability that they require or deserve. Strangely in some instances, even when they are given the authority they fail to take it.

So let’s look at the Test Completion Report. This is the opportunity for the Test Team to give a full account of what has occurred during the testing and make any recommendations. Why is it then, that we do not include as a standard within every report, a recommendation on whether the item is suitable for release into production. Part of the answer may be that the Test Manager does not feel comfortable stepping away from the black and white of scripts passing and failing and the associated defects would appear to make some uncomfortable. But the Test Manager knows the testing which was due to occur, understands how much has occurred and what level of quality has been achieved. Why then is it felt that this is not a logical and calculated recommendation.

On the flip side, why are Application Support departments not demanding proof of testing and a solid opinion on the level of quality that has been achieved? In most organisations, if a poor quality product is released to production, it is not the members of the project that are impacted by the problems, but those in Application Support. Not only are they going to be receiving calls through to the Service Desk, they are potentially going to be calling out support staff in the middle of the night to recover from the problems. Not only this, but any service levels that the Application Support department are supposed to maintain are potentially being compromised. It is therefore imperative that these two departments work together. It is often the case that Application Support will have more clout within the organisation and will therefore be able to assist in the move to a more authoritative testing department.

Another area that the Test Manager should be given more authority is on the exit of Integration in the Small and the entry of Integration in the Large. Two important events occur here. The first is the handover of the development agency’s testing assets and proof. A failure to provide this information is a red flag to the Test Manager and should sound all sorts of alarm bells. This is indicative of a lot of problems about to be experienced during test execution. The second is when the code is first delivered into testing and the process of running those first few scripts. If these start to fail then there is a second indication of problems with quality. Yet so often the Test Manager is not in a position of authority to insist on the production of test assets from development, or indeed to bounce the code back for rectification of defects. If the Test Manager is engaged at the outset of the project, they should be able to insist on the supply of test assets from development. To avoid the problems of poor quality code delivered, take smaller drops of code and start them sooner, so that an indication of quality is understood, or add a specialist phase of testing prior to Integration in the Large, where the sole purpose is the sign off of the code quality delivered, potentially by smoke/sanity testing the application using a selection of the scripts created.

To summarize, ensure that the opinion of the Test Manager is sought. Don’t let them get away with a statement of scripts run and defects raised, open and closed in the Test Completion Report. Insist that they provide an opinion. Work with Application Support to bring additional credibility to the department. Once this has been achieved you can then start to think about how you ensure that all of the Test Managers in a department apply the same level of reasoning when making recommendations for go live. The subject of another article I think.

T.I.G.E.R – What this acronym means to us!

2008-08-17T00:30:00.001+05:30

Transition Consulting Limited, as a group of companies, have a set of values that we expect all of our employees to demonstrate. These are embodied by the T.I.G.E.R acronym.

T = Truthful
It is imperative to us that all members of the company operate in a truthful manner. We need to know that we can rely on what people are saying and that they will be honest with each other and our clients. This is not always easy, but some wise person once said, “Honesty is the best policy”. Well we believe this and have made it part of our policy.

I = Independent
Testing as a discipline needs to remain independent of other functions in order that it can provide an unbiased view on quality. Lack of independence places a testing department under the same constraints in terms of time and cost and therefore quality can become compromised. We pride ourselves on the fact that Testing is all that we do. We live and breathe the subject and can always be relied upon to act independently.

G= Good Willed
We expect our staff to be good willed because this is a personality trait that we embody as an organisation. As a result we are affiliated with several organisations and as a group contribute charitably each year. We work with some local organisations and some as large as the NSPCC.

E = Energetic
Energy is incredibly important to us. We want our employees to work hard and play hard. We expect them to be passionate about testing and what it involves. We expect them to demonstrate an enthusiasm for their work and not view it as just a job.

R = Reliable
We need to be able to rely on our resources and we expect our clients to rely on us also. Reliability is a cornerstone on which we build, taking care of the basics by ensuring that we can be counted upon to be knowledgeable and dependable, providing value into our organisation and those of our clients.

Not only is the acronym easy to remember, but it is a strong image and one with which all of our employees are happy to associate. From a TCL India perspective, the Tiger is probably even more powerful an image, being so highly revered.

SOFTWARE TESTING GLOSSARY (Update)

2008-08-15T00:30:00.006+05:30

This glossary is a living post – so will be edited as we come across terminology that is not included. If you have any suggestions or disagree with an explanation, drop us an e-mail and let us know. TCL India offer this as a means of establishing glossaries of your own or as a point of reference. This update includes 10 new definitions.

A…………………………………………………………………………………………………

Accessibility: (Testing)Looks at an application to ensure that those with disabilities will be able to use the application as an able individual.

Agile: A development method, involving the creation of software, with the contributing parties, including testing, all working on the same item at the same time.

Alpha: (Testing)Testing of an application which is in production, but not available for general use. This is normally performed by users internal to the business.

Analyst: Person who interacts with the business in order to understand and document their requirements for an application or change to an existing one.

Analyst (Test): Person responsible for the preparation and execution of test scripts, recording and progressing of defects, reporting into the Test Team Leader or Test Manager.

Automation: The process of developing a script or program which can be run by software rather than a Test Engineer, to test an aspect of an application. Automation is used to increase the speed of test execution.

Automation Centre: A tool used in the automation of software testing. Seehttps://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-127-24%5E1074_4000_100__&jumpid=reg_R1002_USEN

B……………………………………………………………………………………………………

Beta: (Testing)Testing of an application which is in production, but not available for general use. This is normally performed by a select group of friendly external testers.

Black Box: The process of testing without understanding the internal workings of an application, relying on inputs and outputs only.

Bug: See DefectBusiness Requirement Specification: See Requirement Specification

C……………………………………………………………………………………………………

Case: See ScenarioCode: The software that has been produced by development and is being subjected to testing.

Completion Report: A document produced during the testing and presented to the project manager on completion of testing activity. The document should detail the tests that have been performed along with the associated results and defects. Some reports may include a recommendation for the applications suitability for release to production.

Component: The smallest item that is testable or producible. This often refers to a single file of code.

Configuration Management: The means of managing the components required to make a larger item. Applicable to files, documents, data, hardware, software, tools. Understanding of the versions of each component required in order to be able to re-build the larger item.

Criteria: See Entry Criteria and Exit Criteria

D……………………………………………………………………………………………………

Data: Information pertaining to the use of a system or recorded through the use of a system. Data is used in order to test a system, potentially after data cleansing if personal information is involved.

Data Generator: A tool used to generate high volumes of data in order to be able to test many permutations, or to load test an item.

Data Protection Act (DPA): The act that determines how personal data should be used and protected from abuse.

Data Scrambling: The process of altering personal information from data to be used for testing purposes.

Defect: Where the item under test has been found to inaccurate, resulting from testing. Primarily used in associated with software, but equally valid for static testing of documentation.

Defect Detection: The means of identifying a defect. Can be a metric used to predict the volume of defects expected during the course of a project, or as a means of looking back at a project to understand where testing needs to be concentrated in future projects of a similar nature.

Defect Removal Efficiency: A metric used to assess the ability of testing to remove defects as they are introduced, during the software development life-cycle, keeping the cost of testing later phases to a minimum.

Defect Turnaround: The time taken from the identification of a defect, through to the point of resolution. Different levels of granularity may be used. e.g. A focus on the time taken by development.

Developer: A person responsible for the development of an application.

Development: A process of producing an application by production of low level design, code, unit testing and integration in the small testing.

Dynamic: Testing which occurs on the right hand side of the V-model, with the application present.

E……………………………………………………………………………………………………

Environment: The combination of hardware, software and data as used in development, testing and production. The platform/s upon which the testing occurs.

Entry Criteria: The criteria that must be met prior to a deliverable entering the next phase of testing to another. This is normally associated with documented test assets and pre-agreed volumes of defects.

Error: See Defect

Exit Criteria: The criteria that must be met prior to a deliverable leaving the current phase of testing. This is normally associated with documented test assets and pre-agreed volumes of defects.

Execution: The process of working through a script on the application under test, in the testing environment.

F……………………………………………………………………………………………………

Functional: (Testing) The testing of a products function, against requirements and design.

Functional Specification: A document which extracts all of the functional requirements from the requirement specification.

G……………………………………………………………………………………………………

Glass Box: See White Box

Grey Box: Testing performed by testers with some knowledge of the internal workings of an application. See also Black Box and White Box testing.

H……………………………………………………………………………………………………

High Level Design: A design showing the major components and how these will interface to each other, defining the hardware to be used and the software that will be impacted.

I……………………………………………………………………………………………………

Integration in the Large: Where the application or applications that have been developed are brought together along with those which have remained unchanged, building a production like system around the application/s. Testing is then applied looking at the communication between the different applications.

Integration in the Small: Where the components of the application that have been developed are brought together along with those which have remained unchanged, building the application or major component of a single application. Testing is then applied looking at the communication between the different components.Integration: The act of bringing many parts together to form a whole.

ISEB: Information Systems Examination Board. This was historically the board that was used to certify test professionals at either Foundation (Entry Level) or Practitioner Level (Advanced). See:http://www.bcs.org/server.php?show=nav.5732

ISTQB: International Software Testing Qualifications Board. See:http://www.istqb.org/

J…………………………………………………………………………………………………….
K……………………………………………………………………………………………………

Key Performance Indicator: A mechanism built on one or metrics, which determines a band of acceptable performance, which over time is often targeted towards improvement.

L……………………………………………………………………………………………………

Live: See Production

Load: One of the types of performance testing, this looks at testing for the maximum load on the system.

Load Runner: Tool used to performance test one or many applications, to understand how it handles increases in load. See:https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-15-17%5e8_4000_100__

Low Level Design: Definition of exactly how the application/s will be modified or produced in order to meet the requirements and the high level design. This can extend in some examples to elements of pseudo code being defined.

M……………………………………………………………………………………………………

Metric: A measure of an attribute or occurrence in connection with an organisation, department or functions performance.

N……………………………………………………………………………………………………..

Non-functional: How an application performs, rather than how it does it.

Non-functional Specification: A document which details the non-functional requirements such as performance, security, operational elements.

O……………………………………………………………………………………………………
P……………………………………………………………………………………………………

Performance Centre: A tool used for measuring the performance of an application or series of applications. See:https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-126-17_4000_100__

Performance: Used to describe many types of testing relating to the speed of an application. See Volume, Load and Stress.

Plan: A document produced for a type of testing defining how the testing will be performed, when the testing will be performed and who will be performing it. The test plan lists the test scenarios that will be scripted.

Preparation: The process of generating the test scripts.

Prince2: A project management methodology that is used amongst a lot of blue chip organizations to bring discipline to the software development life cycle.

Priority: The importance of fixing a defect from a business perspective. Defined by business representatives.

Production: The area or a computer network that contains applications which are in use by real users and contains real data. The area that applications are released to on completion of a project.

Q……………………………………………………………………………………………………

Quality: The suitability of an item for its intended purpose and how well it achieves this.

Quality Centre: Tool used to assist with the management of testing, recording and tracking scripts, logging and tracking defects and more. See:https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-127-24_4000_100__

R……………………………………………………………………………………………………

Regression: Retesting of a previously tested program following modification to ensure that faults have not been introduced or uncovered as a result of the changes made.

Requirement Specification: A document normally produced by a business analyst, capturing the needs of the individual in a manner which means that they can be translated into a software solution.

Re-Test: Taking a defect which has failed and executing the associated test script again.

S……………………………………………………………………………………………………

Safety Critical: Used to identify something for which the use has an impact on personal safety, such as medicinal applications or those used by rescue services.

Scenario: A high level description of how a requirement is going to be tested.

Script: Referable back to the scenario, the script defines each step that must be passed through in order to perform a test, along with the expected results. As the script is executed, results are recorded and if they match the expected result are marked as passed, otherwise as failed. A script containing a failure should have a resultant defect raised.

Schedule: A document, similar to a project plan, but detailing the activities associated with testing, when they are due to occur and who will be performing them.

Severity: The importance of a defect to testing and the application. Defined by Testers

Special Interest Group In Software Testing (SIGIST): A group that has been set up by the British Computer Society, made up of people interested in the subject of software testing.

Smoke: A process of proving an application or environment is ready to commence full testing, by running a sample set of scripts testing the primary functionality and/or connectivity.

Software Development Life Cycle (SDLC): The process of taking a requirement and translating it into a fully working application.

Static: The process of testing without the presence of software. Normally refers to the testing of documentation.

Strategy: A document produced at project or programme level, defining what testing is to be performed.

Stress: A form of performance testing, whereby the volume of testing is increased to a point at which the application is deemed to be failing to perform, either due to failure to meet performance criteria or system breakdown.

T……………………………………………………………………………………………………

Technical Architecture Document: Definition of how the business requirements should be translated into a technical solution at the highest level.

Test Bed: A cumulative term used for all of the test environments used by the test department.

Test Director: A tool used for test management, capture of requirements, scripts and defects, with the ability to manage defects through to resolution. Now replaced by Quality Centre.

Test Maturity Model (TMM): A means of measuring a testing department’s level of maturity in a similar manner to that of the capability maturity model (CMM).

Testing: The process of reviewing an item for accuracy and its ability to complete the desired objective.

U……………………………………………………………………………………………………

Unit: The testing of the application by the developers at a component or modular level, looking at whether the code is performing as expected.

User Acceptance: The means of testing applied, normally by members of the business or recipients of the system, whilst still in the test environment. This looks to ensure that business requirements have been understood by all involved in the application production as well as providing an opportunity to check the business processes that the application will be used in conjunction with.

V……………………………………………………………………………………………………

V-Model: A testing focussed software development life-cycle where the process of creativity occurs down the left side of the V and the process of testing up the right side of the V, where each step down the V has a corresponding point of testing up the right of the V. Begins with a requirement and terminates with User Acceptance Testing.

Version: An alpha numeric means of identifying a particular build of software, or a component thereof. The version changes incrementally to reflect each modification of the software, component, document etc.

Volume: (Testing) A type of performance testing which increases the volume of users on a system, in each cycle, taking the volume up to a prescribed limit, normally exceeding optimum load.

W……………………………………………………………………………………………………

W3C: World Wide Web Consortium – body creating web standards and guidelines. http://www.w3.org/

WAI: Web Accessibility Initiative. See: http://www.w3.org/WAI/

Waterfall: Project Management Method whereby each element of the software development life-cycle occurs sequentially.

Win Runner: A tool that was historically used for test automation. See Automation Centre.

White Box: Testing normally performed by developers, looking at the code and ensuring that each model of code is performing as expected. See Unit

X……………………………………………………………………………………………………

Y…………………………………………………………………………………………………….
Z…………………………………………………………………………………………………….

The Very First Bug

2008-08-13T00:30:00.002+05:30

I saw this a while ago on Wikipedia and included it in our company journal. I was amazed at how many people had not heard the story. I am posting it here now in recognition of a similar article on "help i found a bug in mycode" where he draws comparison to a more modern day version.

The similarity is strange, because only recently I had been recounting to colleagues an experience with my son (10 years old). A storm fly, one of those tiny little things about 1mm by 2mm had some how crept into my LCD screen on my monitor and to all appearances, was crawling all over my document. I called my son over remarking, “Come take a look at this bug in my screen”. His response was naïve but endearing as he responded, “Wow, Dad is that a virus?” much to my delight.
The picture above relates to the following text. “While Grace Hopper was working on the Harvard Mark II Computer at Harvard University, her associates discovered a moth stuck in a relay and thereby impeding operation, whereupon she remarked that they were "debugging" the system. Though the term computer bug cannot be definitively attributed to Admiral Hopper, she did bring the term into popularity. The remains of the moth can be found in the group's log book at the Smithsonian Institution's National Museum of American History in Washington, D.C.”

How to Annoy a Test Manager

2008-08-11T00:30:00.004+05:30

A little light relief from all the serious blogging, but some of these do really get me angry.

The coding has finished and we need to test – should we be talking to you now?

I know you were expecting the code today, but when do you really need it?

Can you please stop raising defects as we don’t have time to fix that many problems?

We have only got half the time we thought we had for testing so how are you going to get all of your work done?

We know the quality is bad, but we are releasing anyway?

UAT has been dropped because the business can’t free any resources.

The fact that this problem was fixed in an earlier release is not indicative of poor configuration management.

Why do we need developers, support and environment resources to work the weekend as well as testers?

The environment is nearly the same as live, we just don’t know how nearly!

Testing is supposed to make the software perfect. Why hasn’t it worked?

We have spent 20% of our budget on analysis, 60% on design and development and 10% on other items. You have 10% left, what can you do for us?

We are running out of money because the test execution is taking too long, so we are going to let the testers go and the developers can finish the job.

Don’t worry about getting testers. We have lots of people in the business that we can get to test at lunchtimes.

The supplier was not asked to perform Unit testing and we can’t afford to have it done now as it was not in the budget.

Testing is a service and as such we will let you know when we need you.

I know you asked for three environments, but we can only get one. Don’t worry. I am sure everyone can get on it at the same time.

The testing is occurring on the development environment. That way the developers can change things whenever they need to.

The problems in live are fixed out of hours so we can’t test them.

If we automate all of the testing how much time will it save us?

Why do we need testers on board before the testing starts?

The user spilt coffee on their system and it crashed. I bet you didn’t test for that did you?

We have too many high severity defects, so we are going to hold a meeting and see how many we can downgrade?

We need to do some performance testing. There are no non-functional requirements, so let’s just assume that everything must happen in under a second.

Testing is easy. Anyone can do it.

It’s not a bug. It’s a design feature.

The requirements were captured at the start of the project. But don’t read that it’s way out of date.

Please feel free to add your own by way of comments.

Test Process Structure

2008-08-09T00:30:00.001+05:30

Different organisations will have different needs in terms of the levels of test process that are required and it is important that the correct levels are applied. The Testing Maturity Model is a good source of information for this and can be located at http://www.tmmifoundation.org/

TMM suggests that as organisations mature in line with the Capability Maturity Model, the same is occurring within testing. TMM was introduced to apply CMM type measures to an organisations testing capability. In brief there are 5 stages of test maturity, beginning at level 1 where testing is undisciplined, level 2 where the testing is managed but still seen as a phase of the SDLC (Software Development Life Cycle) that occurs after testing, although at this point process is being put in place on a programme or organisation wide basis. Level 3 sees testing fully integrated in the SDLC and no longer a single phase, with processes become more mature and honed over time. Level 4 sees the testing fully defined, established and measurable and level 5 looks at optimisation of testing efforts, processes and practices where costs are controlled and continuous improvement occurs.

What I would like to suggest is that TMM can be seen as a trifle heavy for some organisations and unless already pursuing CMM may seem quite foreign. I am not suggesting that it does not have a place, but that it focuses entirely on process and not on understanding the drivers of the business itself, which may make it impossible to move up through the maturity levels.

I think that from a more simplistic approach, organisations will follow something more as follows:

Stage 1
Testing is being performed by available resources that are not professional testers. Defects are being recorded in basic form, using a tool like Excel. Some test cases or scripts may be produced.

Stage 2
Testing is being performed by test professionals and they are called in after development has occurred. Test Plans and Scripts are being produced and defects are being logged formally and reported on.

Stage 3
Testing exists as one or more independent functions, each having it’s own process. These small test teams are aligned to the business areas they serve. Projects now have approaches or strategies produced against them.

Stage 4
The disparate teams have now been pooled to form one function which is independent from the development side of the business. A Test Policy has been produced and testing is involved in all aspects of the software development life-cycle. Metrics and KPIs are being gathered and fed back to the organisation.

Stage 5
Testing is improving in it’s capability. The quality of releases to production is high and end users have a high opinion of the IT function. Releases to live occur only with approval from the testing function and testing is viewed as a discipline within the organisation. Metrics and KPIs are now giving the business sufficient information to base decisions on preferred development agencies and technologies, demonstrating those which are most successful.

The key is to ensure that your business is improved and enabled by testing and not crippled by a level of process which does not match that of the rest of the organisation. Learn to walk before you run and build over time.

Role of the Environment Manager

2008-08-06T18:18:00.005+05:30

One area that I have regularly come across which needs further clarification is the role of the Test Environment Manager. I have produced this article for those people looking to understand what exactly should be expected of an Environment Manager and could be used as the basis for a job description or role profile.

Test Environments are required in order to enable the test execution to occur. The Test Environment Manager is responsible for the provision of the test environment, its configuration status, its stability and maintenance. They also provide administrative activities such as backups and purging on request of the project, monitor usage of environments, supplying support throughout the test execution, including out of hours working if required. They will be expected to manage change to the test environments and keep the programme and testing informed of status. They must be able to give advance warning of downtime of any component of the test environment, which may cover multiple platforms. Duties are predominantly based around the organisation’s infrastructure but can extend to cover interfacing with 3rd party organisations which are offering equipment to form part of the project environment.

The environment manager is responsible for ensuring the provision of suitable test environments for all aspects of the testing of the core system. A suitable test environment is one that meets the requirements for the phase of testing and those requirements will be documented and provided to the Environment Manager by the Test Manager.

They shall ensure that the interfacing systems test environments are available and of a known and controlled configuration. For the core system, the environment should be available two weeks prior to test execution commencement, so giving time for any difficulties to be resolved. The environment must remain available for the duration of the project. As many peripheral systems should be delivered on similar timescales.

Thought must be given to the possibility of different environments being required, where phases of testing overlap, or where the requirements of the testing phase are such that the provided environment needs to be changed or upgraded. Each time a new environment is made available, this should also occur two weeks prior to test execution occurring in the new environment. The Environments will be placed under change control and will always be of a known state of configuration.

No upgrade of hardware or software may occur to a test environment without the approval of the Environment Manager. The Environment Manager must place change control over the environments and maintain that level of control. This may be a formalised mechanism, but as a minimum, requires that the Environment Managers approval has been received prior to the change occurring. All changes must be communicated to the following people as a minimum:

1. Development Manager
2. Testing Manager
3. UAT Manager
4. Project Manager

It is expected that this list will grow as time passes and the individual users of the system become known. It is felt that the above will form a minimum conduit for such communications until such time as all individuals are known.

A record of the configuration should be maintained for each environment or element of an environment if multiple environments are required in order to make a whole. The record should detail the hardware build that has been provided by Technical Operations, including versions of operating systems and other utilities provided as part of the basic provision. Over and above this, each item of software placed onto the environment for test, shall have its version recorded, so enabling a clear picture of the configuration at any given time. Any change to any aspect of the configuration should be approved under the change control mechanism and logged in the Configuration Record.

Should the software under test be configurable, a document with its own version control shall be supplied to the Environment Manager so that they can use this as a state of software configuration. If changes are required to that configuration, these should be recorded in an upgraded version of the document and the Configuration Record updated to reflect that this has happened.

Ad hoc changes to the configuration, be that hardware, software or software configuration, are not permissible. Alteration to the configuration must be controlled and the configuration must always be maintained in a known and managed state.

The Environment manager, will ensure that there is support for the system during all times of test execution, be this late at night or over weekends and bank holidays. They will make known a list of all the different elements of the environments and the contact names and numbers of the people that they support.

A plan shall be maintained by the environment manager of who is using what environment for what purpose at what time. This needs to be maintained so as to ensure that changes in environment usage are managed and controlled and so that if multiple environments are required, this can be kept to a minimum by the intelligent employment of the environment resources available.

The Environment Manager, as a result of the change control mechanism, will be able to notify the users of the test environments of any outages. These changes shall be recorded in an outage log, so that upon project completion, the details of outages can be formally reported. It will also enable monitoring of any problems that may occur in this area and be used to aid resolution of repetitive issues.

As part of the standard weekly output, the environment manager will distribute the following information.

1. Planned usage for the week.
2. Planned outages for the week.
3. Planned changes for the week.
4. The statement of configuration for each environment.
5. Planned Backups

The last item on the list is backups. These shall be available to all testers upon request and shall be recorded and requested by the Environment Manager.

I hope that this is found to be useful and informative.

Importance of Accurate Test Environments

2008-08-02T03:45:00.005+05:30

Look - my new test bed. What a brilliant image! How many of us go here for our next test environment.

Do people realise the importance of the environment to the tester? I don’t think they can. I have spent most of my testing life working with make-do arrangements and being expected to test to high standards. I had a conversation with a fellow tester today, who is currently working as an Environment Manager and he and I were both moaning about the problem.

So what happens? Firstly all of the investment on a project is in the production equipment. As the project is creating application X for the first time, the to-be production environment can be used for testing purposes. It is ideal in this situation because it means that we can test on the most suitable infrastructure possible – what it will run on in live.

The problem comes the next time the application is updated. There was no investment in test infrastructure as part of the original project – they didn’t need it. So now the update project has to justify a massive expenditure in test equipment, which could potentially jeopardise the projects financial benefit. This step is therefore avoided and the hunt begins. We need to find a piece of kit that can be used to test the application. Something which is similar to that in production would do. Alright something which resembles the production kit, if you squint at it whilst wearing dark glasses. The good news comes through. Kit has been found and once the rust is scraped off, it is found to be an earlier version of the kit in production. The RAM is reduced, the processor is 2 years earlier than that which is in production and the operating system has fallen out of support. Never mind, it is all we have. Then the test manager explains that he needs three environments to be able to complete the testing on schedule. Don’t worry we can whack in some more RAM and increase the disk space. It’s still not half as good as production but we can split it into three for you.

This is seen as a pragmatic solution by all involved and the test manager must now do his best. (Agggghhhhh!!!!!!!). Let me explain my frustration. As soon as you compromise the test environment, you compromise the testing. So many times defects are raised which are directly attributable to the environment and if that environment is not the same as production, time is going to be spent resolving defects which will not exist in production and potentially defects which will occur due to the production environment will not be located during testing. Certainly in terms of release testing, the process is likely to be different and effectively the results during testing are false and cannot be relied upon. The lack of adequate test environments becomes a project risk in its’ own right and should be logged in the project risk register accordingly.

It is critical that the testing environment is as close to production as possible and that any deviations are fully understood. Only with the differences between production and testing environments being known and measurable can you rely on the testing results.

Take performance testing. If you are running with less RAM, less disk space, less process power, how will this manifest itself in the results? As for security testing, the operating system version becomes paramount as the level of inherent security is likely to change. Know and document the differences and if this cannot be achieved be careful about signing off the product as fit for release.

As a recommendation to all project managers, ensure that you leave an inheritance of test infrastructure for the following projects. Placing a new piece of equipment live is not sufficient. Consideration must be given to ongoing support, testing, development and training and what environments are going to be used for these.

Grading Exit Criteria

2008-08-02T01:51:00.006+05:30

For the sake of this article, let us assume that we have 5 different grades of defect that start with insignificant going on to, minor, significant, major and ending in critical. When we come to exit one phase of a project and enter the next, we need to recognise that criteria will be set by the test manager, which need to be achieved. It is normal that the exit criteria from one phase, becomes the entry criteria for the next phase. For instance, a Test Manager for “Integration in the Large” testing will set his entry criteria, which by default becomes the exit criteria for the phase before. This remains true up to the point of release to live.

What I would like to draw attention to, are two key points. The test manager that says I am not going to have any defects remaining open at the point in time I go live, is living in “cloud cuckoo land”. This is the test manager that wants to be able to test forever and release perfect code. This does not happen. The reality of business is that certain levels of risk become acceptable in order to deliver the project to live and meet the requirements of the business and deliver the business benefits that were used to justify the project in the first instance.

So the next question becomes one of levels of risk and what is likely to be acceptable. This will depend on the nature of the project. One that involves safety critical applications is likely to have a far higher level requirement for quality, than one which is being used by 2 people in a small business. One which is exposed to the public or clients is also going to need to be of a significantly higher level of quality than one which is used internally only. The skill of the test manager is to assess the application, its’ use and define the level of defects that are going to be tolerable. As previously defined in http://tcl-india.blogspot.com/2008/06/entry-and-exit-criteria.html the Project Manager should be bought into these levels in order to ensure support later in the project.

Now we need to look at the defects that are going to be acceptable. A common misconception is to identify that the level of critical defects is 0, be that priority or severity, but that there is no restraint on the level of insignificant defects. This could not be further from the truth. Whilst 1 insignificant defect is not going to stop the release, there comes a volume of insignificant defects that makes the release of the application unacceptable.

I would suggest that we concentrate initially on the severity of defects. We need to understand proportionately that a volume of insignificant is equal to 1 minor, a volume of minor is equal to 1 significant and so on. This will change dependent on the application use, so here are some suggestions for different application types.

Safety Critical
Proportions: 1 Critical = 2 Severe : 1 Severe = 3 Significant : 1 Significant = 5 Minor : 1 Minor = 10 Insignificant
Final Exit Criteria: 0 Critical : 0 Severe : 0 Significant : 5 Minor : 10 Insignificant

Public/Client Facing
Proportions: 1 Critical = 3 Severe : 1 Severe = 5 Significant : 1 Significant = 10 Minor : 1 Minor = 20 Insignificant
Final Exit Criteria: 0 Critical : 0 Severe : 3 Significant : 10 Minor : 20 Insignificant

Internal Consumption (20 + Users)
Proportions: 1 Critical = 4 Severe : 1 Severe = 7 Significant : 1 Significant = 15 Minor : 1 Minor = 50 Insignificant
Final Exit Criteria: 0 Critical : 1 Severe : 5 Significant : 10 Minor : 40 Insignificant

Internal Consumption (0 to 20 Users)
Proportions: 1 Critical = 5 Severe : 1 Severe = 10 Significant : 1 Significant = 20 Minor : 1 Minor = 100 Insignificant
Final Exit Criteria: 0 Critical : 2 Severe : 5 Significant : 10 Minor : 50 Insignificant

Please bear in mind that these are indicative and that the best solution is for the test manager and other members of the project team to determine the levels between them.

Testing in Live – Why you should not!

2008-08-02T01:44:00.005+05:30

As a tester I am often asked why we can’t test in live. There are so many positives that can be gained from it that it surely makes sense. After all, the environment is exactly the same as live, the data is real and the volume of users and traffic is real.

Production is designed to serve a business in operation. It is being used in anger every minute of the day. It may be internally used, it may be used by high street shops or it may be open to public use such as the internet.

The first problem is that the code that needs to be tested is not fit for live because its quality is not known. So by placing the code live you are compromising the production platform. You can cause code which is running perfectly in production to fail, as common routines are modified.

Testing is designed to break things. We need to understand the points of vulnerability and testing tries to identify these. This means that the application could be caused to crash, which could result in a domino effect and take out other elements of infrastructure or code.

In testing terms, the point at which resolving a defect is highest, is in production. One of the primary reasons for this is that outages which affect the production platform can be disastrous. Imagine 1000 customer services personnel being unable to work for an hour. A hourly pay rate of £10.00 per hour and the problem has already cost £10,000.00 But what if you have an on-line presence and you lose that for an hour. You have now lost sales revenue for that period of time and this you cannot recover. Perhaps more damaging is that perception of your organisation has taken some public damage. The potential new customer went somewhere else and you have now lost their trade for the foreseeable future. The existing client who was returning has now been forced elsewhere.

Think also of the press that can be received through public outages. Organisations that experience loss of service will often find this being reported in the media and this causes immeasurable brand damage.

Another consideration is that the data which exists on your production system is real. The testing therefore cannot modify, add or delete information without careful consideration of the consequences. Corruption of data exposes an organisation to the data protection act, but worse, may invalidate information which is crucial to your business. What happens if a change of address occurs which is a test – the client is potentially lost along with their business?

A final point is that your system is in all likelihood linked to other organisations. When I first became involved in development, I had a story relayed where a developer had unit tested his code in a live environment. His code was a credit checking piece of software and he used his own details each time he ran the test. As your credit score is negatively impacted each time a check is performed, the developer managed to take his credit rating from positive to negative in a matter of a day. He did not realise he had done this until such time has he tried to make a credit purchase and was refused. Fortunately, after some discussion with the credit reference agency, his rating was restored. But any kind of financial transaction must be performed in a test environment, linking to other test systems. Imagine the damage that re-running a BACS transaction could cause, or payroll being sent out at the wrong time.

Production is very much live. You would not expect a doctor to test a new drug on a human without going through a certain degree of process first. Even then you would move into a clinical trial under strict process.

What about Beta testing I hear you ask. Yes, there are examples when the software is deemed to have no negative impact on production, where sufficient testing has already been performed as to be confident of its capabilities, that the software may be released with a “Health Warning” attached. It may be that a select group of users will be asked to perform a trial. But in these instances, the application will have been put through its’ paces in a test environment.

It is important that users of systems have good experiences. Failure to achieve this results in loss of the users from the system, negative impact to the reputation of the department and individuals making the release, plus the potential to damage the organisation's brand and revenue-earning potential should make this a course of action that is avoided.

Reinventing the Testing Wheel (Object Oriented Testing)

2008-07-29T21:32:00.006+05:30

I have been thinking about this one for some time and wondered what other views existed. For some time developers have taken a more modular approach to programming, applying techniques such as Object Oriented Design, in order to improve efficiency. Why re-invent the wheel each time a certain piece of code is required? Take something like a calculation of VAT. Once this has been written the first time, that should be enough and if stored and indexed correctly, each time someone else needs a VAT calculation they call a pre-written routine from a library. Not only does this mean that the developer does not spend valuable time writing the same code again, but also that the module should already have been tested and therefore be reliable, reducing the likelihood of errors.

This was seen as a major break through in the development world, but one which has not been replicated to my knowledge in the testing arena. When we write a manual test script, we write it in a manner which is long hand. So how many times have we written scripts to test VAT calculations? Why are we not creating a library of scripts that are re-usable by all testers? Why do we have to sit and think each time that we test a field: to understand what all the permutations are that we need to deal with, how to test it positively and negatively? There are hundreds of thousands of testers all over the world that are repeating the same thought processes, writing the same scripts, duplicating their efforts.

How many times has an input of a person's age been tested? What are all of the parameters? How old before they are at school, how old before they can hold a credit card, bank account, retire? All of these questions have been thought about so often, but we still religiously recreate them each time. So testers are accused of being too slow or not moving with the times, being overly expensive. Well I suppose when you look at something like this, the answer has to be, "Yes".

So when we are asked what the future may hold for testers, I put on my rose-tinted spectacles and imagine…..

I imagine a world where we are sharing our knowledge and expertise, generating a global testing community and capability. One where we are seen as slick operators, contributing massively to technology programmes the world over, with the minimum of input for the maximum return.

How does this happen? If there were a central repository for scripts to which every tester contributed, where we went as a first port of call to locate our script components, or store those components we have to create, we would quickly generate a wealth of stored expertise. A Wikipedia of test scripts – wow! We would need to be able to locate scripts in a fast and efficient manner and the database used to store the script components or objects would need to be referenced in a multitude of ways. But this is not new technology. Developers have been doing a lot of this for years and we can no doubt learn from the mistakes that they made along the way.

I would welcome comment on this. Is it feasible? How could it work? Am I barking mad? If you are a developer and have a story about OOD then share it and let us know if you think this is feasible. Drop me a line and let me know.

Site Under Construction!

2008-07-19T23:21:00.002+05:30

In the current climate of the credit crunch and with so much positive press around the buoyancy of the internet, more organizations are looking to the World Wide Web to maintain or increase sales. What many are doing is realizing that their sites are perhaps not as good as they could be and are investing in updating or de-developing them. For others, the process of site maintenance is an ongoing activity with a dedicated team.

So with all of this development going on, it would be fair to assume that there must be a lot of testing as well? I would suggest that the answer is no. For some reason, when people are requesting development of web sites, it is believed that they are best placed to ensure that it is fit for purpose.

This is a bad assumption for several reasons:
1. Finding the time to thoroughly test the website is difficult to achieve. The people responsible for the new website are normally being asked to do this as one of many tasks.
2. The attention to detail required to check the entire site, is high and demands a certain type of individual. The tester has this mindset and is used to applying methodical and meticulous tests, whereas the business person may not be.
3. This may be the first time that the business person has been asked to test anything. Testers are used to testing websites, this is something that they do on a regular basis and this enables them to make better use of their time, homing in on problem areas.
4. Business resources normally check things positively, making sure that situations that are expected are dealt with. The tester introduces negative testing. Checking to understand that the site can handle abnormal activity, that which deviates from the norm.

So what is happening? Companies are employing development agencies and placing themselves in their care. They are relying on the development agency to ensure the quality of the site. The business are doing their best to make sure that the site is okay and are paying for development when they think it is finished.

In a normal IT project, the business relies on the testers to ensure that the application is to a certain standard before it comes anywhere near to them and this step is not being performed. Three problems result from a lack of in-built quality: The first is that the development agency is paid for the delivery, even though problems may not manifest until later. The second is that the development agency is held in low esteem once they have been found to have under-delivered. The third is that the organization requesting the website, find that they have a tool which does not meet their needs and therefore requires further investment to put the problems right.

The involvement of testing as part of the process ensures that the site is performing as was requested. It can be used as a means of safeguarding the payment, ensuring that the development agency have fully delivered prior to releasing funds. It also protects the development agency from losing their customer due to poor satisfaction.

The design is truly the prerogative of the business and only they can say if the look of the site is what they wanted to achieve. Other than this it should be placed in the hands of the experts. Remember that better quality, means customers are happier to use the site, more likely to find what you have to offer and therefore an increase in sales should result. I have said in many places that a poor web experience results in lost sales and that users are highly likely to leave your site for one offering a better experience or easier to find items.

Why is Testing Sexy

2008-07-11T17:27:00.004+05:30

I suppose the answer depends on the person and for some not at all. But I will endeavour to give you 12 reasons why I think Testing is “sexy”.

1. The world of testing is governed by process and is incredibly black and white. A test passes or it fails; there is no shade of gray.
2. It is like being part of a big family, where the same individuals are involved and you keep bumping into old friends and faces.
3. You have to keep your wits about you and be prepared to change tack quickly, effectively and efficiently.
4. Testing is the underdog of IT. It is trying to bring to people’s attention to the value that it brings to the software development life-cycle.
5. You have to be able to think around the developer, understand what they are trying to achieve and then look at how you can think around it.
6. You need to be able to look at a project as a project manager, analyst, designer, architect, developer and tester, giving you a well rounded grasp of IT.
7. There are opportunities to specialise in testing, by technology, by testing type or by process, giving endless opportunities to grow and learn.
8. The work is endless, because finding perfect code from perfect projects is nigh on impossible, keeping us all in employment.
9. We create tangible assets and benefits for those employing our services, leaving them in a stronger position.
10. Whilst sometime invisible to the end users, we make a massive contribution to their use of IT, without which life would be harder, more frustrating and the helpdesk would be inundated.
11. We are growing as a discipline become more recognised and with more accreditation available in universities across the world.
12. We get to break stuff on a very regular basis.

I hate the word “Sexy” being used to describe something like a subject, but it seems to be used regularly these days. Do I dream of curvy defects at night? ….No! But I am passionate about Testing. I do believe in the value that it brings and the value I can add as part of a project. More and more people are becoming involved and the testing community is growing. Less people are falling into Testing and are making it a career choice.

Ranking Your Web Success

2008-07-10T03:04:00.004+05:30

When talking to a wide group of web owners, it becomes apparent that whilst they are concerned about the quality of their website, they are equally interested in some of the other aspects of their web presence.

Different people have different ways of measuring a web sites success, but there are some which are universally accepted. I have yet to find a site that is not recognised by Google for instance. Google offer a means of looking at web sites in order to determine their importance to the internet community. It does this by a series of complex algorithms and the results are provided on a scale of 0 to 10, with 0 being the lowest and 10 being the highest. I would suggest that an average Google Page Rank is between 4 and 6. Alexa Ranking looks at the popularity of a website in comparison to all others. A score of under 100,000 is particularly good and one that is running above 10,000,000 bad.

A general rule of thumb is that as those sites with a high Google Page Rank will also have a high Alexa Rank, although at times this does not follow precisely. Once you have established your own rankings, you can then start looking at the competition. We all know who our closest competitors are, those that we would like to think we are better than and perhaps those that we would aspire to match. From a web perspective comparing each of the organisations Google and Alexa rankings against those of your own organisation’s, gives a good indication of your standing in the group.

Use of such competitor analysis helps organisations recognise that work is required, or that they are ahead of the game and need to maintain momentum. Whatever your line of business, the importance of being found quickly on the web is high. More traffic comes to your site and you can then reflect on what is happening to traffic when it arrives.

When checking a web site, make sure that you do not forget to consider the simpler aspects. Your rankings are an excellent gauge on how you are performing.

TCL India ensure that information of this nature is included as part of our Web Check results. These are quality indicators, although of a different type. Supply us with your list of competitors and we will also include the results for them in your report.

Reasons for Offshore Failure

2008-07-04T00:58:00.005+05:30

It is often the case that Indian resources are accused of failing to deliver what was requested. In my experience, whilst they are not faultless, more often than not the root cause of the problem is that what is required has been poorly specified. In all walks of life, when purchasing we expect the contents of the tin to be that which is described on the label. Why should testing be any different?

The reality is that people have been greedy for the savings that can be achieved from global economies. Regardless of a project’s suitability for offshore work, the project is forced down this path and may fail. The resources employed do their level best to deliver, but the reality is that with poor specifications and requirements, the chance of delivering to a high standard is significantly lowered.

Another area, in which mistakes are made, is the desire of the client to specify how the offshore operation will work. This may be flying in the face of years of experience, but the desire to control and make sure it works is so high that these factors are ignored, having the opposite effect. The offshore capabilities may also be culpable for not being stronger or refusing to engage in certain situations, but someone once said “the customer is always right”.

Whatever the culture that you are dealing with, be respectful of it. Accept it for it’s strengths and weaknesses and learn how to work with, through or around them. Recognize that as a nation you are likely to have your own idiosyncrasies and that these will be magnified to other nationalities.

The English reserve, the stiff upper lip, the desire to queue, being over polite, being rude, thugs, hooligans, the english have been accused of them all and no doubt there are those amongst us to whom elements remain true in part or whole. We expect others to deal with our traits, so why should we not be able to deal with others?

Beware of getting what you ask for. If you ask someone to take the shortest route to the other side of the mountain, it is technically accurate that this will involve tunnelling through the middle. This will take infinitely longer but is what was ordered. The expectation may have been that someone would either go around or over the top, but that was not what was requested. If your instructions to an offshore capability are specified precisely, the likelihood is that these instructions will be followed. Making sure that they are what you want takes some self-questioning before commencing.

Software Testing Interview Questions

2008-07-01T00:19:00.004+05:30

I am not suggesting that these are the only questions that would be asked, but are some samples for consideration. These are questions that I have used when interviewing others:

1. How many test scripts can you write in a day?
I am looking here for the ability to estimate. Whilst in reality the question is impractical, because it depends on so many different factors, I want to see the interviewee come up with some kind of answer. When estimating, there is a need to make assumptions and having a ball park figure enables someone to provide rough estimates more quickly.

2. How many test scripts can you execute in a day?
This is a repetition of the first question. I normally ask one directly after the other. The more junior resources tend to struggle with the first question and then the second also. The better resources learn from the experience of the first question and respond in a more positive manner to the second. Now I am not only looking at the ability to estimate, but also the ability to learn and an indication of the resources chances of seniority going forward.

3. Do you see testing as a service or a discipline?
Personally I am quite passionate about this one. I very much see testing as a discipline and a part of the software life-cycle that is as important as analysis, design and development. What I am trying to understand is the background that the individual someone has come from. Consultancy can demand one or other mindset and someone coming from either background can adapt, but I would suggest it is easier to revert from discipline to service than vice versa.

4. What is the most interesting defect you have found?
This is a passion question. I am looking to see if the individual can recount a particular incident and in what degree of detail. This begins to tell me whether they are a career tester or someone who is doing a job.

5. What are the key components of a test script?
Someone who is raising scripts every day should be able to define what information is required to be recorded. Different organisations have different standards, but there are key aspects to the answer. The requirement should be referenced or potentially included showing understanding that scripts should always be traceable to their point of origin. The script should have steps through, each of which defines the action to be taken, the data to be used and whether the step has passed or failed. This is the most basic of information and fundamental to all testers so an inability to provide this information is probably going to fail the candidate.

6. Can you define Black, White and Glass box testing?
This question is used to understand whether people have a basic understanding of testing jargon, as well as having attended the ISEB Foundation in Software Testing, where Black and White are covered. The inclusion of Glass box testing throws some people completely, others will think about it and try to define the answer; some will immediately define Glass box as being the same as White box.

Scenario Based Questions

You are the test manager on a project. The code is going to be delivered late and you now need to complete four weeks testing in two. How are you going to cope with this?

I am looking for managerial level thinking and understanding. There are numerous ways of dealing with this, more staff, overtime, weekend working, shift working, risk based testing, reduction in deliverables. The question is how many can they come up with and do they understand how to achieve them and the associated problems.

Risk Based Testing – How are they going to achieve this? Are the tests already grouped by risk? Are the requirements already grouped by risk or have the risk identified against each.

More test effort – Whilst increasing test effort is a simple means of achieving more in the same time frame, there needs to be a realisation that other departments will also need to apply similar Resourcing, to cover out of hours working, to handle increased volumes of defect fixing.

Cost Implication – Does the person discuss the increase in cost that can be associated with their suggested course of action? Are they thinking at project level or something lower?

Test Script Storage

2008-06-26T23:30:00.002+05:30

Storage of anything should be in a manner whereby retrieval of the item is easy. If we look at a library, everyone is familiar with the process. Find the section dealing with your subject matter and then search for the book by author. It has a proven track record and we are all able to go into a library and find what we are looking for (as long as it is there).

Test scripts are no different. As testers we generate vast volumes of scripts and need to be able to re-use them. It strikes me as odd therefore, that scripts are so often stored against a project. There may be subdivisions within the project structure for each of the applications being tested, perhaps even a definition of which version the scripts have been written against. Indeed If the testers are a constant from one phase of the project to the next, the scripts are likely to be re-used.

But for each project there is normally a primary application that is being tested and the others are interfacing systems. The scripts that are generated dealing with the interface do not necessarily cover the entire application. The first problem occurs when someone outside of the project needs to test the primary application. If they are unaware of the project name, the chances are that they will produce their own set of scripts. We now have a duplication of effort occurring. Each new script that is written is potentially wasting effort.

The second problem is that the scripts that were written to test a change to an interfacing system, will not be stored in a way that is usable or locatable by those testing it as a primary application. This generates two problems, the first being that the test suite for the interfacing system is now out of date. The second problem is that the regression pack has not been updated to reflect the changes and therefore future changes may introduce problems into the new code which will not be identified. At best the difference in version may alert the testers to the fact that a change has occurred and time will be spent trying to re-locate the scripts, or more likely, they will find out what the change was and generate their own scripts.

I believe the way to avoid this is to ensure that scripts are not stored by project, but always by application. The application should then be divided into a series of suites, one for each version. Lastly a regression pack should exist as a permanent fixture. This allows the storage of scripts as deltas to the original suite and then as each new version of the application is developed, the scripts can be reviewed for those which would best augment the regression pack. It may be prudent to have each version contain all scripts, as some will have been modified or removed from the test suite.

This means that any one on any project, can look at the applications they are working with and know without doubt, where to look for the scripts. I can only see one argument against this, which is around end-to-end tests and UAT, where the focus on the application is not present and the focus is more around the business process. In such instances these scripts may need to be stored against the project.

To my way of thinking, this is symptomatic of testing being seen as a service to a project. Therefore the need is to satisfy the project’s needs by storing everything in a very project-centric manner. The reality is that this is costing organisations a vast amount of time and money as scripts are permanently re-generated in order to meet the current requirement.
The situation becomes worse where large volumes of external resources are involved. A lack of consistency and knowledge leaving the client organisation becomes an issue and the likelihood of finding a previously written set of scripts lowers dramatically.

Lastly, the regression pack is key to ensure the stability of the production platform and anything which compromises the regression pack undermines the efficiency of the business.

So – to summarise, ensure that test scripts are written and stored by application wherever possible. Breakdown the scripts under the application folder by version and have a generic regression pack sub-folder either at application level or version level.

W3C 29 out of 30 Sites Fail

2008-06-26T19:12:00.003+05:30

W3C makes the following statement:

“W3C primarily pursues its mission through the creation of Web standards and guidelines. Since 1994, W3C has published more than 110 such standards, called W3C Recommendations. W3C also engages in education and outreach, develops software, and serves as an open forum for discussion about the Web. In order for the Web to reach its full potential, the most fundamental Web technologies must be compatible with one another and allow any hardware and software used to access the Web to work together. W3C refers to this goal as “Web interoperability.” By publishing open (non-proprietary) standards for Web languages and protocols, W3C seeks to avoid market fragmentation and thus Web fragmentation.”

http://www.w3.org/Consortium/

So how come so many sites are not following these standards. As a piece of work recently, we looked at 30 random web sites and checked them for W3C compliance. We found that only 1 of the 30 passed the check.

What was perhaps more surprising, was that around one third of the sites that were reviewed, had an error count of less than 30. Would it not be reasonable to suspect that an organisation which is so close to compliance to the foremost standards on the web, would go the extra mile and achieve compliance? I must therefore assume that these organisations are unaware of the fact that they are nearly W3C compliant and are developing their websites in ignorance, relying solely upon the development agencies to abide by their own standards, some of which may happen to coincide with W3C.

So what are the problems that organisations that are not W3C compliant going to face? Foremost, the transfer of the development from one agency to another is going to become more complex. Rather than being able to transfer code that is well written and understood by many, the site owner may become tied to a certain development agency, as they are the only ones that understand the code. The cost of changing a website completely is often far too expensive to consider, resulting in a reliance on one particular supplier.

Secondly, these standards once employed facilitate more effective and efficient crawling by web robots, who gather the information that search engines use. This means that poorly written code, translates to hard work for robots, poor understanding by search engines, lowering chances of the site being identified and reducing traffic to the site. Reduced traffic means reduced sales.

Thirdly, for those that are using the site from a disabled perspective, the tools that they use to try and surf the web, are hindered by poor code, making it more difficult to look at the site. We have already discussed elsewhere in the blog the importance of making web usage easy, yet here we find another example of poor user experience. Some organisations have even had lawsuits filed due to them failing to meet their obligations to disabled users.

Lastly, the site is less likely to transfer to other platforms used to surf the web, such as mobile phones and handheld devices. This is again restricting the use of the site, barring individuals that do not come on line using the mainly prescribed mechanisms.

My last point on W3C for the moment, so as not to be taken as a complete hypocrite, is that when I checked a couple of blogs, including my own, the volume of errors was in the hundreds. I am not yet certain whether this is something that can be remedied, but I can assure you that we will be looking at it and will let you know the results.

In summary, get your site W3C checked. Know that you are giving all users the best possible chance to use your site, have a good experience and possibly even generate a sale or two.

Middleware Regression Testing

2008-06-19T19:49:00.004+05:30

I have been working for a client on an interesting problem recently, which I thought I would share on the blog. The problem has been around the regression testing of an item which is intrinsic by nature, to many other projects. The item is classed as middleware, which provides a translation between two different systems, but is being relied upon increasingly, with in excess of 50 systems now interfacing. Most projects now require the middleware to be updated and the volume of change is placing the regression testing under increasing pressure as more releases occur.

The problem is amplified as the projects are working individually and the middleware changes are independent as a result. A group of projects are collated to form a release and once normal testing is completed the middleware changes are converged to form a single drop of code, which is then passed into the regression testing phase.

The first mistake has been made. The projects are not taking accountability for testing the converged code and in some instances have even been disbanded prior to this testing occurring. At this point the release has lost the testers with the understanding of the end to end function and more pressure is applied to the regression test team, to trace the scripts from the project, interpret them and generate new scripts. It is important that The testing of the converged code is now taking up large amounts of the already pressured regression testing window. This needs to be pushed back to the projects and the convergence owned by them.

By removing the testing of the converged code from the regression process, more time is now allowed and full regression can be performed, but this does not provide a future proof solution. The same window is now being fully utilised and as the volume of utilisation increases over time, the regression pack will increase and the window will become too small again.

This is an ideal opportunity to make the most of an automated solution. If we look to industry standards, automation makes a test 7 times faster to execute but 7 times longer to write. This means that the testing duration, once automated can be reduced to 1.5 days. Now we have a regression test that is only performing regression; that can be executed swiftly and provides scope for future growth.

The testing environments were another level of complexity. Whilst the project completed it’s testing with all the required components/systems/applications, the regression team, within their two week window had to acquire their own equivalent. Again this was another drag on time, but more importantly the regression was sometimes being performed on an environment that was incomplete. The result has been to make the regression middleware environment available to the projects for the convergence testing, but to utilise the projects interfacing systems. In order to future proof, the regression testing will be moved onto a pre-production environment.

There were other aspects to this problem, but those listed were particularly pertinent. The following feedback was received from two sources as a result of the work and presentation of the findings:

“I was delighted that your slides prompted so much discussion, even if it made your presentation difficult, and that by all accounts we have got the main individuals on board.”

“I was very appreciative of the clear way you presented where we are and your proposals for going forward, together with a clear capture of issues that are spoken about in lots of quarters but never pulled together into a single picture. Great.”

In summary – regression testing should encompass the entire system.
Projects should retain responsibility past the point of release to production.
Automation is ideally suited to regression testing.
Testing should always occur on as complete an environment as possible.

Dealing with Delays Impacting Software Testing

2008-06-16T13:58:00.011+05:30

We have all been there, on a project where the inevitable has happened. You guessed it, the code is late into testing, delivered at the 11th hour. Entry criteria has come under threat and possibly been ignored completely. Your test environment has been delivered but remains unproven as the code has not been available. The Project Manager has been harassed by the business stakeholders for the development delay and is not interested in your problems. IT Management are pressing to see the project delivered on time and to top it all, Marketing have arranged for a campaign launch to occur on the prescribed delivery date. As the Test Manager, you are now the primary obstacle to go live; the success of the project is resting on your shoulders. Oh yes, and your six week testing window has been reduced to four.

This is the stress (no pun intended), of the test execution phase. Not only are you now required to think out of the box in order to complete the testing, but you must also be thinking at a far wider level than just testing. There are actions that can be taken at a project level which can make a massive difference to the work of the testers.

Let’s start by looking at this from the wider perspective. I remember a situation on a very early project I was managing. Sat in a project progress meeting and being asked how we could test with an execution window reduced from four weeks to two. I dug my heels in and refused to budge, resulting in a separate meeting immediately afterwards with the PM, being instructed that this was not the right course of action and getting some early tuition in the art of testing.

Back to our problem, there are several actions that can be taken at project level to help with the situation and it may be possible to assist the PM by making some recommendations. (1) Does the application all need to be placed live? Are there any aspects of the application that could be put live as part of a second release, reducing the scope of the testing required? (2) Can the release occur to internal users only? This minimises the risk of damage in the event of production defects, meaning that exit criteria may be reviewed. (3) Can additional development effort be applied to recover whilst in development?

Regardless of the answers at a project level, the following can be applied by the Test Manager to handle the situation from within testing:

(1) Insert the testers into the development team and increase the unit and integration in the small testing. Improve the quality before it hits testing, reducing the volume of defects found and therefore the duration.
(2) Consider taking components of the application that are finished, prior to the arranged delivery date. It is likely that there are only certain parts of the application that are causing the delay and not the entirety of the application. Bringing in some parts early, increases the testing window and enables recovery of some of the lost time.
(3) Apply a risk based testing technique. Test the high risk elements of the application first and work through the testing in order of risk. When time runs out, this should mean that the only the items of lower risk have been omitted.
(4) Increase the hours being worked by the team. Look at options around overtime and weekend working. If using offshore capabilities then look at working two days in one. (A word of warning at this point, if the testers are working, they need to be supported by the environment support staff and the developers. Testing alone will only increase the flow of defects making it harder for development to keep up and an environment issue could stop all out of hours work.)
(5) Consider overlapping some of the testing phases. E.g. If UAT is being run as a separate phase, after functional testing, look at overlapping some of the UAT with the functional testing that is occurring.
(6) Ensure that there is a high focus on defect prioritisation by the business. Make sure the developers are fixing what needs to be fixed first. (Don’t ignore severity at this point.)
(7) Monitor defect turnaround. If the development has arrived late it is indicative of problems and a slow defect turnaround will cripple the project and whilst testing may complete, whilst the exit criteria will have been corrupted.
(8) Can more environments be made available? There is likely to already be a requirement for multiple environments, but if you begin overlapping test phases, functional with non-functional, with UAT, then the volume of environments needed may increase.
(9) Carry out a review of the exit criteria for the project. Bear in mind that this was set prior to the problems occurring and although they are the desired outcome, some compromise may need to be reached. Work out what is acceptable and don’t forget that if coverage is reduced, the numbers of defects are indicative of only a percentage of the testing. i.e. 80% coverage, means that you have possibly only have discovered 80% of the defects and it is good to assume that 20% remain unfound.

Actions (2), (3) and (5) above, all increase risk in some manner. Ensure that the project and stakeholders have agreed to these risks and that where possible mitigating actions are in place.

To summarise, there are many actions that can be taken within testing to deal with a slippage, whilst maintaining the original delivery date. Don't forget the project elements that can make a difference. I am sure there are more but do ensure that all aspects are being looked at and only then start to compromise on the testing.

Static Testing - Do you have a requirement?

2008-06-12T15:23:00.009+05:30

Static Testing is the first form of testing that can be applied during the software development life-cycle. Once the requirement document has been produced it should be checked against the 8 points identified below. This will ensure the removal of defects prior to other departments or resources becoming involved and therefore minimise costs.

60% of all defects are attributable to the requirement specification.

The 8 point check applied in order to perform static testing against a requirement document is as follows:

-Singular: One requirement that does not refer to others or use words like "and".
-Unambiguous: Not open to more than one interpretation. Clear and easy to understand.
-Measurable: Avoids use of words like instant, approximately. Specifies given units, such as hours minutes and seconds.
-Complete: The requirement is not lacking information or supporting data.
-Developable: The developers will be able to implement the requirement.
-Testable: The testers will be able to test the requirement.
-Achievement Driven: A benefit is associated, which is tangible.
-Business Owned: A member of the business owns each requirement, providing a point of reference and approval.

Within the testing discipline, 60% of defects are believed to be attributable to the requirement specification. This means that one of the largest contributors to poor software quality can be remedied at project initiation. It is also widely accepted that each phase later in the software development life-cycle that a defect is found, it costs 10 times more to resolve. In the simplest form, this means that for every £1 it costs to resolve a defect at the point of the requirement being defined, if missed until the resulting application is live, will cost 10,000 time more. This is due to the effort that will have been spent designing, developing and testing something which was wrong, but the real cost comes when a problem in live causes an outage affecting large volumes of users and perhaps even stopping them working. Not only does this impact the individual, but dependent on what they are doing, could stop income to an organisation's other departments: Sales, Billing, Payroll. All of these become disasters and if in the public eye can lead to bad press for the organisation.

It is 10,000 times cheaper to fix a defect in the requirement compared to live!

We are trying to make static testing for organisations as easy and affordable to achieve as possible to achieve for organisations. Without complexities of billing and non-disclosure, the document can be e-mailed to either of the contacts detailed below. Our pricing is fixed so there are no surprises along the way.

Grant.Obermaier@TransitionConsulting.in or Mick.Morey@TransitionConsulting.in

Testing will be scheduled and will take between 5 and 30 working days. If a faster turn around is required please specify any deadlines and we will advise on achievability.

Early Defect Detection

2008-06-12T03:30:00.012+05:30

They say that it is the early bird that catches the worm. The same is true in software testing.

The process starts with a requirement and from this point onwards, there is gradually more and more effort being applied to the creation or modification of an application. More people become involved and the cost of the solution increases. This is the natural course of events. It is therefore a natural conclusion, that if a mistake is made in the requirement, the cheapest time to fix the problem, is as the requirement is being defined, or immediately afterwards. This keeps the cost to a minimum. The same logic applies to design, development and test. Yes, even testing has the ability to insert defects, with badly written scenarios, scripts or reporting defects that are not real.

Defects are introduced throughout the software development lifecycle and the art of testing is to find as many of them as possible when they are inserted. It is widely recognised that there is a parabolic curve of defect insertion. The starting point is the requirement specification which begins by inserting 60% of the defects. The curve terminates with live or production, where the intended result is to find 0 defects. The tester should report on completion of the project, the defect detection efficiency. This looks at understanding for each defect, where it was inserted and where it was detected. A perfect test process would look to identify each defect as it is inserted. This is highly unlikely and the reality is that some defects will be found in later phases of testing, or indeed in live.

It is important to understand that projects testing budgets can double when critical steps such as static testing are not performed. This forces 60% of the defects into later phases, incurring as a minimum the cost of the design. Projects also often suffer from slippages and with test execution occurring at the end of the process, is often squeezed or compromised. The project must also realise that reducing the testing duration or coverage, is likely to increase the risk of a defect being found in live, where in comparison to it being found and removed in the requirements phase, will cost 10,000 times more to resolve, at best 10 times more. The reality here is that a defect undetected until live, can be business impacting, brand damaging and could cause a business to fail.

It is therefore important that testing is involved in the project from the outset, not as something that is included if there is the time, the budget and the inclination. Only by the application of systematic testing throughout the project is the quality level going to be understood and the opportunity to remedy problems in a timely manner presented.

Entry and Exit Criteria

2008-06-09T13:01:00.007+05:30

As you pass from one phase of testing into the next, there is a need for control. For the purpose of this post, we will refer to the prior phase as the supplier and the current phase as the recipient. The supplier needs to retain control of their testing phase, until such time as it is deemed ready for release. The recipient needs to ensure that the testing performed by the supplier has achieved a sufficiently high standard as to be acceptable.

The means of achieving this is referred to as Exit Criteria for the supplier and Entry Criteria for the recipient. These criteria are documented in the test plan and define the standards that should be achieved entering and exiting, the test phase described by the document.

The criteria are set by the Test Manager or nominated delegate. They may take any form that the test manager deems necessary, although are more frequently based on defect volumes of certain severity and priority, along with test assets from the supplier.

Consideration should be given to the use of other information, drawing on experiences of dealing with particular project teams, development agencies etc. This requires the Test Manager to be able to think outside of the pure testing aspects of the project, to look and see what else may impact them. Closer to the heart of the Test Manager are subjects like the Test Environment/s and Configuration Management. The environment is one of the main enablers to testing and should be made available two weeks prior to test execution commencing. Without comprehensive Configuration Management the likelihood of controlled deliveries from development into testing is severely reduced. Such events and dependencies make excellent criteria and are sound reasons for not entering into a subsequent phase of testing.

Looking at the testing components only, Test Managers should ensure that the supplier of each testing phase is held accountable for the production of the associated test assets. The volume of defects is often seen as a point of contention. Test Managers should avoid making statements such as zero defects allowed, or setting the bar so high that entry or exit becomes impossible to achieve. Having to relax these criteria because they are impractical corrupts the integrity of the testing function and undermines the credibility of the Test Manager. Work with the Project Manager when setting these criteria, ensuring that they are bought into them and don’t rely on just a signature on a document. This gives a higher level of buy in and ensures that any future change to the criteria is going to be resisted by both the Test Manager and the Project Manager. Always ensure that such changes are formally change controlled.

It is recommended that on entry into Integration in the Large, there should be no top priority or top severity defects outstanding. This is not to say that there can be no defects, only that items which are so important to the business as to warrant a top priority, should be resolved as a priority. Defects of the highest severity are perhaps more debatable, but delivering an item with high severity defects indicates that elements of the testing cannot be completed and the deliverable is therefore significantly below the level of functionality that has been specified and is expected.

If circumstances dictate that entry or exit criteria are going to be overruled, as a minimum the risk register needs to be updated to reflect that this has occurred, detailing the impact and mitigating actions. It is worth noting that this is normally indicative of a project in trouble and that timescales are now seriously at risk.

SOFTWARE TESTING GLOSSARY

2008-06-07T02:18:00.010+05:30

A………………………………………………………………………………………………………………………………………………………………
Accessibility: (Testing)Looks at an application to ensure that those with disabilities will be able to use the application as an able individual.
Agile: A development method, involving the creation of software, with the contributing parties, including testing, all working on the same item at the same time.
Alpha: (Testing)Testing of an application which is in production, but not available for general use. This is normally performed by users internal to the business.
Analyst: Person who interacts with the business in order to understand and document their requirements for an application or change to an existing one.
Analyst (Test): Person responsible for the preparation and execution of test scripts, recording and progressing of defects, reporting into the Test Team Leader or Test Manager.
Automation: The process of developing a script or program which can be run by software rather than a Test Engineer, to test an aspect of an application. Automation is used to increase the speed of test execution.
Automation Centre: A tool used in the automation of software testing. See
https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-127-24%5E1074_4000_100__&jumpid=reg_R1002_USEN
B………………………………………………………………………………………………………………………………………………………………
Beta: (Testing)Testing of an application which is in production, but not available for general use. This is normally performed by a select group of friendly external testers.
Black Box: The process of testing without understanding the internal workings of an application, relying on inputs and outputs only.
Bug: See Defect
Business Requirement Specification: See Requirement Specification
C………………………………………………………………………………………………………………………………………………………………
Case: See Scenario
Code: The software that has been produced by development and is being subjected to testing.
Completion Report: A document produced during the testing and presented to the project manager on completion of testing activity. The document should detail the tests that have been performed along with the associated results and defects. Some reports may include a recommendation for the applications suitability for release to production.
Configuration Management: The means of managing the components required to make a larger item. Applicable to files, documents, data, hardware, software, tools. Understanding of the versions of each component required in order to be able to re-build the larger item.
Criteria: See Entry Criteria and Exit Criteria
D………………………………………………………………………………………………………………………………………………………………
Data: Information pertaining to the use of a system or recorded through the use of a system. Data is used in order to test a system, potentially after data cleansing if personal information is involved.
Data Generator: A tool used to generate high volumes of data in order to be able to test many permutations, or to load test an item.
Data Scrambling: The process of altering personal information from data to be used for testing purposes.
Defect: Where the item under test has been found to inaccurate, resulting from testing. Primarily used in associated with software, but equally valid for static testing of documentation.
Defect Detection: The means of identifying a defect. Can be a metric used to predict the volume of defects expected during the course of a project, or as a means of looking back at a project to understand where testing needs to be concentrated in future projects of a similar nature.
Defect Removal Efficiency: A metric used to assess the ability of testing to remove defects as they are introduced, during the software development life-cycle, keeping the cost of testing later phases to a minimum.
Defect Turnaround: The time taken from the identification of a defect, through to the point of resolution. Different levels of granularity may be used. e.g. A focus on the time taken by development.
Developer: A person responsible for the development of an application.
Development: A process of producing an application by production of low level design, code, unit testing and integration in the small testing.
Dynamic: Testing which occurs on the right hand side of the V-model, with the application present.
E………………………………………………………………………………………………………………………………………………………………
Environment: The combination of hardware, software and data as used in development, testing and production. The platform/s upon which the testing occurs.
Entry Criteria: The criteria that must be met prior to a deliverable entering the next phase of testing to another. This is normally associated with documented test assets and pre-agreed volumes of defects.
Error: See Defect
Exit Criteria: The criteria that must be met prior to a deliverable leaving the current phase of testing. This is normally associated with documented test assets and pre-agreed volumes of defects.
Execution: The process of working through a script on the application under test, in the testing environment.
F………………………………………………………………………………………………………………………………………………………………
Functional: (Testing) The testing of a products function, against requirements and design.
Functional Specification: A document which extracts all of the functional requirements from the requirement specification.
G………………………………………………………………………………………………………………………………………………………………
Glass Box: See White Box
Grey Box: Testing performed by testers with some knowledge of the internal workings of an application. See also Black Box and White Box testing.
H………………………………………………………………………………………………………………………………………………………………
High Level Design: A design showing the major components and how these will interface to each other, defining the hardware to be used and the software that will be impacted.
I………………………………………………………………………………………………………………………………………………………………
Integration in the Large: Where the application or applications that have been developed are brought together along with those which have remained unchanged, building a production like system around the application/s. Testing is then applied looking at the communication between the different applications.
Integration in the Small: Where the components of the application that have been developed are brought together along with those which have remained unchanged, building the application or major component of a single application. Testing is then applied looking at the communication between the different components.
Integration: The act of bringing many parts together to form a whole.
ISEB: Information Systems Examination Board. This was historically the board that was used to certify test professionals at either Foundation (Entry Level) or Practitioner Level (Advanced). See:
http://www.bcs.org/server.php?show=nav.5732
ISTQB: International Software Testing Qualifications Board. See:
http://www.istqb.org/
J………………………………………………………………………………………………………………………………………………………………
K………………………………………………………………………………………………………………………………………………………………
Key Performance Indicator: A mechanism built on one or metrics, which determines a band of acceptable performance, which over time is often targeted towards improvement.
L………………………………………………………………………………………………………………………………………………………………
Load: One of the types of performance testing, this looks at testing for the maximum load on the system.
Load Runner: Tool used to performance test one or many applications, to understand how it handles increases in load. See:
https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-15-17%5e8_4000_100__
Low Level Design: Definition of exactly how the application/s will be modified or produced in order to meet the requirements and the high level design. This can extend in some examples to elements of pseudo code being defined.
M………………………………………………………………………………………………………………………………………………………………
Metric: A measure of an attribute or occurrence in connection with an organisation, department or functions performance.
N………………………………………………………………………………………………………………………………………………………………
Non-functional: How an application performs, rather than how it does it.
Non-functional Specification: A document which details the non-functional requirements such as performance, security, operational elements.
O………………………………………………………………………………………………………………………………………………………………
P………………………………………………………………………………………………………………………………………………………………
Performance Centre: A tool used for measuring the performance of an application or series of applications. See:
https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-126-17_4000_100__
Performance: Used to describe many types of testing relating to the speed of an application. See Volume, Load and Stress.
Plan: A document produced for a type of testing defining how the testing will be performed, when the testing will be performed and who will be performing it. The test plan lists the test scenarios that will be scripted.
Preparation: The process of generating the test scripts.
Priority: The importance of fixing a defect from a business perspective. Defined by business representatives.
Q………………………………………………………………………………………………………………………………………………………………
Quality: The suitability of an item for its intended purpose and how well it achieves this.
Quality Centre: Tool used to assist with the management of testing, recording and tracking scripts, logging and tracking defects and more. See:
https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-127-24_4000_100__
R………………………………………………………………………………………………………………………………………………………………
Regression: Retesting of a previously tested program following modification to ensure that faults have not been introduced or uncovered as a result of the changes made
Requirement Specification: A document normally produced by a business analyst, capturing the needs of the individual in a manner which means that they can be translated into a software solution.
Re-Test: Taking a defect which has failed and executing the associated test script again.
S………………………………………………………………………………………………………………………………………………………………
Scenario: A high level description of how a requirement is going to be tested.
Script: Referable back to the scenario, the script defines each step that must be passed through in order to perform a test, along with the expected results. As the script is executed, results are recorded and if they match the expected result are marked as passed, otherwise as failed. A script containing a failure should have a resultant defect raised.
Schedule: A document, similar to a project plan, but detailing the activities associated with testing, when they are due to occur and who will be performing them.
Severity: The importance of a defect to testing and the application. Defined by Testers
Smoke: A process of proving an application or environment is ready to commence full testing, by running a sample set of scripts testing the primary functionality and/or connectivity.
Static: The process of testing without the presence of software. Normally refers to the testing of documentation.
Strategy: A document produced at project or programme level, defining what testing is to be performed.
Stress: A form of performance testing, whereby the volume of testing is increased to a point at which the application is deemed to be failing to perform, either due to failure to meet performance criteria or system breakdown.
T………………………………………………………………………………………………………………………………………………………………
Technical Architecture Document: Definition of how the business requirements should be translated into a technical solution at the highest level.
Test Director: A tool used for test management, capture of requirements, scripts and defects, with the ability to manage defects through to resolution. Now replaced by Quality Centre.
Testing: The process of reviewing an item for accuracy and its ability to complete the desired objective.
U………………………………………………………………………………………………………………………………………………………………
Unit: The testing of the application by the developers at a component or modular level, looking at whether the code is performing as expected.
User Acceptance: The means of testing applied, normally by members of the business or recipients of the system, whilst still in the test environment. This looks to ensure that business requirements have been understood by all involved in the application production as well as providing an opportunity to check the business processes that the application will be used in conjunction with.
V………………………………………………………………………………………………………………………………………………………………
V-Model: A testing focussed software development life-cycle where the process of creativity occurs down the left side of the V and the process of testing up the right side of the V, where each step down the V has a corresponding point of testing up the right of the V. Begins with a requirement and terminates with User Acceptance Testing.
Version: An alpha numeric means of identifying a particular build of software, or a component thereof. The version changes incrementally to reflect each modification of the software, component, document etc.
Volume: (Testing) A type of performance testing which increases the volume of users on a system, in each cycle, taking the volume up to a prescribed limit, normally exceeding optimum load.
W………………………………………………………………………………………………………………………………………………………………
W3C: World Wide Web Consortium – body creating web standards and guidelines.
http://www.w3.org/
WAI: Web Accessibility Initiative. See:
http://www.w3.org/WAI/
Waterfall: Project Management Method whereby each element of the software development life-cycle occurs sequentially.
Win Runner: A tool that was historically used for test automation. See Automation Centre.
White Box: Testing normally performed by developers, looking at the code and ensuring that each model of code is performing as expected. See Unit
X………………………………………………………………………………………………………………………………………………………………
Y………………………………………………………………………………………………………………………………………………………………
Z………………………………………………………………………………………………………………………………………………………………

Looking to Recruit a Tester

2008-06-04T21:15:00.007+05:30

You have a vacancy or you may just want to breathe new energy into your test function. The logical option appears to be a new Software Test Engineer and you would like them sooner rather than later.

The employment of a UK Test Engineer is going to cost an organization between £250 and £300 per day, ignoring the cost of recruitment. Why do this when you can have a four man offshore team for the same price?

Think of the difference that you can make with a four man team, in comparison to the one UK resource that you are currently seeking. Stop your own team members from having to do the mundane tasks. How many times do you see the look of pain appear on someone's face at the prospect of having to perform more scripting? Umpteen permutations of the same thing – yawn!!!! Relieve the boredom and place the work offshore. Then start thinking about what the rest of your team can do, now that so much repetitive work is out of the way.

This paves the way for your current team to:

- Increase the innovation and use the time to find ways to improve the department's capability.
- Find time to work on the processes that always take second place to the projects.
- Increase the throughput of work and stop being seen as a bottleneck to progress.
- Apply more resource to existing projects and increase the coverage of testing.
- Increased staff satisfaction resulting in greater resource and knowledge retention.

Ultimately you have already invested in your existing team and they will relish the opportunity to move away from the tedium and onto more interesting and stimulating work. This will increase the job satisfaction and naturally improve team morale.

If this can all be achieved at the price of one permanent resource, can you really afford not to try it?

TCL India will provide a four man team, comprising a lead and three test engineers for £300 per day. You will have a single point of contact offshore and TCL India’s UK Management will ensure that the function works for you, lending our wealth of experience in offshore working. This can be put in place very quickly and you can start realizing the benefits of having four people instead of one.

E-mail : Grant.Obermaier@TransitionConsulting.co.uk

or Mick.Morey@TransitionConsulting.co.uk

If you have come across this as a result of looking for work, we will shortly be recruiting in India and currently looking in the UK. Please feel free to e-mail your CV.

A View on Test Automation

2008-06-03T03:04:00.007+05:30

Testing has been a long and laborious task for many years, probably since its inception. I can’t comment on that, as I was not around. I have seen the introduction of automation as a means of reducing the time taken to test, but as with all new ideas, people jump on the band wagon and the results are not always as expected.

With vast sums of money invested, the cost of automation was high (when compared to a manual tester), a skill set which was hard to come by, but with the results of their labours there was an expectation that greater savings would be made. One man would be able to do the work of ten etc. The results however often failed to live up to expectation over a period of time. Early successes were not invested in and the result was a lot of automation started to gather dust.

One of the leading sellers of automation software once indicated that an automated script would take seven times longer to write, but would take one seventh of the time to execute. This would mean that a script that was automated would need to be run seven times without change in order to cover the cost of its creation. Here lies the main problem.

Initially automated scripts were often based on a record and playback capability, which whilst of extreme use had a finite life if the application was changed in any way. As time has moved on this has now become a basic form of automation and the capability is now more based around intelligence, not relying on the position in which it appears on the screen, but the fact that it is there as an object. It must still be understood that an application which is undergoing regular and significant change, is far less suitable than something which is relatively stable and unchanging.

Automation does however have a place. Regression testing, which relies on the same scripts being repeated, is an ideal candidate for automation. Another area is smoke testing or sanity testing, the process of ensuring that an application is performing as expected with a reduced regression set, perhaps for the purposes of checking that a test environment has been set up correctly. It should also be remembered that certain applications will go through many functional cycles of testing for a single release, again making them more suitable for automation.

This method of testing is often viable and the greater the capability of the automation engineer the more likely the success of the automation. Try to avoid a wholesale approach, which is likely to leave you with a lot of shelf-ware gathering dust. Look at each project and application individually and understand whether you are likely to obtain the returns you seek. Look at all the types of products that are available and select one that suits your purpose, but also for which the skills are available. Last, but by no means least, ensure that you plan the automation from the start, allowing for the extra time to develop the scripts. Once started, continue to evaluate the approach, bearing in mind that the scripts are likely to need updating with changes to some extent, generating a maintenance cost.

If you are looking at automation of regression packs, or for smoke testing purposes, consider the offshore option. The scripts are pre-defined and therefore the work understood. The cost is as always significantly lower.

Severity vs Priority

2008-06-02T01:22:00.004+05:30

When raising a defect, there is a common confusion that occurs for testers and others alike. What is the difference between a defects severity and its priority?

So let’s start with looking at the dictionary definitions:
Severity - Causing very great pain, difficulty, anxiety, damage, etc.
Priority - Something that is very important and must be dealt with before other things.

We can see that the two words have completely different meanings. Why then is their confusion between them?

The severity is the domain of the tester and they should be capable of recording this. The severity to the testers is the impact of the defect on the application and their ability to continue testing. The priority is the domain of the business and should be entered by them against each defect raised to reflect the importance of the change to them.

For instance, a spelling mistake would be deemed as a low severity by the tester, but if this mistake occurs in the company name or address, this would be classed as high priority by the business. An inability to access a rarely used menu option may be of low priority to the business, but the severity is high as a series of tests cannot be executed, all dependent on access to the option.

The mistake that we have seen made many times is to assume that the tester is also capable of recording the priority. Whilst it may be possible for the tester to make an educated assessment, the priority is the business’ means of defining what must be repaired prior to release to production and the order in which effort should be applied to the fixing of defects. Testers who have been involved with a particular application for some period of time may be able to do this, but it is essential to have adequate business representation on the project and their involvement in the life-cycle of a defect and the defect management process.

When a project enters test execution, the focus will be on fixing defects of the highest priority. This means that the application will be released with the minimum amount of priority defects unresolved. Care should be taken by the Project Manager to ensure that whilst the priority is paramount, severity is not ignored. What is needed is a balanced approach, which favours the business priority. At the end of the project the volume of high severity and high priority defects should have at least been reduced, if not removed, in order to meet the exit criteria defined in the test strategy.

To summarize:
Priority = Business = Order of Fixing
Severity = Tester = Failure of Application

Components of a Bug

2008-05-31T00:14:00.006+05:30

Legs, body, head, eyes, mouth; or are we talking about software bugs? The bug, error, defect; it little matters what it is called, but some are more touchy about the subject than others and like to use more politically correct statements, like issue or observation. The reality is that during the testing of an application or piece of software, something is identified as not performing in the expected manner, as driven by experience or indeed by the requirement specification, functional specification or design documents.

Call it what you will, we will use the term defect. Ensuring that you capture the right information at this point is critical. From a testing and development perspective, it is essential to be able to reproduce the defect at a later date, either as part of the remedial process or proving that a remedy has worked. TCL India would recommend that the following points are recorded for each defect:

· Unique defect identifier – a unique reference by which the defect can be identified.
· The application name or identifier and version – the software in which the defect was located.
· Environment specifics – the environment that was being used to test the application.
· Test script identifier (if applicable) – A unique reference from the test script, enabling each defect to be identified by script.
· Defect synopsis – A brief description of the defect that has been encountered.
· Detailed description of Defect – Full definition of the defect, enabling it to be reproduced by developers or those trying to resolve it.
· Test Steps Executed – from the test script, details of the steps executed and the detail of the step that has failed
· Expected Result for Test Step – The expected results for the step, e.g. the displayed logo should have been blue and white
· Actual Result for Test Step – The actual results encountered, e.g. the displayed logo was red and white
· Evidence of defect – Additional information showing the defect, such as screen shots
· Severity estimate – The impact of the defect on the ability of the tester to complete testing
· Priority – The importance of resolving the defect to the organisation
· Tester name – The name of the tester who has reported the defect
· Test date – The date that the tester located the defect
· Defect reporting date – The date that the defect is located. This should be immediate
· Defect assigned to – Who is being asked to resolve the defect

Each time responsibility for a defect is passed to another party, that fact should be recorded to form a history. The value becomes obvious when investigating why certain defects may not have been resolved in a timely manner. Investigation may be required as to why defects are bouncing backwards and forwards between different people, or those which are with an individual or team for a long period of time, impacting on the defect fix turn around time, potentially breaking SLAs.

In some more advanced tools used for defect management you will be able to prescribe the life-cycle of a defect. Such as:

-Raised by the tester

-Passed to the business analyst to prioritise

-Passed to the environment manager for specification of the environment and ensuring the defect was not caused by environmental issues

-Then onto development to fix code
-Back to the tester for re-testing

-Then onto the test manager or defect manager for closure.

The key then is to report on the defects on a daily basis during test execution, keeping the project informed of the number of defects raised and closed that day and the resulting cumulative totals. This enables the Project Manager to make decisions on the project's likelihood to go live on time, whether to apply more resources to certain aspects of the projects, whether overtime is required etc.

Care should be taken not to raise the same defect over and over again. When volumes of defects are high it is worth employing a defect manager to ensure that defects are not being repeatedly raised, wasting the time of resources. The defect manager should also ensure that all defects are being progressed. Sometimes defects that have been remedied are re-introduced erroneously: this is usually symptomatic of poor configuration management.

Offshore Communication

2008-05-30T04:00:00.006+05:30

Communication is a critical point for a successful offshore arrangement. Our TCL COMS (Consultancy led Offshore Managed Service) is built around our understanding of this. But beware, communication can be difficult, even amongst people of our own nationality and culture. Speaking from personal experience, there is a huge between speaking to someone with a broad Scottish, Geordie or Wiltshire accent, when compared to someone speaking “The Queens English”. I mean no offence to any of the above, but accents are hard to deal with face to face and even harder when a telephone is used to handle the communication.

When dealing with other nationalities, where English is not the first language, the problems are amplified and whilst there are some foreigners who speak English perfectly the chances of finding such people becomes more difficult.

What we have found is that when people from other countries come and work for extended periods in the UK, their language skills naturally increase. As a result of this TCL India look to deploy resources locally, that have a long exposure to the UK, it’s people, their idiosyncrasies and the colloquialisms they use. These resources become highly useful, because not only can they communicate with us, they can also communicate with those offshore, utilising combinations of their own language and English. This provides an excellent conduit for communication, but it cannot be relied upon as the sole answer to the problem.

Written communication is far easier and less liable to be misunderstood than verbal, if interacting directly with an offshore capability. E-mail can be used and tools such as MSN Messenger if a more conversational approach is required.

Video Conferencing offers an ability to see as well as hear, but whilst it is great to see the faces of those you are talking to, there are inherent problems. Not only are you dealing with accents, but you have a time lapse and perhaps poor image quality, all of which is tending to distract you. There is also the potential lack of visual and audio synchronisation that your brain will be trying to cope with. It is suggested that if video conferencing is to be used, it does not occur at the beginning of the relationship and that time is taken to become accustomed to the experience.

In all offshore communication, look for feedback to demonstrate understanding. Query their understanding of the requests and make sure that theirs is the same as yours. Controversially perhaps, for the offshore element of a relationship, use the same techniques back at the client. Make sure that your grasp is right. Make sure that they have understood your communication. Whilst this is written very much from a UK perspective, the rules apply the world over.

Beware of getting back exactly what you have asked for and not what you wanted. Be careful in your communication. You could be taken literally.

Contractually Demanding Testing from Developers

2008-05-29T01:21:00.004+05:30

It is an experience that testers all over the world will instantly understand. The contract with the development agency makes no demand for them to perform unit and integration (in the small) testing.

How does this manifest itself? Firstly, the Test Manager will specify within the Test Strategy or Test Plans a series of entry criteria, from development into testing. This should comprise test plans, a full suite of test scripts and a series of defects that have been raised. As the development nears completion, the Test Manager will begin to ask for the evidence of the testing and look to meet the entry criteria for Integration (in the large) or formal testing.

At this point the response can be that the unit and integration (in the small) testing has not been contractually specified. The supplier often takes one of two stances, stating that this cannot be achieved, or insisting that the testing can be performed, but at additional cost and increased timescale. This immediately places the Project Manager in an awkward position: they will be unable to agree to the additional time and will not want to incur the increased costs. The normal action at this point is for the Project Manager to accept that a mistake has been made, log a risk and accept that the code will be delivered into testing without development testing having occurred (or proven to have occurred).

The quality of code is now likely to be of a lower standard and the volume of defects expected in the formal testing higher. This may result in additional cycles of testing and increases in costs from this perspective (part of the risks that the Project Manager will have taken on board). As the volume of defects found during the black box testing grows, the demand on the development agency increases and care should be taken to ensure that defect fix times are monitored. These are likely to slow as the volume of defects increases and this should be reported to the Project Manager as soon as it is seen. A good contract will have service level agreements(SLAs) in place, to ensure that defect turn-around is at a specified rate depending on severity and priority.

If the development agency has failed to perform their own testing, the demand is placed on the testing team or test supplier and the estimates that they will be working to should be re-visited. If testing is working on a fixed price contract, failure to meet entry criteria should constitute a change request and the ability to re-price.

It may be possible to insert some of the test team with the developers, allowing them to test unit and integration (in the small) in parallel to the developers. Alternatively, testers with a technical bent can be asked to look at the code, perhaps performing dry runs. If the testing is being performed but the deliverables are not being made available, then it may be possible to witness some of the testing that is occurring. It must be said that these last two points are more geared to damage limitation, rather than ensuring that the right level of quality has been built into the product.

We need to avoid this in the first place, so ensure that your organisation's Supply Chain has a standard wording that is inserted into all contracts with development agencies protecting from such events. For smaller organisations that do not have a Supply Chain function, bring the Test Manager on board for up-front consultancy; they should look to ensure that this does not happen. A further measure is for the development agency to be asked to sign off the test strategy, which should detail the entry criteria into the formal (black box) testing phase, giving the buyer some level of recourse. Lastly, as a Test Manager, don't leave it until the end of development to find you have a problem - identify it as early as possible and get written agreement that the appropriate testing will take place and that the test assets produced will form part of a deliverable.

WebCheck - More Information

2008-05-28T18:48:00.014+05:30

We thought we would give some more information on the Web Check process as an addendum to the previous Introduction posting.

http://tcl-india.blogspot.com/2008/05/tcl-india-introduce-web-check.html

Websites are critical to all of us and with 15% of all retail trade in the UK conducted on line and whilst the credit crunch is hitting retail elsewhere, the effectiveness of the web ensures that it remains buoyant.

Some organizations are happy to rely on automated tools to assess their website/s, but can they really experience the web site as their designers intended? Programs are as intelligent as the programmer and whilst they may be very good, recognising that the layout of a page is jarring on the eye is going to be hard to achieve to say the least. Using the web is to some extent an emotional experience and this needs people to experience it. Most of us use the internet to shop to some extent, be that a book from Amazon or shopping from Sainsbury or Tesco. As a high street shopper, the ability to move from one store to another takes time and effort. The ability to ask a shop assistant for help is simple. On the internet it is not so easy. We hit problems and change site at the click of a button - opportunity missed! Zero time and effort and zero tolerance for problems or challenges that make shopping harder to achieve.

WebCheck is performed by professional testers, familiar with web testing, experienced in understanding the needs of the user and most of all…human. We don’t ignore the capabilities of web crawlers as this would be foolhardy: They do have a place and we augment our manual process with them.

As an output from each of our WebChecks we provide a certificate showing the quality percentage achieved. Further, a detailed report of our findings, including defects, observations and recommendations is provided. We also look at aspects such as Google and Alexa rankings, understanding the importance of your site to the internet community and the likelihood that people will come to your site. If desired, we can provide comparisons between your site and others, from a ranking perspective or user experience. We are not looking to develop, host or design your site. We are testers and want to help clients understand their sites problems and capabilities.

If you would like a quotation for the Webcheck service on your website, please e-mail us using the contact details on the left of the page.

Raising Awareness of Software Testing

2008-05-24T00:05:00.016+05:30

In some organisations that have their own internal testing departments, they can suffer from a lack of awareness and sometimes understanding of testing. How do you communicate what testing is all about to the masses? An idea that was had some time ago, was to put posters up around the business, gentle reminders that the testers exist, memory joggers to get people thinking about testing. A trawl of the internet found various means of visually assisting and we thought we would share some of these with you.

This one has to be “Load Testing”. Can your systems cope with the expected load? A great photograph that makes me smile every time I see it.

How about this one for “disaster recovery testing”?
Can your crew recover from this?

“Regression testing” stretches the mind a little but we came up with this. Do you understand the impact of change? Have you handled all of the ripples?

On the subject of “Stress Testing”: At what point will it break? This one is very visual and in our opinion instantly understandable.

What we did find, was that different people had very different perceptions of what some images meant to them. It was very much a personal interpretation. Testing is still one of the least well known disciplines of the software development life-cycle and departments which are run in a service like manner, called upon when needed and not a “must have”, need to let organisations know of their existence, encourage thinking about testing and encourage engagement on projects.

Why Use External Testers

2008-05-23T03:31:00.007+05:30

In the modern organisation, IT is intrinsic to most of what goes on. From a laptop running MS products, to the multitude of systems used by a corporate, the need for software testing is growing as the uptake of IT solutions to resolve our business needs increases. The testing department is coming under increasing demand to become involved in ever increasing volumes of projects. If the volume of work were static the problem would be more easily resolved with additional resources being employed. The situation that presents most test departments is far from a balanced flow of work.

Lots of businesses have particular times during the calendar year where the volume of project activity increases. For some retailers this would be Christmas, for companies involved in education it could be the start of a term, but the pressures come in peaks and troughs. A test department may be able to cope with some periodic increase in demand but may find itself in a situation where it is perceived as a bottleneck to the desired throughput of projects. This reflects badly on the department and can be avoided by bring in external testers.

By understanding the flow of work coming through the department and plotting this over the year ahead (if possible), the size of the core team can be understood. The core team is the size of team required at the lowest point of utilisation. External testers are then used to deal with the demand for resource over and above the core team, covering the peaks of activity. The external testers can be employed for the duration of the peak and then released, dropping the team back to it’s core size.

Recruitment of good quality test resources is becoming increasingly difficult. Less people are leaving university with IT related degrees in this country. Numbers which are a couple of years old suggested that there were only 30,000 per annum in the UK and the number was dropping, where as in India the number was 500,000 and increasing. The cost of recruitment is also prohibitive and can be avoided by use of external resources. If activities can be performed offsite, then some of the additional overheads of Resourcing in terms of phones, computers, desk space, HR and training can be avoided, making the use of some external resources here in the UK, cheaper than employing them. Cost is an ever present consideration and if external resources can be used and released as is required, provide the right level of expertise and be utilised offsite then these factors begin to form a really strong case for the use of the external tester.

Flexibility is key to the way we work and the supplier of external test resources that can provide this at a price which is competitive compared to employment, are worthy of consideration.

Does Offshore Work

2008-05-22T05:10:00.005+05:30

Simply – yes – if performed correctly. Where so many fail, is by misunderstanding some of the key factors that need to be in place. The simplest way is to treat it as a black box scenario, offshoring well specified work packages, with quantifiable and specified deliverables against known timescales. This reduced need for communication simplifies the process, but it is only as good as the inputs. If these can be well specified then the chance of a good delivery is excellent. A prime example of offshore capability being used in this manner would be the generation of test scripts. This is often seen as a tedious activity that bores the UK Test Analyst, being repetitive in nature. Providing an offshore capability with a requirement specification that has been statically tested, perhaps even with the test scenarios defined, should enable a good offshore team to produce a well defined suite of scripts. Confidence can be obtained by sampling scripts as they are being produced and example scripts can be provided by the client to define the standards to be followed. Having mentioned static testing, the passing of a single document to an offshore team for review against the 8 point check is simple and requires minimal communication. Neither the test scripting nor the static testing require anything more than e-mail to send information, so setting up links between the offshore destination and your own test environments is not required. If you are not currently utilising offshore then we would strongly recommend giving one of these a try and sampling the option.

Once confidence to this level has been achieved the next easiest element to place offshore is the test execution. This does involve setting up links to the offshore capability, but some software can be used to achieve this quite simply. Access to defect management tools needs to be arranged to ensure that defects are reported as they are found and not gathered for end of day reporting. Freeware is available such as Bugzilla to achieve this and only needs to be hosted. The data protection act comes into play here, but it should be remembered that personal data should be scrambled prior to testing anyway, if not using simulated data. At this point it is worth pointing out that we do not recommend that all of the test execution is performed offshore, as interfacing with developers, business analysts, environment managers and project managers is required, so some onsite presence is required. Again if this is new to your organisation, try just placing a small fraction of the work offshore and increase the volume as confidence grows.

As more onus is placed on the offshore capability, the increase in management and communication occurs. This should start moving the operation into that of a service and the management mechanisms should be introduced as defined in http://tcl-india.blogspot.com/2008/05/tcl-coms.html.

From a service perspective, it is critical that both the offshore and onshore sides of the operation need to understand each other, how they work, what their expectations are and how to get things done. Too many people trust in organisations that put people in place to manage an offshore setup that the client has no knowledge of or relationship with. This can work, but beware the teething troubles that can occur. A good mechanism is to have the head of offshore resource work onsite for a considerable period of time, at least 3 months if not 6, before moving into an offshore service model. Most importantly rely on a team that has successfully managed offshore engagements and work well as a unit spread across the two countries.

To summarise, start simple, build confidence, know your offshore lead and gradually move to an offshore model. If moving straight into offshore, get expert assistance.

TCL India would be happy to discuss how to run an offshore trial with you. Please contact either Mick or Grant for further information or to set up a meeting.

Tester vs. Developer

2008-05-21T21:11:00.006+05:30

Often a battle of many rounds, the tester is seen as the anti-developer, with the developer standing for creation and the tester the destroyer of the developers work. On many occasions you will see their two departments almost fighting battles as each tries to prove its worth and capability. The smart developer will realise that the tester is there to enhance the impression given by the developer. A developer who is told where the bugs are is able to modify their code and the result is a far happier end user. Not only that, but if the tester is involved in static testing of requirements, the difference to the developer can be huge, with each requirement being unambiguous and so far easier to develop in the first instance. When code goes live and the volume of bugs is minimal with positive user experience, it is rarely the tester that is thanked for their diligence: more likely to be the developer thanked for their fantastic software.

Yet all too frequently, the developer and their skill set are valued above the tester. Within the project life-cycle, development is considered essential, testing as a "nice to have". The disciplines are two sides of the same coin and occasionally someone migrates from the dark side to the light or vice versa. (Up to you to decide which is which?) The developers mind set is one of creation whereas the tester is one of destruction. There is no doubt that the tester becomes redundant if there is no development whereas the opposite is not true. Many projects are still run today with no focus on testing and the onus is placed on the users to experience the problems and report them.

It is well cited within the testing industry, that each progression of a defect to a later phase in the software development life-cycle (SDLC), makes it 10 times more expensive to resolve. So a requirement costs £1 to fix, design costs £10 to fix, development costs £100 to fix, testing costs £1000 and live costs £10000. Defects that translate into an outage for a business, where large volumes of resource are stopped from working, are indeed expensive. Surely this makes the role of the tester incredibly valuable to the SDLC.

Here are some other considerations on the value of the humble tester:

The tester must be able to analyse requirements and in static testing take on part of the guise of the business analyst
Rather than looking singularly at how a function should be used, look at all of the permutations then look at all the permutations of how it should not be used
Pressure is high as the time allocated for testing on a project is regularly reduced as earlier phases slip, resulting in late code delivery
The tester is trained in multiple testing disciplines and techniques, enabling them to perform sufficient testing
There are as many types of testing that the tester should be able to handle, as there are development languages
Automation, Performance and Security Testing require levels of expertise akin to a developer or network specialist
The go-live decision of a project relies primarily on the advice of the tester and the evidence that they can provide to support the decision
Advising and working with the business to perform acceptance testing
Able to witness the unit and integration in the small testing performed by the developer
Efficient management of defects through to closure.

Test Estimation (or) How Long is a Piece of String?

2008-05-20T22:02:00.005+05:30

Estimation is an art form at the best of times, depending on the experience of the test manager, the stage of engagement in the software development life-cycle and many other considerations. Different organisations will have different expectations and it is important to understand this as a key consideration. E.g. If you start including every possible variety of non-functional testing in your estimate, in an organisation that does not normally look at this, you are going to have a result which is wildly inaccurate.

Some companies will have rules in terms of percentages of the total project cost; 20% Analysis and Design, 40% Development and 40% Testing, for example. This will vary depending on an organisation's focus on quality, but could be as low as 10% for testing.

We would suggest that the following aspects are considered:

General Risk : How mature is the relationship with the development agency? How often the application is being changed? Are more requirements anticipated? Do Service Level Agreements governign defect fix turn-around exist? Is the Configuration Management process mature?

Environments : Do they exist? Are there processes for environment management? Are the environments maintained in line with production? Will the environments be established and proven prior to test execution?

Quality : Have the requirements been static tested? How many defects or production issues are currently associated with the application/s? What level of unit and integration in the small testing is being performed? How many defects were found during earlier phases? Is the project confident that entry and exit criteria are being met? Is the project already slipping or maintaining schedule? How mature is the project team – is this something they are all familiar with doing? Is this a safety critical application/system?

Effort/Scale : How many requirements are to be tested? What percentage of the application/s is/are changing? How many cycles of execution are required? Do regression test packs already exist? What deliverables are needed/required?

Complexity : How many suppliers and development agencies are involved in the process? How many technologies and platforms are involved? How many systems are being changed? How many systems are interfacing? How many stakeholders are involved with the project/programme?

Timing : Is the project taking place over a drawn out period of time? Will the project stop and start? Is the testing expected to be performed over a very short time frame when compared to the effort required?

Any of the above have the ability to influence what is needed to complete the project, impacting different phases of the testing life-cycle, or the volume of effort that is required by different resources. Don’t forget that when project size or complexity increases, the strain on the test manager increases, and the introduction of additional resource such as defect managers may be necessary.

The base estimate should look at the volume of scripts that are required. From this you can calculate how many can be written and executed in a day and work out resource requirements accordingly. The management time can often equate to a further 33% effort. As a guideline a conservative estimate would look to write 5 scripts per day and execute 10 scripts per day. This is dependent on the content of the script and the experience of the testers, but can be significantly higher. Bear in mind that the cycles of execution will each act at as a multiplier to the volume of test execution. Two cycles should be the minimum. More than four would be indicative of problems.

TCL India have a mechanism for applying the above considerations to our estimates and weightings associated with each of the different elements.

TCL India - Testing Help and Advice

2008-05-20T04:30:00.002+05:30

The IT industry is becoming more co-operative and sharing of it's knowledge, both with other IT professionals and the business community as a whole. To this end, TCL India would like to offer advice around testing issues, derived from our experience and those of our colleagues.

If you have questions about testing, offshore or just about the blog, then please post your questions to Grant.Obermaier@TransitionConsulting.co.uk and we will do our best to respond.

Please note that there is no charge for this activity but that TCL India or any other orgranisation associated with the Transition Consulting Limited group of companies, cannot be held liable for any consequence resulting from following recommendations made.

An Introduction to Testing

2008-05-16T22:26:00.011+05:30

It is amazing in this day and age, that we are still asked why you should test. When you are really passionate about a subject it is hard, to say the least, to go right back to basics and justify why you are needed as part of the Software Lifecycle.

Lets look at some definitions of testing:

http://www.answers.com/topic/test?cat=technology
“a procedure for critical evaluation; a means of determining the presence, quality, or truth of something”

http://www.askoxford.com/concise_oed/test_1?view=uk
“a procedure intended to establish the quality, performance, or reliability of something.”

http://m-w.com/dictionary/test

“the procedure of submitting a statement to such conditions or operations as will lead to its proof or disproof or to its acceptance or rejection”

To put our own interpretation on this, we believe that testing helps to:-

- Find problems early

- Identify if build meets requirements

- Improve user perception

- Avoid unnecessary outages

- Reduce maintenance

- Establish quality of purchases

- Understand the risks

It is not the complete answer and will not ensure that your code is bug free before it goes live, but it will enable you to understand the risk of going live. Testing looks to perform sufficient checks to understand the quality: it cannot on it's own resolve the problems. Testing does not look at every single permutation, or we would be testing forever, but looks to test sufficient permutations to give confidence in the level of quality attained. In certain instances, such as safety critical applications, the volume of testing does increase as the confidence required needs to be higher.

The above graphic shows a series of testing types. This is often an area of confusion for someone new to testing, but each of these has a specific function. They can also often be references to the same thing, such as White Box and Unit from the above list. Part of the skills of the test manager are in knowing what test types to apply to a project and the level of intensity of the testing performed therein.

Different types of testing are performed by different people or roles associated with a project. The graphic below shows a series of testing types and the roles that would normally be expected to perform the testing. It is worth noting that the Test Manager associated with a project, would take responsibility for all testing to some greater or less degree.

From a Testing perspective, the testing is broken into 5 primary areas, Engagement, Planning, Preparation, Execution and Completion. A series of deliverables are produced through the life-cycle of a project

Strategy: A document defining the types of testing to be applied; the criteria that allow you to start or stop testing. The shaping activities such as risk identification.

TTRM: Traceability to Requirements Matrix, providing traceability from each test case/scenario back to the original requirement specification. Proving that testing is covering the full set of specifications.

Test Plan: A document defining the type of testing to be performed, the test cases/scenarios to be covered, when and by whom.

Test Schedule: A project management type schedule showing all activities performed by what resources, at what time and for what duration. Further dependencies on other activities will also be included.

Test Scripts: Detail of the tests to be performed, the steps to go through and the expected results, with reference back to the test cases/scenarios.

Defect: Detail of each script that has failed, why it has failed and under what circumstances. The priority of the defect, it’s severity to testing and additional information allowing the defect to be replicated.

Completion/Exit Report: A summary of the findings of the testing team, showing evidence of issues and stipulating an opinion of the readiness of the item under to test to be placed live.

Regression Pack: A suite of scripts that can be reused to test the primary functionality of the item in the future.

To summarize, we believe that testing is a discipline in the same manner as analysis, architecture, design or development. It helps the client understand the risks associated with the release of a newly developed or modified application. It is an essential means of introducing quality into the software development life-cycle. It protects the users of the system, looking to ensure that IT is delivering what was requested. Pre-defined outputs are produced naturally throughout the engagement.

TCL COMS

2008-05-14T17:52:00.018+05:30

TCL COMS (Consultancy led Offshore Managed Service) offers our customers the perfect marriage of our consultancy capability with the economic advantage of global delivery units. It represents the way in which TCL believes offshore should be best utilised and brings considerable industry experience to bear in its execution. There is a focus on ensuring that the management of the offshore capability is seamless and near invisible to the customer in terms of communication and management.

Onsite teams are made up of local and offshore resources, with the latter having a minimum of 4 years experience in the country of delivery. This provides the customer with a consistent onsite communicative presence that is capable of overcoming any issues that may arise offshore. We also advocate the use of messaging services, such as MSN to ensure that key communications are recorded and can be referred back to. TCL COMS is driven by a clear communication process, up and down through the management chain, and back and forth through peer level contact. In all instances the implementations are performed by resources familiar with distributed working practices and that they are able to demonstrate a successful track record in such environments.

In the earlier diagram, the area surrounded by red dashes can be fulfilled by either the client or TCL resources. In this manner, the client can either make TCL accountable for the entire project testing, or can retain control and use the TCL resources purely to facilitate an offshore capability.

The solutions are further enhanced with the addition of a dedicated Offshore Delivery Management Team with representation in the UK and India, which ensures efficiency in responding to demand, issue resolution and service improvement. We believe that whilst there are financial advantages to placing work offshore, roles which require high levels of communication are better resourced from the locale in which the role will interface. To this end a blended solution is offered, where prime roles, that of the Account Manager and Test Manager (s), will be sourced from the United Kingdom. Other roles may be sourced globally in order to offer high skill sets at economic cost.

As a preference, TCL work towards an optimisation of resource utilisation as defined below. It is recognised that some activities lend themselves to being performed offshore, but that there is always a need for onshore presence and communication. India is far away and the customer can often feel that they are too remote. This is especially important at the start and end of a project. By managing the testing activity carefully and over a period of time, this blend of resources should be achieved. The time taken to achieve this will vary from client to client and project to project.

The resources applying the effort change over the life-cycle of a project, starting and ending with a heavier focus on management, to define and shape the piece of work and increasing the volume of engineers to take on the more prescribed roles of scripting and execution. The areas where the majority of effort occur are the Preparation and Execution phases and these are also the areas where we would have the majority of the effort offshore.

The team will naturally assume a set service structure as defined below. This looks to ensure that a core presence is maintained both onshore and offshore, with a management layer. This team is then flexed to cope with demand from the client, where the majority of resource increase or reduction is handled offshore. This ensures consistency in the touch points onsite for the project, but also that communication paths once established with offshore remain constant.

Those resources which remain constant will be responsible for the interfacing with the project and ensuring that the testing remains on track, that resources flex as is demanded by the client and that the work undertaken remains on target, both in terms of financials and time.

Continuous improvement is key to the way TCL have grown and worked and this is something that we look to bring to all client relationships. TCL COMS looks to implement service structures that deliver to the clients requirements and then surpass expectations. This is a highly professional service geared to deliver and excel.

Metrics & KPIs

2008-05-14T16:20:00.009+05:30

Metrics and KPIs (Key Performance Indicators) are a key aspect to any successful Test Function and to any service delivery. TCL India recognise the importance and can deliver Metrics and KPIs as a natural part of their service capability, bringing them together to form a balance scorecard. This accumulation of data is sometimes referred to as a dashboard and is designed to provide a single visual display of how the service is performing against expectation.

The TCL India Scorecard is broken into 7 key areas, although more areas of interest can be added if the client requires. These are the key components that are considered to be essential to the measurement of the service and the Testing it delivers.

Estimation : Accuracy : Overspend Areas : Environments : Code Delivery : Design : Requirements

Cost : Blended Rate : Cost per Defect : Onshore vs. Offshore : Cost per Project : Scale : Complexity

Responsiveness : Acknowledging Resource Requests : Identification of Candidates : Placement of Candidates : Staff Replacement : Issue Resolution

Satisfaction : Survey Responses : Project Feedback : Value Add : Innovations

Deliverables : Quality : Timely Delivery : Appropriate Sign Off : Resource Quality

Development Quality : Unit Testing : Code Delivery : Accuracy of Code : Defect Turnaround

Effectiveness : Defects by Phase : Root Cause : Analysis : Defects in Live : Script Coverage

These metrics are grouped together and provided to the client in a single diagram. The rings represent the targeted achievements for each of the primary areas, converted into a single scale. The points of the star are representative of how close to achieving the target the service has achieved. This means that the rings can be set at the required level for a period of time, or adjusted monthly, driving through change within the service.

Regression Testing

2008-05-14T15:35:00.006+05:30

The SIGIST “Special Interest Group In Software Testing”, have a glossary of terms located at http://www.testingstandards.co.uk/living_glossary.htm

Here we find regression testing defined as: “Retesting of a previously tested program following modification to ensure that faults have not been introduced or uncovered as a result of the changes made”

Regression Testing is a key component of any project and should be the final test performed prior to release to the production platform.

So what comprises a regression test?

The best way of describing this, is that an application which has been previously released, will be in a known state. Tests will have been previously written and executed against this and a selection of them will have been extracted to form the regression test pack. The tests would look to exercise as a minimum the core functionality of the application, in a positive manner. In this way, whenever a change is made to the application in the future, the regression pack can be run to ensure that faults have not been introduced or uncovered in functionality that was previously deemed to be working correctly. The regression pack should be updated as each new version of the application is released, building in new functionality to the suite of regression tests and maintaining it in line with the applications capabilities.

Some debate occurs as to what a regression test should comprise and it is fair to say that the larger the regression suite, the more confidence will exist in the new version of the application, but there is a balance to be achieved. The more tests, the longer they take to execute and the higher the cost to change. Pragmatism must be applied and rather than testing every permutation, coverage should ensure that the functionality of the application has not been impacted by change. Tests which apply different data to the same function, access the same function from a different point or negatively test the function (trying to break it), do not need to be part of the regression suite.

Experience shows that there is a distinct need for regression testing. Problems with Configuration Management or code control are often identified during regression, where previously fixed problems are re-introduced. Code which is common to different modules within the application is also a primary area of difficulty, as a change for the benefit of one module may adversely impact another.

It is key from a regression point of view, that no test is performed that has not been run successfully on a previous occasion. In a similar manner, no code should be regression tested which has not been previously tested.

A recent experience we had with webMethods regression testing was where, due to the nature of webMethods, several projects were being released together and the webMethods code was being converged and tested for the first time during the regression test. Because the convergence was seen as something over and above the need of the projects, it was passed onto a different team to build and a different test team to regression test. Not only was this an inaccurate use of regression testing, but the testers with the most pertinent knowledge around what was expected, were not involved.

As testers we must protect the regression testing activity. Ensure that it is full, up-to-date and complete. Failure to run regression properly increases the risk of faults in production.

Functional Testing

2008-05-08T23:00:00.013+05:30

Our primary offshore service is functional testing. Why?
· The progress can be systematically measured
· Prescriptive testing , easily packaged
· The testing is as good or better than the inputs received
· Communication is handled via reporting and defect identification
· Interfacing with other project members can be obviated

According to IEEE90, “Functional Testing ignores the internal mechanism of a system or component and focuses solely on the outputs generated in response to selected inputs and execution conditions.”

In this way, many of the perceived problems with offshore testing are removed from the equation. It is our proven intention that the hurdles, obstacles and reasons for failure of offshore are removed.

Typically 75% of testing effort is based around the production of test scripts and their subsequent execution. Our Functional Testing capability will maximize the benefits of offshore. With offshore prices starting as low as £75 per day, in comparison to permanent resources costing £250 per day. This gives a 70% saving to three quarters of the testing effort.

If this is put into perspective, a project that has a UK testing budget of £50,000 would spend £37,500 of its budget on scripting and execution. This can be reduced to £11,250 using our offshore setup. This means that the total testing budget would be £23,750, less than half that which was originally estimated. The beauty of the solution is that all of the management activity and the governance around the testing can remain onsite, where it can be controlled and interfaces managed. The offshore activity becomes near invisible.

The process is simple, a series of inputs are required from the project. These are normally in the form of a Project Plan, High Level Design, Requirement Specification and Functional Specification. These are not mandatory, but the more of these the better the project is suited to offshore effectiveness.

From a testing perspective, the Test Plan, Schedule and Scenarios should be provided. Again, these are not mandatory and can be produced offshore if required, but in order that control is retained in the UK, the controlling documents should be specified and owned there.

Once all of the inputs have been received or created, the process of producing the scripts can begin, based on the test plan and the test scenarios. Reporting now begins in terms of the anticipated volume of test scripts and how many have been produced against this. This phase is often referred to as the Preparation phase. During this period the onshore project team should be looking to ensure that the test environment is ready and available to the offshore team. Progress can be reported on a basis that makes the client feel comfortable, daily or weekly, depending on the length of the work.

As execution nears commencement, resources will look to smoke test, or prove that the environment is ready for testing. Once testing commences, defects will be logged in the client prescribed tool, or if not provided, we will adopt a suitable mechanism for defect capture. Statistical data will now be produced on a daily basis and provided to the client, ensuring full awareness of issues as they arise.

On completion of the test execution, the team will generate a functional test completion or exit report, showing the full details of the testing that has been performed and the results of that testing. An executive summary is included which specifies our opinion on the projects readiness for go-live. All test assets will be delivered to the client with scripts highlighted as forming the regression test pack. This ensures that all of the intellectual property associated with the project is returned to the client if they desire. It is our intention to ensure that the client is free to choose how future testing should be performed and not tie them into a service that they may not want.

TCL India Introduce Web Check

2008-05-08T22:16:00.006+05:30

Over the last couple of months we have developed a new solution, called Web Check, designed to service a clients web site and ensure that the user experience associated with its use is as the client would expect it and as both the client and the user would like it to be.

Web sites are now a medium often used by organisations as their prime marketing tool, reinforcement of their brand or as a means of communicating internally or externally. Some organisations only exist through their web site. Therefore the importance that this brings to the owner of a web site is potentially, extremely high and could result in the success or failure of their business, or a large proportion of it. Because of this, web sites are often changing, with new products on offer, new goods to sell or a new message to put across. Some of these changes are made by means of adding new software or functionality and a lot are handled by making changes to content. The reality is that web sites grow, alter and adjust regularly and that often, finding the time to ensure that everything is as it should be, is difficult.

When things are important to us as individuals, like cars, we take care of them by performing regular services, to ensure that they are safe and will perform the next time they are needed. We do not like to have failures occur. A breakdown, be it at home or on a journey, is something to be avoided. Web Check has been designed to perform a service of a web site, giving the site owner peace of mind that it is working correctly and that the user experience is a positive one.

Web Check looks at key attributes of each web site and by employing formal testing techniques looks to ensure that these work as expected. The output report identifies all of the checks that have been performed and the results of those checks. These are translated into a single quality percentage against the checks that have taken place, providing a singular view of quality achieved.

The TCL Vision - Where will we be by 2020?

2008-02-08T20:35:00.000+05:30

We have a Big Hairy Audacious Goal (BHAG) at TCL, which is to say that we have a dream. A dream of what the company will become and we've set out to achieve it by 2020. It's not the only dream the company will ever have, but for now it's the priority one and it's the one that is very personal to the founders, shareholders and owners of the company.

So what does this dream look like? The short answer is that there is no short answer to that question. Like most dreams it's a little hazy when you wake up, but the outline looks something like this:

We want TCL to become a world wide, world class, centre of testing excellence. This means that we want to establish TCL as a Centre of Testing excellence which has world class solutions and we are able to deliver them to any industry in any country.

When we have reached that goal, the company will look something like this:

Our company will benefit its employees and their families. They will benefit financially, emotionally, ethically, technically and physically through the way the company is run.

Our company will be financially robust and secure. We will have sufficient alerts and contingency plans in place to ensure we can survive economic peaks and troughs whatever the root cause.

The communities in which we operate will also benefit from TCL. We will actively make efforts to be involved in those communities and the issues that are important to them. We will actively demonstrate in our social conscience and encourage it in our team and the Clients and Suppliers with whom we engage.

We will also ensure that some of our profits are given to worthwhile charities. These are charities that are important to the people in our company.

Our culture will be totally inclusive, with all levels of the organisation being informed and involved. Each person in the company will have a personal understanding and empathy with their colleagues in relationships that demonstrate integrity and respect at every juncture. There will be awards each year for long service and through this long service we will ensure a continuity of culture and corporate memory at the foundation of the company.

A team that supports and focuses on the vision will regularly introduce new ideas to keep the company aligned with the purpose, values and BHAGs and will be a proven source of information and inspiration around the company. Competition to be part of this forum will be high, and it will be necessary for the participants to undergo three month secondments to the team to get the work done.

The TCL solutions will be aligned and accredited with organisations such as ISO and the BCS. Our method SMaRT will be recognised as a management model and be used as a template and standard in blue chip organisations around the world.

Our employees will be proud to belong to TCL and will contribute to the company progression each day with energy and enthusiasm.

We will be using the latest technology and involved with the strategic decision making process with each of our clients having demonstrated consistently our ability to add value to that process.

Our reputation will be such that there will be healthy competition to join TCL, and the selection process will be rigorous to ensure that only people who display the core values and have the aptitude to support and contribute to the TCL vision are involved.

We want to take testing to a new level. A professional discipline that is consistently recognised for its value and resourced, planned and funded accordingly within all development lifecycles.

Our Research and Development capability will lead the way with innovative and intuitive solutions that are derived from the latest theory, technology and thinking across the world. We will involve and include academic research and consideration as well as commercial best practice.

Our solutions will have been benchmarked with companies in five continents and proven within more than ten industries including Defence.

We will be the testing partner of choice for Safety Critical Applications and SIL levels 3 and 4.

We will have offices on three continents and research and development capabilities at each. These will have links to Universities and we will provide some financial support and assistance to these universities in the areas of research that will further help the TCL purpose.

Our presentations at industry forums will be actively sought and solicited and our representatives praised for their credentials, innovation and inspirational style of presenting.

When a project declares that their testing will be conducted by TCL it will be perceived at all levels of an organisation that this is the best way to go and that both cost and quality will be in ideal balance for this delivery.

We will be seen for our Quality and Innovation but we will never be seen as overpriced - simply realistic for the demonstrable value we add.

Now, there are lots of things left to think about with regards to how TCL will look in 2020, and each year we challenge the dream and things become progressively clearer as to what we can achieve, what the market wants us to achieve and where things are all going. I hope from reading this article you can already see that some of the things are here now, and some are on there way - in fact you may be involved in making it happen. The spirit of the vision is alive and well. But I suspect that you might also see that some things around the company aren't quite there yet - and from reading this today you will know that we don't want them to stay like that. It just isn't our dream.

I truly believe that if you understand this dream, then you will understand the essence of what TCL is about and what the founders of the company are trying to achieve. As the "custodian" of the vision, I welcome any thoughts you have on this dream, this BHAG, of ours and warmly invite you to call or email me to discuss things. You can also take advantage of my time at roadshows, company parties and promotion days to talk to me in person.

Stewart Noakes

Chairman

Transition Consulting Limited

2007-12-12T23:34:00.000+05:30

Bangalore (officially Bengaluru) is a colourful and vibrant city, having been built at 3000 feet it has a temperate climate all year round. The history of the city goes back to 900 CE but is formally recorded from the 1500s. With a population of over 5 million Bangalore is the third most highly populated city in India.

We have chosen Bangalore to be the centre of operations for TCL India for many reasons. We have a good network of contacts in the vicinity, it is recognised as the IT Hub of India and attracts many resources to it's populace. It has good schools and universities generating a rich vein of future employees for our business; an area I am sure we will explore once established, will be working with some of these in the same manner that TCL work with Exeter and Bristol universities in the UK and Boston univeristy in the states. More importantly we have a track record of working successfully in India and Bangalore. Other markets are emerging but India is still way ahead of the competition. Areas like China are already in contention, but for the immediate future, India is still leading the way. We are looking to return to India as soon as the formal business concern has been created, which should see us there in February 2008. From that point onwards we will be returning on a quarterly basis.

2007-12-12T22:58:00.000+05:30

Hi, my name is Grant Obermaier and I am working with Mick on the creation of TCL India. I have been working in the IT industry for over 20 years, the last 6 of which have been in the software testing industry. Mick and I have been working together throughout the last 6 years and we are bringing two sides of the equation together, myself from a customer perspective and Mick from that of a supplier. I have managed teams of over 150 test resources delivering a vast array of projects from a testing perspective.

Over the course of the last year, under the banner of Project Nirvana, we have successfully delivered onshore and offshore programmes. Our offshore efforts have resulted in client commendations and repeat work.

I have seen offshore work to great effect over the last few years, delivering massive benefits to the client. In a time where it is hard to find the right resources for testing, India is delivering vast numbers into the workplace and we will be capitalising on this going forward.

2007-12-12T22:42:00.000+05:30

Hi, my name is Mick Morey, I have been working in the IT sector for the last 15 years and have spent the last 7 working in the Software Testing industry. I am the Managing Director for Transition Consulting India PVT Limited (TCL India). This business is being operated by myself and Grant Obermaier and between us we bring a wealth of knowledge in the offshore arena. Over the last year Grant Obermaier and I have been working on creating a flow of business into Transition Consulting Limited in the UK, through a project called Nirvana. The success of this project now enables the formal creation of a sister business in India.

I will look to update the blog on a monthly basis and let all of the readers know what is going on and how we are progressing. This is an exciting journey and time in our lives and I look forward to sharing more of this with you. If you have any questions or would like to know more about the services we offer, please e-mail me at Mick.Morey@TransitionConsulting.co.uk.