SOFTWARE TESTING GLOSSARY (Update)

This glossary is a living post – so will be edited as we come across terminology that is not included. If you have any suggestions or disagree with an explanation, drop us an e-mail and let us know. TCL India offer this as a means of establishing glossaries of your own or as a point of reference. This update includes 10 new definitions.

A…………………………………………………………………………………………………

Accessibility: (Testing)Looks at an application to ensure that those with disabilities will be able to use the application as an able individual.

Agile: A development method, involving the creation of software, with the contributing parties, including testing, all working on the same item at the same time.

Alpha: (Testing)Testing of an application which is in production, but not available for general use. This is normally performed by users internal to the business.

Analyst: Person who interacts with the business in order to understand and document their requirements for an application or change to an existing one.

Analyst (Test): Person responsible for the preparation and execution of test scripts, recording and progressing of defects, reporting into the Test Team Leader or Test Manager.

Automation: The process of developing a script or program which can be run by software rather than a Test Engineer, to test an aspect of an application. Automation is used to increase the speed of test execution.

Automation Centre: A tool used in the automation of software testing. Seehttps://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-127-24%5E1074_4000_100__&jumpid=reg_R1002_USEN

B……………………………………………………………………………………………………

Beta: (Testing)Testing of an application which is in production, but not available for general use. This is normally performed by a select group of friendly external testers.

Black Box: The process of testing without understanding the internal workings of an application, relying on inputs and outputs only.

Bug: See DefectBusiness Requirement Specification: See Requirement Specification

C……………………………………………………………………………………………………

Case: See ScenarioCode: The software that has been produced by development and is being subjected to testing.

Completion Report: A document produced during the testing and presented to the project manager on completion of testing activity. The document should detail the tests that have been performed along with the associated results and defects. Some reports may include a recommendation for the applications suitability for release to production.

Component: The smallest item that is testable or producible. This often refers to a single file of code.

Configuration Management: The means of managing the components required to make a larger item. Applicable to files, documents, data, hardware, software, tools. Understanding of the versions of each component required in order to be able to re-build the larger item.

Criteria: See Entry Criteria and Exit Criteria

D……………………………………………………………………………………………………

Data: Information pertaining to the use of a system or recorded through the use of a system. Data is used in order to test a system, potentially after data cleansing if personal information is involved.

Data Generator: A tool used to generate high volumes of data in order to be able to test many permutations, or to load test an item.

Data Protection Act (DPA): The act that determines how personal data should be used and protected from abuse.

Data Scrambling: The process of altering personal information from data to be used for testing purposes.

Defect: Where the item under test has been found to inaccurate, resulting from testing. Primarily used in associated with software, but equally valid for static testing of documentation.

Defect Detection: The means of identifying a defect. Can be a metric used to predict the volume of defects expected during the course of a project, or as a means of looking back at a project to understand where testing needs to be concentrated in future projects of a similar nature.

Defect Removal Efficiency: A metric used to assess the ability of testing to remove defects as they are introduced, during the software development life-cycle, keeping the cost of testing later phases to a minimum.

Defect Turnaround: The time taken from the identification of a defect, through to the point of resolution. Different levels of granularity may be used. e.g. A focus on the time taken by development.

Developer: A person responsible for the development of an application.

Development: A process of producing an application by production of low level design, code, unit testing and integration in the small testing.

Dynamic: Testing which occurs on the right hand side of the V-model, with the application present.

E……………………………………………………………………………………………………

Environment: The combination of hardware, software and data as used in development, testing and production. The platform/s upon which the testing occurs.

Entry Criteria: The criteria that must be met prior to a deliverable entering the next phase of testing to another. This is normally associated with documented test assets and pre-agreed volumes of defects.

Error: See Defect

Exit Criteria: The criteria that must be met prior to a deliverable leaving the current phase of testing. This is normally associated with documented test assets and pre-agreed volumes of defects.

Execution: The process of working through a script on the application under test, in the testing environment.

F……………………………………………………………………………………………………

Functional: (Testing) The testing of a products function, against requirements and design.

Functional Specification: A document which extracts all of the functional requirements from the requirement specification.

G……………………………………………………………………………………………………

Glass Box: See White Box

Grey Box: Testing performed by testers with some knowledge of the internal workings of an application. See also Black Box and White Box testing.

H……………………………………………………………………………………………………

High Level Design: A design showing the major components and how these will interface to each other, defining the hardware to be used and the software that will be impacted.

I…………………………………………………………………………………………………… 


Integration in the Large: Where the application or applications that have been developed are brought together along with those which have remained unchanged, building a production like system around the application/s. Testing is then applied looking at the communication between the different applications.

Integration in the Small: Where the components of the application that have been developed are brought together along with those which have remained unchanged, building the application or major component of a single application. Testing is then applied looking at the communication between the different components.Integration: The act of bringing many parts together to form a whole.

ISEB: Information Systems Examination Board. This was historically the board that was used to certify test professionals at either Foundation (Entry Level) or Practitioner Level (Advanced). See:http://www.bcs.org/server.php?show=nav.5732

ISTQB: International Software Testing Qualifications Board. See:http://www.istqb.org/

J…………………………………………………………………………………………………….
K……………………………………………………………………………………………………

Key Performance Indicator: A mechanism built on one or metrics, which determines a band of acceptable performance, which over time is often targeted towards improvement.

L……………………………………………………………………………………………………

Live: See Production

Load: One of the types of performance testing, this looks at testing for the maximum load on the system.

Load Runner: Tool used to performance test one or many applications, to understand how it handles increases in load. See:https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-15-17%5e8_4000_100__

Low Level Design: Definition of exactly how the application/s will be modified or produced in order to meet the requirements and the high level design. This can extend in some examples to elements of pseudo code being defined.

M……………………………………………………………………………………………………

Metric: A measure of an attribute or occurrence in connection with an organisation, department or functions performance.

N……………………………………………………………………………………………………..

Non-functional: How an application performs, rather than how it does it.

Non-functional Specification: A document which details the non-functional requirements such as performance, security, operational elements.

O……………………………………………………………………………………………………
P……………………………………………………………………………………………………

Performance Centre: A tool used for measuring the performance of an application or series of applications. See:https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-126-17_4000_100__

Performance: Used to describe many types of testing relating to the speed of an application. See Volume, Load and Stress.

Plan: A document produced for a type of testing defining how the testing will be performed, when the testing will be performed and who will be performing it. The test plan lists the test scenarios that will be scripted.

Preparation: The process of generating the test scripts.

Prince2: A project management methodology that is used amongst a lot of blue chip organizations to bring discipline to the software development life cycle.

Priority: The importance of fixing a defect from a business perspective. Defined by business representatives.

Production: The area or a computer network that contains applications which are in use by real users and contains real data. The area that applications are released to on completion of a project.

Q……………………………………………………………………………………………………

Quality: The suitability of an item for its intended purpose and how well it achieves this.

Quality Centre: Tool used to assist with the management of testing, recording and tracking scripts, logging and tracking defects and more. See:https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-127-24_4000_100__

R……………………………………………………………………………………………………

Regression: Retesting of a previously tested program following modification to ensure that faults have not been introduced or uncovered as a result of the changes made.

Requirement Specification: A document normally produced by a business analyst, capturing the needs of the individual in a manner which means that they can be translated into a software solution.

Re-Test: Taking a defect which has failed and executing the associated test script again.

S……………………………………………………………………………………………………

Safety Critical: Used to identify something for which the use has an impact on personal safety, such as medicinal applications or those used by rescue services.

Scenario: A high level description of how a requirement is going to be tested.

Script: Referable back to the scenario, the script defines each step that must be passed through in order to perform a test, along with the expected results. As the script is executed, results are recorded and if they match the expected result are marked as passed, otherwise as failed. A script containing a failure should have a resultant defect raised.

Schedule: A document, similar to a project plan, but detailing the activities associated with testing, when they are due to occur and who will be performing them.

Severity: The importance of a defect to testing and the application. Defined by Testers

Special Interest Group In Software Testing (SIGIST): A group that has been set up by the British Computer Society, made up of people interested in the subject of software testing.

Smoke: A process of proving an application or environment is ready to commence full testing, by running a sample set of scripts testing the primary functionality and/or connectivity.

Software Development Life Cycle (SDLC): The process of taking a requirement and translating it into a fully working application.

Static: The process of testing without the presence of software. Normally refers to the testing of documentation.

Strategy: A document produced at project or programme level, defining what testing is to be performed.

Stress: A form of performance testing, whereby the volume of testing is increased to a point at which the application is deemed to be failing to perform, either due to failure to meet performance criteria or system breakdown.

T……………………………………………………………………………………………………

Technical Architecture Document: Definition of how the business requirements should be translated into a technical solution at the highest level.

Test Bed: A cumulative term used for all of the test environments used by the test department.

Test Director: A tool used for test management, capture of requirements, scripts and defects, with the ability to manage defects through to resolution. Now replaced by Quality Centre.

Test Maturity Model (TMM): A means of measuring a testing department’s level of maturity in a similar manner to that of the capability maturity model (CMM).

Testing: The process of reviewing an item for accuracy and its ability to complete the desired objective.

U……………………………………………………………………………………………………

Unit: The testing of the application by the developers at a component or modular level, looking at whether the code is performing as expected.

User Acceptance: The means of testing applied, normally by members of the business or recipients of the system, whilst still in the test environment. This looks to ensure that business requirements have been understood by all involved in the application production as well as providing an opportunity to check the business processes that the application will be used in conjunction with.

V……………………………………………………………………………………………………

V-Model: A testing focussed software development life-cycle where the process of creativity occurs down the left side of the V and the process of testing up the right side of the V, where each step down the V has a corresponding point of testing up the right of the V. Begins with a requirement and terminates with User Acceptance Testing.

Version: An alpha numeric means of identifying a particular build of software, or a component thereof. The version changes incrementally to reflect each modification of the software, component, document etc.

Volume: (Testing) A type of performance testing which increases the volume of users on a system, in each cycle, taking the volume up to a prescribed limit, normally exceeding optimum load.

W……………………………………………………………………………………………………

W3C: World Wide Web Consortium – body creating web standards and guidelines. http://www.w3.org/

WAI: Web Accessibility Initiative. See: http://www.w3.org/WAI/

Waterfall: Project Management Method whereby each element of the software development life-cycle occurs sequentially.

Win Runner: A tool that was historically used for test automation. See Automation Centre.

White Box: Testing normally performed by developers, looking at the code and ensuring that each model of code is performing as expected. See Unit

X……………………………………………………………………………………………………

Y…………………………………………………………………………………………………….
Z…………………………………………………………………………………………………….

How to Annoy a Test Manager

A little light relief from all the serious blogging, but some of these do really get me angry.

The coding has finished and we need to test – should we be talking to you now?

I know you were expecting the code today, but when do you really need it?

Can you please stop raising defects as we don’t have time to fix that many problems?

We have only got half the time we thought we had for testing so how are you going to get all of your work done?

We know the quality is bad, but we are releasing anyway?

UAT has been dropped because the business can’t free any resources.

The fact that this problem was fixed in an earlier release is not indicative of poor configuration management.

Why do we need developers, support and environment resources to work the weekend as well as testers?

The environment is nearly the same as live, we just don’t know how nearly!

Testing is supposed to make the software perfect. Why hasn’t it worked?

We have spent 20% of our budget on analysis, 60% on design and development and 10% on other items. You have 10% left, what can you do for us?

We are running out of money because the test execution is taking too long, so we are going to let the testers go and the developers can finish the job.

Don’t worry about getting testers. We have lots of people in the business that we can get to test at lunchtimes.

The supplier was not asked to perform Unit testing and we can’t afford to have it done now as it was not in the budget.

Testing is a service and as such we will let you know when we need you.

I know you asked for three environments, but we can only get one. Don’t worry. I am sure everyone can get on it at the same time.

The testing is occurring on the development environment. That way the developers can change things whenever they need to.

The problems in live are fixed out of hours so we can’t test them.

If we automate all of the testing how much time will it save us?

Why do we need testers on board before the testing starts?

The user spilt coffee on their system and it crashed. I bet you didn’t test for that did you?

We have too many high severity defects, so we are going to hold a meeting and see how many we can downgrade?

We need to do some performance testing. There are no non-functional requirements, so let’s just assume that everything must happen in under a second.

Testing is easy. Anyone can do it.

It’s not a bug. It’s a design feature.

The requirements were captured at the start of the project. But don’t read that it’s way out of date.

Please feel free to add your own by way of comments.

Grading Exit Criteria

For the sake of this article, let us assume that we have 5 different grades of defect that start with insignificant going on to, minor, significant, major and ending in critical. When we come to exit one phase of a project and enter the next, we need to recognise that criteria will be set by the test manager, which need to be achieved. It is normal that the exit criteria from one phase, becomes the entry criteria for the next phase. For instance, a Test Manager for “Integration in the Large” testing will set his entry criteria, which by default becomes the exit criteria for the phase before. This remains true up to the point of release to live.


What I would like to draw attention to, are two key points. The test manager that says I am not going to have any defects remaining open at the point in time I go live, is living in “cloud cuckoo land”. This is the test manager that wants to be able to test forever and release perfect code. This does not happen. The reality of business is that certain levels of risk become acceptable in order to deliver the project to live and meet the requirements of the business and deliver the business benefits that were used to justify the project in the first instance.


So the next question becomes one of levels of risk and what is likely to be acceptable. This will depend on the nature of the project. One that involves safety critical applications is likely to have a far higher level requirement for quality, than one which is being used by 2 people in a small business. One which is exposed to the public or clients is also going to need to be of a significantly higher level of quality than one which is used internally only. The skill of the test manager is to assess the application, its’ use and define the level of defects that are going to be tolerable. As previously defined in http://tcl-india.blogspot.com/2008/06/entry-and-exit-criteria.html the Project Manager should be bought into these levels in order to ensure support later in the project.


Now we need to look at the defects that are going to be acceptable. A common misconception is to identify that the level of critical defects is 0, be that priority or severity, but that there is no restraint on the level of insignificant defects. This could not be further from the truth. Whilst 1 insignificant defect is not going to stop the release, there comes a volume of insignificant defects that makes the release of the application unacceptable.


I would suggest that we concentrate initially on the severity of defects. We need to understand proportionately that a volume of insignificant is equal to 1 minor, a volume of minor is equal to 1 significant and so on. This will change dependent on the application use, so here are some suggestions for different application types.


Safety Critical
Proportions: 1 Critical = 2 Severe : 1 Severe = 3 Significant : 1 Significant = 5 Minor : 1 Minor = 10 Insignificant
Final Exit Criteria: 0 Critical : 0 Severe : 0 Significant : 5 Minor : 10 Insignificant


Public/Client Facing
Proportions: 1 Critical = 3 Severe : 1 Severe = 5 Significant : 1 Significant = 10 Minor : 1 Minor = 20 Insignificant
Final Exit Criteria: 0 Critical : 0 Severe : 3 Significant : 10 Minor : 20 Insignificant


Internal Consumption (20 + Users)
Proportions: 1 Critical = 4 Severe : 1 Severe = 7 Significant : 1 Significant = 15 Minor : 1 Minor = 50 Insignificant
Final Exit Criteria: 0 Critical : 1 Severe : 5 Significant : 10 Minor : 40 Insignificant


Internal Consumption (0 to 20 Users)
Proportions: 1 Critical = 5 Severe : 1 Severe = 10 Significant : 1 Significant = 20 Minor : 1 Minor = 100 Insignificant
Final Exit Criteria: 0 Critical : 2 Severe : 5 Significant : 10 Minor : 50 Insignificant

Please bear in mind that these are indicative and that the best solution is for the test manager and other members of the project team to determine the levels between them.

Testing in Live – Why you should not!

As a tester I am often asked why we can’t test in live. There are so many positives that can be gained from it that it surely makes sense. After all, the environment is exactly the same as live, the data is real and the volume of users and traffic is real.

Production is designed to serve a business in operation. It is being used in anger every minute of the day. It may be internally used, it may be used by high street shops or it may be open to public use such as the internet.

The first problem is that the code that needs to be tested is not fit for live because its quality is not known. So by placing the code live you are compromising the production platform. You can cause code which is running perfectly in production to fail, as common routines are modified.

Testing is designed to break things. We need to understand the points of vulnerability and testing tries to identify these. This means that the application could be caused to crash, which could result in a domino effect and take out other elements of infrastructure or code.

In testing terms, the point at which resolving a defect is highest, is in production. One of the primary reasons for this is that outages which affect the production platform can be disastrous. Imagine 1000 customer services personnel being unable to work for an hour. A hourly pay rate of £10.00 per hour and the problem has already cost £10,000.00 But what if you have an on-line presence and you lose that for an hour. You have now lost sales revenue for that period of time and this you cannot recover. Perhaps more damaging is that perception of your organisation has taken some public damage. The potential new customer went somewhere else and you have now lost their trade for the foreseeable future. The existing client who was returning has now been forced elsewhere.

Think also of the press that can be received through public outages. Organisations that experience loss of service will often find this being reported in the media and this causes immeasurable brand damage.

Another consideration is that the data which exists on your production system is real. The testing therefore cannot modify, add or delete information without careful consideration of the consequences. Corruption of data exposes an organisation to the data protection act, but worse, may invalidate information which is crucial to your business. What happens if a change of address occurs which is a test – the client is potentially lost along with their business?

A final point is that your system is in all likelihood linked to other organisations. When I first became involved in development, I had a story relayed where a developer had unit tested his code in a live environment. His code was a credit checking piece of software and he used his own details each time he ran the test. As your credit score is negatively impacted each time a check is performed, the developer managed to take his credit rating from positive to negative in a matter of a day. He did not realise he had done this until such time has he tried to make a credit purchase and was refused. Fortunately, after some discussion with the credit reference agency, his rating was restored. But any kind of financial transaction must be performed in a test environment, linking to other test systems. Imagine the damage that re-running a BACS transaction could cause, or payroll being sent out at the wrong time.

Production is very much live. You would not expect a doctor to test a new drug on a human without going through a certain degree of process first. Even then you would move into a clinical trial under strict process.

What about Beta testing I hear you ask. Yes, there are examples when the software is deemed to have no negative impact on production, where sufficient testing has already been performed as to be confident of its capabilities, that the software may be released with a “Health Warning” attached. It may be that a select group of users will be asked to perform a trial. But in these instances, the application will have been put through its’ paces in a test environment.

It is important that users of systems have good experiences. Failure to achieve this results in loss of the users from the system, negative impact to the reputation of the department and individuals making the release, plus the potential to damage the organisation's brand and revenue-earning potential should make this a course of action that is avoided.

Reinventing the Testing Wheel (Object Oriented Testing)

I have been thinking about this one for some time and wondered what other views existed. For some time developers have taken a more modular approach to programming, applying techniques such as Object Oriented Design, in order to improve efficiency. Why re-invent the wheel each time a certain piece of code is required? Take something like a calculation of VAT. Once this has been written the first time, that should be enough and if stored and indexed correctly, each time someone else needs a VAT calculation they call a pre-written routine from a library. Not only does this mean that the developer does not spend valuable time writing the same code again, but also that the module should already have been tested and therefore be reliable, reducing the likelihood of errors.

This was seen as a major break through in the development world, but one which has not been replicated to my knowledge in the testing arena. When we write a manual test script, we write it in a manner which is long hand. So how many times have we written scripts to test VAT calculations? Why are we not creating a library of scripts that are re-usable by all testers? Why do we have to sit and think each time that we test a field: to understand what all the permutations are that we need to deal with, how to test it positively and negatively? There are hundreds of thousands of testers all over the world that are repeating the same thought processes, writing the same scripts, duplicating their efforts.

How many times has an input of a person's age been tested? What are all of the parameters? How old before they are at school, how old before they can hold a credit card, bank account, retire? All of these questions have been thought about so often, but we still religiously recreate them each time. So testers are accused of being too slow or not moving with the times, being overly expensive. Well I suppose when you look at something like this, the answer has to be, "Yes".

So when we are asked what the future may hold for testers, I put on my rose-tinted spectacles and imagine…..

I imagine a world where we are sharing our knowledge and expertise, generating a global testing community and capability. One where we are seen as slick operators, contributing massively to technology programmes the world over, with the minimum of input for the maximum return.

How does this happen? If there were a central repository for scripts to which every tester contributed, where we went as a first port of call to locate our script components, or store those components we have to create, we would quickly generate a wealth of stored expertise. A Wikipedia of test scripts – wow! We would need to be able to locate scripts in a fast and efficient manner and the database used to store the script components or objects would need to be referenced in a multitude of ways. But this is not new technology. Developers have been doing a lot of this for years and we can no doubt learn from the mistakes that they made along the way.

I would welcome comment on this. Is it feasible? How could it work? Am I barking mad? If you are a developer and have a story about OOD then share it and let us know if you think this is feasible. Drop me a line and let me know.

Site Under Construction!

In the current climate of the credit crunch and with so much positive press around the buoyancy of the internet, more organizations are looking to the World Wide Web to maintain or increase sales. What many are doing is realizing that their sites are perhaps not as good as they could be and are investing in updating or de-developing them. For others, the process of site maintenance is an ongoing activity with a dedicated team.

So with all of this development going on, it would be fair to assume that there must be a lot of testing as well? I would suggest that the answer is no. For some reason, when people are requesting development of web sites, it is believed that they are best placed to ensure that it is fit for purpose.

This is a bad assumption for several reasons:
1. Finding the time to thoroughly test the website is difficult to achieve. The people responsible for the new website are normally being asked to do this as one of many tasks.
2. The attention to detail required to check the entire site, is high and demands a certain type of individual. The tester has this mindset and is used to applying methodical and meticulous tests, whereas the business person may not be.
3. This may be the first time that the business person has been asked to test anything. Testers are used to testing websites, this is something that they do on a regular basis and this enables them to make better use of their time, homing in on problem areas.
4. Business resources normally check things positively, making sure that situations that are expected are dealt with. The tester introduces negative testing. Checking to understand that the site can handle abnormal activity, that which deviates from the norm.

So what is happening? Companies are employing development agencies and placing themselves in their care. They are relying on the development agency to ensure the quality of the site. The business are doing their best to make sure that the site is okay and are paying for development when they think it is finished.

In a normal IT project, the business relies on the testers to ensure that the application is to a certain standard before it comes anywhere near to them and this step is not being performed. Three problems result from a lack of in-built quality: The first is that the development agency is paid for the delivery, even though problems may not manifest until later. The second is that the development agency is held in low esteem once they have been found to have under-delivered. The third is that the organization requesting the website, find that they have a tool which does not meet their needs and therefore requires further investment to put the problems right.

The involvement of testing as part of the process ensures that the site is performing as was requested. It can be used as a means of safeguarding the payment, ensuring that the development agency have fully delivered prior to releasing funds. It also protects the development agency from losing their customer due to poor satisfaction.

The design is truly the prerogative of the business and only they can say if the look of the site is what they wanted to achieve. Other than this it should be placed in the hands of the experts. Remember that better quality, means customers are happier to use the site, more likely to find what you have to offer and therefore an increase in sales should result. I have said in many places that a poor web experience results in lost sales and that users are highly likely to leave your site for one offering a better experience or easier to find items.

Ranking Your Web Success

When talking to a wide group of web owners, it becomes apparent that whilst they are concerned about the quality of their website, they are equally interested in some of the other aspects of their web presence.

Different people have different ways of measuring a web sites success, but there are some which are universally accepted. I have yet to find a site that is not recognised by Google for instance. Google offer a means of looking at web sites in order to determine their importance to the internet community. It does this by a series of complex algorithms and the results are provided on a scale of 0 to 10, with 0 being the lowest and 10 being the highest. I would suggest that an average Google Page Rank is between 4 and 6. Alexa Ranking looks at the popularity of a website in comparison to all others. A score of under 100,000 is particularly good and one that is running above 10,000,000 bad.

A general rule of thumb is that as those sites with a high Google Page Rank will also have a high Alexa Rank, although at times this does not follow precisely. Once you have established your own rankings, you can then start looking at the competition. We all know who our closest competitors are, those that we would like to think we are better than and perhaps those that we would aspire to match. From a web perspective comparing each of the organisations Google and Alexa rankings against those of your own organisation’s, gives a good indication of your standing in the group.

Use of such competitor analysis helps organisations recognise that work is required, or that they are ahead of the game and need to maintain momentum. Whatever your line of business, the importance of being found quickly on the web is high. More traffic comes to your site and you can then reflect on what is happening to traffic when it arrives.

When checking a web site, make sure that you do not forget to consider the simpler aspects. Your rankings are an excellent gauge on how you are performing.

TCL India ensure that information of this nature is included as part of our Web Check results. These are quality indicators, although of a different type. Supply us with your list of competitors and we will also include the results for them in your report.

Software Testing Interview Questions

I am not suggesting that these are the only questions that would be asked, but are some samples for consideration. These are questions that I have used when interviewing others:

1. How many test scripts can you write in a day?
I am looking here for the ability to estimate. Whilst in reality the question is impractical, because it depends on so many different factors, I want to see the interviewee come up with some kind of answer. When estimating, there is a need to make assumptions and having a ball park figure enables someone to provide rough estimates more quickly.

2. How many test scripts can you execute in a day?
This is a repetition of the first question. I normally ask one directly after the other. The more junior resources tend to struggle with the first question and then the second also. The better resources learn from the experience of the first question and respond in a more positive manner to the second. Now I am not only looking at the ability to estimate, but also the ability to learn and an indication of the resources chances of seniority going forward.

3. Do you see testing as a service or a discipline?
Personally I am quite passionate about this one. I very much see testing as a discipline and a part of the software life-cycle that is as important as analysis, design and development. What I am trying to understand is the background that the individual someone has come from. Consultancy can demand one or other mindset and someone coming from either background can adapt, but I would suggest it is easier to revert from discipline to service than vice versa.

4. What is the most interesting defect you have found?
This is a passion question. I am looking to see if the individual can recount a particular incident and in what degree of detail. This begins to tell me whether they are a career tester or someone who is doing a job.

5. What are the key components of a test script?
Someone who is raising scripts every day should be able to define what information is required to be recorded. Different organisations have different standards, but there are key aspects to the answer. The requirement should be referenced or potentially included showing understanding that scripts should always be traceable to their point of origin. The script should have steps through, each of which defines the action to be taken, the data to be used and whether the step has passed or failed. This is the most basic of information and fundamental to all testers so an inability to provide this information is probably going to fail the candidate.

6. Can you define Black, White and Glass box testing?
This question is used to understand whether people have a basic understanding of testing jargon, as well as having attended the ISEB Foundation in Software Testing, where Black and White are covered. The inclusion of Glass box testing throws some people completely, others will think about it and try to define the answer; some will immediately define Glass box as being the same as White box.

Scenario Based Questions

You are the test manager on a project. The code is going to be delivered late and you now need to complete four weeks testing in two. How are you going to cope with this?

I am looking for managerial level thinking and understanding. There are numerous ways of dealing with this, more staff, overtime, weekend working, shift working, risk based testing, reduction in deliverables. The question is how many can they come up with and do they understand how to achieve them and the associated problems.

Risk Based Testing – How are they going to achieve this? Are the tests already grouped by risk? Are the requirements already grouped by risk or have the risk identified against each.

More test effort – Whilst increasing test effort is a simple means of achieving more in the same time frame, there needs to be a realisation that other departments will also need to apply similar Resourcing, to cover out of hours working, to handle increased volumes of defect fixing.

Cost Implication – Does the person discuss the increase in cost that can be associated with their suggested course of action? Are they thinking at project level or something lower?

Test Script Storage

Storage of anything should be in a manner whereby retrieval of the item is easy. If we look at a library, everyone is familiar with the process. Find the section dealing with your subject matter and then search for the book by author. It has a proven track record and we are all able to go into a library and find what we are looking for (as long as it is there).

Test scripts are no different. As testers we generate vast volumes of scripts and need to be able to re-use them. It strikes me as odd therefore, that scripts are so often stored against a project. There may be subdivisions within the project structure for each of the applications being tested, perhaps even a definition of which version the scripts have been written against. Indeed If the testers are a constant from one phase of the project to the next, the scripts are likely to be re-used.

But for each project there is normally a primary application that is being tested and the others are interfacing systems. The scripts that are generated dealing with the interface do not necessarily cover the entire application. The first problem occurs when someone outside of the project needs to test the primary application. If they are unaware of the project name, the chances are that they will produce their own set of scripts. We now have a duplication of effort occurring. Each new script that is written is potentially wasting effort.

The second problem is that the scripts that were written to test a change to an interfacing system, will not be stored in a way that is usable or locatable by those testing it as a primary application. This generates two problems, the first being that the test suite for the interfacing system is now out of date. The second problem is that the regression pack has not been updated to reflect the changes and therefore future changes may introduce problems into the new code which will not be identified. At best the difference in version may alert the testers to the fact that a change has occurred and time will be spent trying to re-locate the scripts, or more likely, they will find out what the change was and generate their own scripts.

I believe the way to avoid this is to ensure that scripts are not stored by project, but always by application. The application should then be divided into a series of suites, one for each version. Lastly a regression pack should exist as a permanent fixture. This allows the storage of scripts as deltas to the original suite and then as each new version of the application is developed, the scripts can be reviewed for those which would best augment the regression pack. It may be prudent to have each version contain all scripts, as some will have been modified or removed from the test suite.

This means that any one on any project, can look at the applications they are working with and know without doubt, where to look for the scripts. I can only see one argument against this, which is around end-to-end tests and UAT, where the focus on the application is not present and the focus is more around the business process. In such instances these scripts may need to be stored against the project.

To my way of thinking, this is symptomatic of testing being seen as a service to a project. Therefore the need is to satisfy the project’s needs by storing everything in a very project-centric manner. The reality is that this is costing organisations a vast amount of time and money as scripts are permanently re-generated in order to meet the current requirement.
The situation becomes worse where large volumes of external resources are involved. A lack of consistency and knowledge leaving the client organisation becomes an issue and the likelihood of finding a previously written set of scripts lowers dramatically.

Lastly, the regression pack is key to ensure the stability of the production platform and anything which compromises the regression pack undermines the efficiency of the business.

So – to summarise, ensure that test scripts are written and stored by application wherever possible. Breakdown the scripts under the application folder by version and have a generic regression pack sub-folder either at application level or version level.

W3C 29 out of 30 Sites Fail

W3C makes the following statement:

“W3C primarily pursues its mission through the creation of Web standards and guidelines. Since 1994, W3C has published more than 110 such standards, called W3C Recommendations. W3C also engages in education and outreach, develops software, and serves as an open forum for discussion about the Web. In order for the Web to reach its full potential, the most fundamental Web technologies must be compatible with one another and allow any hardware and software used to access the Web to work together. W3C refers to this goal as “Web interoperability.” By publishing open (non-proprietary) standards for Web languages and protocols, W3C seeks to avoid market fragmentation and thus Web fragmentation.”

http://www.w3.org/Consortium/

So how come so many sites are not following these standards. As a piece of work recently, we looked at 30 random web sites and checked them for W3C compliance. We found that only 1 of the 30 passed the check.

What was perhaps more surprising, was that around one third of the sites that were reviewed, had an error count of less than 30. Would it not be reasonable to suspect that an organisation which is so close to compliance to the foremost standards on the web, would go the extra mile and achieve compliance? I must therefore assume that these organisations are unaware of the fact that they are nearly W3C compliant and are developing their websites in ignorance, relying solely upon the development agencies to abide by their own standards, some of which may happen to coincide with W3C.

So what are the problems that organisations that are not W3C compliant going to face? Foremost, the transfer of the development from one agency to another is going to become more complex. Rather than being able to transfer code that is well written and understood by many, the site owner may become tied to a certain development agency, as they are the only ones that understand the code. The cost of changing a website completely is often far too expensive to consider, resulting in a reliance on one particular supplier.

Secondly, these standards once employed facilitate more effective and efficient crawling by web robots, who gather the information that search engines use. This means that poorly written code, translates to hard work for robots, poor understanding by search engines, lowering chances of the site being identified and reducing traffic to the site. Reduced traffic means reduced sales.

Thirdly, for those that are using the site from a disabled perspective, the tools that they use to try and surf the web, are hindered by poor code, making it more difficult to look at the site. We have already discussed elsewhere in the blog the importance of making web usage easy, yet here we find another example of poor user experience. Some organisations have even had lawsuits filed due to them failing to meet their obligations to disabled users.

Lastly, the site is less likely to transfer to other platforms used to surf the web, such as mobile phones and handheld devices. This is again restricting the use of the site, barring individuals that do not come on line using the mainly prescribed mechanisms.

My last point on W3C for the moment, so as not to be taken as a complete hypocrite, is that when I checked a couple of blogs, including my own, the volume of errors was in the hundreds. I am not yet certain whether this is something that can be remedied, but I can assure you that we will be looking at it and will let you know the results.

In summary, get your site W3C checked. Know that you are giving all users the best possible chance to use your site, have a good experience and possibly even generate a sale or two.

Middleware Regression Testing

I have been working for a client on an interesting problem recently, which I thought I would share on the blog. The problem has been around the regression testing of an item which is intrinsic by nature, to many other projects. The item is classed as middleware, which provides a translation between two different systems, but is being relied upon increasingly, with in excess of 50 systems now interfacing. Most projects now require the middleware to be updated and the volume of change is placing the regression testing under increasing pressure as more releases occur.

The problem is amplified as the projects are working individually and the middleware changes are independent as a result. A group of projects are collated to form a release and once normal testing is completed the middleware changes are converged to form a single drop of code, which is then passed into the regression testing phase.

The first mistake has been made. The projects are not taking accountability for testing the converged code and in some instances have even been disbanded prior to this testing occurring. At this point the release has lost the testers with the understanding of the end to end function and more pressure is applied to the regression test team, to trace the scripts from the project, interpret them and generate new scripts. It is important that The testing of the converged code is now taking up large amounts of the already pressured regression testing window. This needs to be pushed back to the projects and the convergence owned by them.

By removing the testing of the converged code from the regression process, more time is now allowed and full regression can be performed, but this does not provide a future proof solution. The same window is now being fully utilised and as the volume of utilisation increases over time, the regression pack will increase and the window will become too small again.

This is an ideal opportunity to make the most of an automated solution. If we look to industry standards, automation makes a test 7 times faster to execute but 7 times longer to write. This means that the testing duration, once automated can be reduced to 1.5 days. Now we have a regression test that is only performing regression; that can be executed swiftly and provides scope for future growth.

The testing environments were another level of complexity. Whilst the project completed it’s testing with all the required components/systems/applications, the regression team, within their two week window had to acquire their own equivalent. Again this was another drag on time, but more importantly the regression was sometimes being performed on an environment that was incomplete. The result has been to make the regression middleware environment available to the projects for the convergence testing, but to utilise the projects interfacing systems. In order to future proof, the regression testing will be moved onto a pre-production environment.

There were other aspects to this problem, but those listed were particularly pertinent. The following feedback was received from two sources as a result of the work and presentation of the findings:

“I was delighted that your slides prompted so much discussion, even if it made your presentation difficult, and that by all accounts we have got the main individuals on board.”

“I was very appreciative of the clear way you presented where we are and your proposals for going forward, together with a clear capture of issues that are spoken about in lots of quarters but never pulled together into a single picture. Great.”

In summary – regression testing should encompass the entire system.
Projects should retain responsibility past the point of release to production.
Automation is ideally suited to regression testing.
Testing should always occur on as complete an environment as possible.

Dealing with Delays Impacting Software Testing

We have all been there, on a project where the inevitable has happened. You guessed it, the code is late into testing, delivered at the 11th hour. Entry criteria has come under threat and possibly been ignored completely. Your test environment has been delivered but remains unproven as the code has not been available. The Project Manager has been harassed by the business stakeholders for the development delay and is not interested in your problems. IT Management are pressing to see the project delivered on time and to top it all, Marketing have arranged for a campaign launch to occur on the prescribed delivery date. As the Test Manager, you are now the primary obstacle to go live; the success of the project is resting on your shoulders. Oh yes, and your six week testing window has been reduced to four.

This is the stress (no pun intended), of the test execution phase. Not only are you now required to think out of the box in order to complete the testing, but you must also be thinking at a far wider level than just testing. There are actions that can be taken at a project level which can make a massive difference to the work of the testers.

Let’s start by looking at this from the wider perspective. I remember a situation on a very early project I was managing. Sat in a project progress meeting and being asked how we could test with an execution window reduced from four weeks to two. I dug my heels in and refused to budge, resulting in a separate meeting immediately afterwards with the PM, being instructed that this was not the right course of action and getting some early tuition in the art of testing.

Back to our problem, there are several actions that can be taken at project level to help with the situation and it may be possible to assist the PM by making some recommendations. (1) Does the application all need to be placed live? Are there any aspects of the application that could be put live as part of a second release, reducing the scope of the testing required? (2) Can the release occur to internal users only? This minimises the risk of damage in the event of production defects, meaning that exit criteria may be reviewed. (3) Can additional development effort be applied to recover whilst in development?

Regardless of the answers at a project level, the following can be applied by the Test Manager to handle the situation from within testing:

(1) Insert the testers into the development team and increase the unit and integration in the small testing. Improve the quality before it hits testing, reducing the volume of defects found and therefore the duration.
(2) Consider taking components of the application that are finished, prior to the arranged delivery date. It is likely that there are only certain parts of the application that are causing the delay and not the entirety of the application. Bringing in some parts early, increases the testing window and enables recovery of some of the lost time.
(3) Apply a risk based testing technique. Test the high risk elements of the application first and work through the testing in order of risk. When time runs out, this should mean that the only the items of lower risk have been omitted.
(4) Increase the hours being worked by the team. Look at options around overtime and weekend working. If using offshore capabilities then look at working two days in one. (A word of warning at this point, if the testers are working, they need to be supported by the environment support staff and the developers. Testing alone will only increase the flow of defects making it harder for development to keep up and an environment issue could stop all out of hours work.)
(5) Consider overlapping some of the testing phases. E.g. If UAT is being run as a separate phase, after functional testing, look at overlapping some of the UAT with the functional testing that is occurring.
(6) Ensure that there is a high focus on defect prioritisation by the business. Make sure the developers are fixing what needs to be fixed first. (Don’t ignore severity at this point.)
(7) Monitor defect turnaround. If the development has arrived late it is indicative of problems and a slow defect turnaround will cripple the project and whilst testing may complete, whilst the exit criteria will have been corrupted.
(8) Can more environments be made available? There is likely to already be a requirement for multiple environments, but if you begin overlapping test phases, functional with non-functional, with UAT, then the volume of environments needed may increase.
(9) Carry out a review of the exit criteria for the project. Bear in mind that this was set prior to the problems occurring and although they are the desired outcome, some compromise may need to be reached. Work out what is acceptable and don’t forget that if coverage is reduced, the numbers of defects are indicative of only a percentage of the testing. i.e. 80% coverage, means that you have possibly only have discovered 80% of the defects and it is good to assume that 20% remain unfound.

Actions (2), (3) and (5) above, all increase risk in some manner. Ensure that the project and stakeholders have agreed to these risks and that where possible mitigating actions are in place.

To summarise, there are many actions that can be taken within testing to deal with a slippage, whilst maintaining the original delivery date. Don't forget the project elements that can make a difference. I am sure there are more but do ensure that all aspects are being looked at and only then start to compromise on the testing.

Static Testing - Do you have a requirement?

Static Testing is the first form of testing that can be applied during the software development life-cycle. Once the requirement document has been produced it should be checked against the 8 points identified below. This will ensure the removal of defects prior to other departments or resources becoming involved and therefore minimise costs.

60% of all defects are attributable to the requirement specification.

The 8 point check applied in order to perform static testing against a requirement document is as follows:

-Singular: One requirement that does not refer to others or use words like "and".
-Unambiguous: Not open to more than one interpretation. Clear and easy to understand.
-Measurable: Avoids use of words like instant, approximately. Specifies given units, such as hours minutes and seconds.
-Complete: The requirement is not lacking information or supporting data.
-Developable: The developers will be able to implement the requirement.
-Testable: The testers will be able to test the requirement.
-Achievement Driven: A benefit is associated, which is tangible.
-Business Owned: A member of the business owns each requirement, providing a point of reference and approval.

Within the testing discipline, 60% of defects are believed to be attributable to the requirement specification. This means that one of the largest contributors to poor software quality can be remedied at project initiation. It is also widely accepted that each phase later in the software development life-cycle that a defect is found, it costs 10 times more to resolve. In the simplest form, this means that for every £1 it costs to resolve a defect at the point of the requirement being defined, if missed until the resulting application is live, will cost 10,000 time more. This is due to the effort that will have been spent designing, developing and testing something which was wrong, but the real cost comes when a problem in live causes an outage affecting large volumes of users and perhaps even stopping them working. Not only does this impact the individual, but dependent on what they are doing, could stop income to an organisation's other departments: Sales, Billing, Payroll. All of these become disasters and if in the public eye can lead to bad press for the organisation.

It is 10,000 times cheaper to fix a defect in the requirement compared to live!

We are trying to make static testing for organisations as easy and affordable to achieve as possible to achieve for organisations. Without complexities of billing and non-disclosure, the document can be e-mailed to either of the contacts detailed below. Our pricing is fixed so there are no surprises along the way.

Grant.Obermaier@TransitionConsulting.in or Mick.Morey@TransitionConsulting.in

Testing will be scheduled and will take between 5 and 30 working days. If a faster turn around is required please specify any deadlines and we will advise on achievability.

Early Defect Detection

They say that it is the early bird that catches the worm. The same is true in software testing.

The process starts with a requirement and from this point onwards, there is gradually more and more effort being applied to the creation or modification of an application. More people become involved and the cost of the solution increases. This is the natural course of events. It is therefore a natural conclusion, that if a mistake is made in the requirement, the cheapest time to fix the problem, is as the requirement is being defined, or immediately afterwards. This keeps the cost to a minimum. The same logic applies to design, development and test. Yes, even testing has the ability to insert defects, with badly written scenarios, scripts or reporting defects that are not real.

Defects are introduced throughout the software development lifecycle and the art of testing is to find as many of them as possible when they are inserted. It is widely recognised that there is a parabolic curve of defect insertion. The starting point is the requirement specification which begins by inserting 60% of the defects. The curve terminates with live or production, where the intended result is to find 0 defects. The tester should report on completion of the project, the defect detection efficiency. This looks at understanding for each defect, where it was inserted and where it was detected. A perfect test process would look to identify each defect as it is inserted. This is highly unlikely and the reality is that some defects will be found in later phases of testing, or indeed in live.

It is important to understand that projects testing budgets can double when critical steps such as static testing are not performed. This forces 60% of the defects into later phases, incurring as a minimum the cost of the design. Projects also often suffer from slippages and with test execution occurring at the end of the process, is often squeezed or compromised. The project must also realise that reducing the testing duration or coverage, is likely to increase the risk of a defect being found in live, where in comparison to it being found and removed in the requirements phase, will cost 10,000 times more to resolve, at best 10 times more. The reality here is that a defect undetected until live, can be business impacting, brand damaging and could cause a business to fail.

It is therefore important that testing is involved in the project from the outset, not as something that is included if there is the time, the budget and the inclination. Only by the application of systematic testing throughout the project is the quality level going to be understood and the opportunity to remedy problems in a timely manner presented.

Severity vs Priority

When raising a defect, there is a common confusion that occurs for testers and others alike. What is the difference between a defects severity and its priority?

So let’s start with looking at the dictionary definitions:
Severity - Causing very great pain, difficulty, anxiety, damage, etc.
Priority - Something that is very important and must be dealt with before other things.

We can see that the two words have completely different meanings. Why then is their confusion between them?

The severity is the domain of the tester and they should be capable of recording this. The severity to the testers is the impact of the defect on the application and their ability to continue testing. The priority is the domain of the business and should be entered by them against each defect raised to reflect the importance of the change to them.

For instance, a spelling mistake would be deemed as a low severity by the tester, but if this mistake occurs in the company name or address, this would be classed as high priority by the business. An inability to access a rarely used menu option may be of low priority to the business, but the severity is high as a series of tests cannot be executed, all dependent on access to the option.

The mistake that we have seen made many times is to assume that the tester is also capable of recording the priority. Whilst it may be possible for the tester to make an educated assessment, the priority is the business’ means of defining what must be repaired prior to release to production and the order in which effort should be applied to the fixing of defects. Testers who have been involved with a particular application for some period of time may be able to do this, but it is essential to have adequate business representation on the project and their involvement in the life-cycle of a defect and the defect management process.

When a project enters test execution, the focus will be on fixing defects of the highest priority. This means that the application will be released with the minimum amount of priority defects unresolved. Care should be taken by the Project Manager to ensure that whilst the priority is paramount, severity is not ignored. What is needed is a balanced approach, which favours the business priority. At the end of the project the volume of high severity and high priority defects should have at least been reduced, if not removed, in order to meet the exit criteria defined in the test strategy.

To summarize:
Priority = Business = Order of Fixing
Severity = Tester = Failure of Application

Components of a Bug

Legs, body, head, eyes, mouth; or are we talking about software bugs? The bug, error, defect; it little matters what it is called, but some are more touchy about the subject than others and like to use more politically correct statements, like issue or observation. The reality is that during the testing of an application or piece of software, something is identified as not performing in the expected manner, as driven by experience or indeed by the requirement specification, functional specification or design documents.

Call it what you will, we will use the term defect. Ensuring that you capture the right information at this point is critical. From a testing and development perspective, it is essential to be able to reproduce the defect at a later date, either as part of the remedial process or proving that a remedy has worked. TCL India would recommend that the following points are recorded for each defect:

· Unique defect identifier – a unique reference by which the defect can be identified.
· The application name or identifier and version – the software in which the defect was located.
· Environment specifics – the environment that was being used to test the application.
· Test script identifier (if applicable) – A unique reference from the test script, enabling each defect to be identified by script.
· Defect synopsis – A brief description of the defect that has been encountered.
· Detailed description of Defect – Full definition of the defect, enabling it to be reproduced by developers or those trying to resolve it.
· Test Steps Executed – from the test script, details of the steps executed and the detail of the step that has failed
· Expected Result for Test Step – The expected results for the step, e.g. the displayed logo should have been blue and white
· Actual Result for Test Step – The actual results encountered, e.g. the displayed logo was red and white
· Evidence of defect – Additional information showing the defect, such as screen shots
· Severity estimate – The impact of the defect on the ability of the tester to complete testing
· Priority – The importance of resolving the defect to the organisation
· Tester name – The name of the tester who has reported the defect
· Test date – The date that the tester located the defect
· Defect reporting date – The date that the defect is located. This should be immediate
· Defect assigned to – Who is being asked to resolve the defect

Each time responsibility for a defect is passed to another party, that fact should be recorded to form a history. The value becomes obvious when investigating why certain defects may not have been resolved in a timely manner. Investigation may be required as to why defects are bouncing backwards and forwards between different people, or those which are with an individual or team for a long period of time, impacting on the defect fix turn around time, potentially breaking SLAs.

In some more advanced tools used for defect management you will be able to prescribe the life-cycle of a defect. Such as:

-Raised by the tester

-Passed to the business analyst to prioritise

-Passed to the environment manager for specification of the environment and ensuring the defect was not caused by environmental issues

-Then onto development to fix code
-Back to the tester for re-testing

-Then onto the test manager or defect manager for closure.

The key then is to report on the defects on a daily basis during test execution, keeping the project informed of the number of defects raised and closed that day and the resulting cumulative totals. This enables the Project Manager to make decisions on the project's likelihood to go live on time, whether to apply more resources to certain aspects of the projects, whether overtime is required etc.

Care should be taken not to raise the same defect over and over again. When volumes of defects are high it is worth employing a defect manager to ensure that defects are not being repeatedly raised, wasting the time of resources. The defect manager should also ensure that all defects are being progressed. Sometimes defects that have been remedied are re-introduced erroneously: this is usually symptomatic of poor configuration management.

Offshore Communication

Communication is a critical point for a successful offshore arrangement. Our TCL COMS (Consultancy led Offshore Managed Service) is built around our understanding of this. But beware, communication can be difficult, even amongst people of our own nationality and culture. Speaking from personal experience, there is a huge between speaking to someone with a broad Scottish, Geordie or Wiltshire accent, when compared to someone speaking “The Queens English”. I mean no offence to any of the above, but accents are hard to deal with face to face and even harder when a telephone is used to handle the communication.

When dealing with other nationalities, where English is not the first language, the problems are amplified and whilst there are some foreigners who speak English perfectly the chances of finding such people becomes more difficult.

What we have found is that when people from other countries come and work for extended periods in the UK, their language skills naturally increase. As a result of this TCL India look to deploy resources locally, that have a long exposure to the UK, it’s people, their idiosyncrasies and the colloquialisms they use. These resources become highly useful, because not only can they communicate with us, they can also communicate with those offshore, utilising combinations of their own language and English. This provides an excellent conduit for communication, but it cannot be relied upon as the sole answer to the problem.

Written communication is far easier and less liable to be misunderstood than verbal, if interacting directly with an offshore capability. E-mail can be used and tools such as MSN Messenger if a more conversational approach is required.

Video Conferencing offers an ability to see as well as hear, but whilst it is great to see the faces of those you are talking to, there are inherent problems. Not only are you dealing with accents, but you have a time lapse and perhaps poor image quality, all of which is tending to distract you. There is also the potential lack of visual and audio synchronisation that your brain will be trying to cope with. It is suggested that if video conferencing is to be used, it does not occur at the beginning of the relationship and that time is taken to become accustomed to the experience.

In all offshore communication, look for feedback to demonstrate understanding. Query their understanding of the requests and make sure that theirs is the same as yours. Controversially perhaps, for the offshore element of a relationship, use the same techniques back at the client. Make sure that your grasp is right. Make sure that they have understood your communication. Whilst this is written very much from a UK perspective, the rules apply the world over.

Beware of getting back exactly what you have asked for and not what you wanted. Be careful in your communication. You could be taken literally.

Contractually Demanding Testing from Developers

It is an experience that testers all over the world will instantly understand. The contract with the development agency makes no demand for them to perform unit and integration (in the small) testing.

How does this manifest itself? Firstly, the Test Manager will specify within the Test Strategy or Test Plans a series of entry criteria, from development into testing. This should comprise test plans, a full suite of test scripts and a series of defects that have been raised. As the development nears completion, the Test Manager will begin to ask for the evidence of the testing and look to meet the entry criteria for Integration (in the large) or formal testing.

At this point the response can be that the unit and integration (in the small) testing has not been contractually specified. The supplier often takes one of two stances, stating that this cannot be achieved, or insisting that the testing can be performed, but at additional cost and increased timescale. This immediately places the Project Manager in an awkward position: they will be unable to agree to the additional time and will not want to incur the increased costs. The normal action at this point is for the Project Manager to accept that a mistake has been made, log a risk and accept that the code will be delivered into testing without development testing having occurred (or proven to have occurred).

The quality of code is now likely to be of a lower standard and the volume of defects expected in the formal testing higher. This may result in additional cycles of testing and increases in costs from this perspective (part of the risks that the Project Manager will have taken on board). As the volume of defects found during the black box testing grows, the demand on the development agency increases and care should be taken to ensure that defect fix times are monitored. These are likely to slow as the volume of defects increases and this should be reported to the Project Manager as soon as it is seen. A good contract will have service level agreements(SLAs) in place, to ensure that defect turn-around is at a specified rate depending on severity and priority.

If the development agency has failed to perform their own testing, the demand is placed on the testing team or test supplier and the estimates that they will be working to should be re-visited. If testing is working on a fixed price contract, failure to meet entry criteria should constitute a change request and the ability to re-price.

It may be possible to insert some of the test team with the developers, allowing them to test unit and integration (in the small) in parallel to the developers. Alternatively, testers with a technical bent can be asked to look at the code, perhaps performing dry runs. If the testing is being performed but the deliverables are not being made available, then it may be possible to witness some of the testing that is occurring. It must be said that these last two points are more geared to damage limitation, rather than ensuring that the right level of quality has been built into the product.

We need to avoid this in the first place, so ensure that your organisation's Supply Chain has a standard wording that is inserted into all contracts with development agencies protecting from such events. For smaller organisations that do not have a Supply Chain function, bring the Test Manager on board for up-front consultancy; they should look to ensure that this does not happen. A further measure is for the development agency to be asked to sign off the test strategy, which should detail the entry criteria into the formal (black box) testing phase, giving the buyer some level of recourse. Lastly, as a Test Manager, don't leave it until the end of development to find you have a problem - identify it as early as possible and get written agreement that the appropriate testing will take place and that the test assets produced will form part of a deliverable.