SOFTWARE TESTING GLOSSARY (Update)

This glossary is a living post – so will be edited as we come across terminology that is not included. If you have any suggestions or disagree with an explanation, drop us an e-mail and let us know. TCL India offer this as a means of establishing glossaries of your own or as a point of reference. This update includes 10 new definitions.

A…………………………………………………………………………………………………

Accessibility: (Testing)Looks at an application to ensure that those with disabilities will be able to use the application as an able individual.

Agile: A development method, involving the creation of software, with the contributing parties, including testing, all working on the same item at the same time.

Alpha: (Testing)Testing of an application which is in production, but not available for general use. This is normally performed by users internal to the business.

Analyst: Person who interacts with the business in order to understand and document their requirements for an application or change to an existing one.

Analyst (Test): Person responsible for the preparation and execution of test scripts, recording and progressing of defects, reporting into the Test Team Leader or Test Manager.

Automation: The process of developing a script or program which can be run by software rather than a Test Engineer, to test an aspect of an application. Automation is used to increase the speed of test execution.

Automation Centre: A tool used in the automation of software testing. Seehttps://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-127-24%5E1074_4000_100__&jumpid=reg_R1002_USEN

B……………………………………………………………………………………………………

Beta: (Testing)Testing of an application which is in production, but not available for general use. This is normally performed by a select group of friendly external testers.

Black Box: The process of testing without understanding the internal workings of an application, relying on inputs and outputs only.

Bug: See DefectBusiness Requirement Specification: See Requirement Specification

C……………………………………………………………………………………………………

Case: See ScenarioCode: The software that has been produced by development and is being subjected to testing.

Completion Report: A document produced during the testing and presented to the project manager on completion of testing activity. The document should detail the tests that have been performed along with the associated results and defects. Some reports may include a recommendation for the applications suitability for release to production.

Component: The smallest item that is testable or producible. This often refers to a single file of code.

Configuration Management: The means of managing the components required to make a larger item. Applicable to files, documents, data, hardware, software, tools. Understanding of the versions of each component required in order to be able to re-build the larger item.

Criteria: See Entry Criteria and Exit Criteria

D……………………………………………………………………………………………………

Data: Information pertaining to the use of a system or recorded through the use of a system. Data is used in order to test a system, potentially after data cleansing if personal information is involved.

Data Generator: A tool used to generate high volumes of data in order to be able to test many permutations, or to load test an item.

Data Protection Act (DPA): The act that determines how personal data should be used and protected from abuse.

Data Scrambling: The process of altering personal information from data to be used for testing purposes.

Defect: Where the item under test has been found to inaccurate, resulting from testing. Primarily used in associated with software, but equally valid for static testing of documentation.

Defect Detection: The means of identifying a defect. Can be a metric used to predict the volume of defects expected during the course of a project, or as a means of looking back at a project to understand where testing needs to be concentrated in future projects of a similar nature.

Defect Removal Efficiency: A metric used to assess the ability of testing to remove defects as they are introduced, during the software development life-cycle, keeping the cost of testing later phases to a minimum.

Defect Turnaround: The time taken from the identification of a defect, through to the point of resolution. Different levels of granularity may be used. e.g. A focus on the time taken by development.

Developer: A person responsible for the development of an application.

Development: A process of producing an application by production of low level design, code, unit testing and integration in the small testing.

Dynamic: Testing which occurs on the right hand side of the V-model, with the application present.

E……………………………………………………………………………………………………

Environment: The combination of hardware, software and data as used in development, testing and production. The platform/s upon which the testing occurs.

Entry Criteria: The criteria that must be met prior to a deliverable entering the next phase of testing to another. This is normally associated with documented test assets and pre-agreed volumes of defects.

Error: See Defect

Exit Criteria: The criteria that must be met prior to a deliverable leaving the current phase of testing. This is normally associated with documented test assets and pre-agreed volumes of defects.

Execution: The process of working through a script on the application under test, in the testing environment.

F……………………………………………………………………………………………………

Functional: (Testing) The testing of a products function, against requirements and design.

Functional Specification: A document which extracts all of the functional requirements from the requirement specification.

G……………………………………………………………………………………………………

Glass Box: See White Box

Grey Box: Testing performed by testers with some knowledge of the internal workings of an application. See also Black Box and White Box testing.

H……………………………………………………………………………………………………

High Level Design: A design showing the major components and how these will interface to each other, defining the hardware to be used and the software that will be impacted.

I…………………………………………………………………………………………………… 


Integration in the Large: Where the application or applications that have been developed are brought together along with those which have remained unchanged, building a production like system around the application/s. Testing is then applied looking at the communication between the different applications.

Integration in the Small: Where the components of the application that have been developed are brought together along with those which have remained unchanged, building the application or major component of a single application. Testing is then applied looking at the communication between the different components.Integration: The act of bringing many parts together to form a whole.

ISEB: Information Systems Examination Board. This was historically the board that was used to certify test professionals at either Foundation (Entry Level) or Practitioner Level (Advanced). See:http://www.bcs.org/server.php?show=nav.5732

ISTQB: International Software Testing Qualifications Board. See:http://www.istqb.org/

J…………………………………………………………………………………………………….
K……………………………………………………………………………………………………

Key Performance Indicator: A mechanism built on one or metrics, which determines a band of acceptable performance, which over time is often targeted towards improvement.

L……………………………………………………………………………………………………

Live: See Production

Load: One of the types of performance testing, this looks at testing for the maximum load on the system.

Load Runner: Tool used to performance test one or many applications, to understand how it handles increases in load. See:https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-15-17%5e8_4000_100__

Low Level Design: Definition of exactly how the application/s will be modified or produced in order to meet the requirements and the high level design. This can extend in some examples to elements of pseudo code being defined.

M……………………………………………………………………………………………………

Metric: A measure of an attribute or occurrence in connection with an organisation, department or functions performance.

N……………………………………………………………………………………………………..

Non-functional: How an application performs, rather than how it does it.

Non-functional Specification: A document which details the non-functional requirements such as performance, security, operational elements.

O……………………………………………………………………………………………………
P……………………………………………………………………………………………………

Performance Centre: A tool used for measuring the performance of an application or series of applications. See:https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-126-17_4000_100__

Performance: Used to describe many types of testing relating to the speed of an application. See Volume, Load and Stress.

Plan: A document produced for a type of testing defining how the testing will be performed, when the testing will be performed and who will be performing it. The test plan lists the test scenarios that will be scripted.

Preparation: The process of generating the test scripts.

Prince2: A project management methodology that is used amongst a lot of blue chip organizations to bring discipline to the software development life cycle.

Priority: The importance of fixing a defect from a business perspective. Defined by business representatives.

Production: The area or a computer network that contains applications which are in use by real users and contains real data. The area that applications are released to on completion of a project.

Q……………………………………………………………………………………………………

Quality: The suitability of an item for its intended purpose and how well it achieves this.

Quality Centre: Tool used to assist with the management of testing, recording and tracking scripts, logging and tracking defects and more. See:https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-127-24_4000_100__

R……………………………………………………………………………………………………

Regression: Retesting of a previously tested program following modification to ensure that faults have not been introduced or uncovered as a result of the changes made.

Requirement Specification: A document normally produced by a business analyst, capturing the needs of the individual in a manner which means that they can be translated into a software solution.

Re-Test: Taking a defect which has failed and executing the associated test script again.

S……………………………………………………………………………………………………

Safety Critical: Used to identify something for which the use has an impact on personal safety, such as medicinal applications or those used by rescue services.

Scenario: A high level description of how a requirement is going to be tested.

Script: Referable back to the scenario, the script defines each step that must be passed through in order to perform a test, along with the expected results. As the script is executed, results are recorded and if they match the expected result are marked as passed, otherwise as failed. A script containing a failure should have a resultant defect raised.

Schedule: A document, similar to a project plan, but detailing the activities associated with testing, when they are due to occur and who will be performing them.

Severity: The importance of a defect to testing and the application. Defined by Testers

Special Interest Group In Software Testing (SIGIST): A group that has been set up by the British Computer Society, made up of people interested in the subject of software testing.

Smoke: A process of proving an application or environment is ready to commence full testing, by running a sample set of scripts testing the primary functionality and/or connectivity.

Software Development Life Cycle (SDLC): The process of taking a requirement and translating it into a fully working application.

Static: The process of testing without the presence of software. Normally refers to the testing of documentation.

Strategy: A document produced at project or programme level, defining what testing is to be performed.

Stress: A form of performance testing, whereby the volume of testing is increased to a point at which the application is deemed to be failing to perform, either due to failure to meet performance criteria or system breakdown.

T……………………………………………………………………………………………………

Technical Architecture Document: Definition of how the business requirements should be translated into a technical solution at the highest level.

Test Bed: A cumulative term used for all of the test environments used by the test department.

Test Director: A tool used for test management, capture of requirements, scripts and defects, with the ability to manage defects through to resolution. Now replaced by Quality Centre.

Test Maturity Model (TMM): A means of measuring a testing department’s level of maturity in a similar manner to that of the capability maturity model (CMM).

Testing: The process of reviewing an item for accuracy and its ability to complete the desired objective.

U……………………………………………………………………………………………………

Unit: The testing of the application by the developers at a component or modular level, looking at whether the code is performing as expected.

User Acceptance: The means of testing applied, normally by members of the business or recipients of the system, whilst still in the test environment. This looks to ensure that business requirements have been understood by all involved in the application production as well as providing an opportunity to check the business processes that the application will be used in conjunction with.

V……………………………………………………………………………………………………

V-Model: A testing focussed software development life-cycle where the process of creativity occurs down the left side of the V and the process of testing up the right side of the V, where each step down the V has a corresponding point of testing up the right of the V. Begins with a requirement and terminates with User Acceptance Testing.

Version: An alpha numeric means of identifying a particular build of software, or a component thereof. The version changes incrementally to reflect each modification of the software, component, document etc.

Volume: (Testing) A type of performance testing which increases the volume of users on a system, in each cycle, taking the volume up to a prescribed limit, normally exceeding optimum load.

W……………………………………………………………………………………………………

W3C: World Wide Web Consortium – body creating web standards and guidelines. http://www.w3.org/

WAI: Web Accessibility Initiative. See: http://www.w3.org/WAI/

Waterfall: Project Management Method whereby each element of the software development life-cycle occurs sequentially.

Win Runner: A tool that was historically used for test automation. See Automation Centre.

White Box: Testing normally performed by developers, looking at the code and ensuring that each model of code is performing as expected. See Unit

X……………………………………………………………………………………………………

Y…………………………………………………………………………………………………….
Z…………………………………………………………………………………………………….

Why is Testing Sexy

I suppose the answer depends on the person and for some not at all. But I will endeavour to give you 12 reasons why I think Testing is “sexy”.

1. The world of testing is governed by process and is incredibly black and white. A test passes or it fails; there is no shade of gray.
2. It is like being part of a big family, where the same individuals are involved and you keep bumping into old friends and faces.
3. You have to keep your wits about you and be prepared to change tack quickly, effectively and efficiently.
4. Testing is the underdog of IT. It is trying to bring to people’s attention to the value that it brings to the software development life-cycle.
5. You have to be able to think around the developer, understand what they are trying to achieve and then look at how you can think around it.
6. You need to be able to look at a project as a project manager, analyst, designer, architect, developer and tester, giving you a well rounded grasp of IT.
7. There are opportunities to specialise in testing, by technology, by testing type or by process, giving endless opportunities to grow and learn.
8. The work is endless, because finding perfect code from perfect projects is nigh on impossible, keeping us all in employment.
9. We create tangible assets and benefits for those employing our services, leaving them in a stronger position.
10. Whilst sometime invisible to the end users, we make a massive contribution to their use of IT, without which life would be harder, more frustrating and the helpdesk would be inundated.
11. We are growing as a discipline become more recognised and with more accreditation available in universities across the world.
12. We get to break stuff on a very regular basis.

I hate the word “Sexy” being used to describe something like a subject, but it seems to be used regularly these days. Do I dream of curvy defects at night? ….No! But I am passionate about Testing. I do believe in the value that it brings and the value I can add as part of a project. More and more people are becoming involved and the testing community is growing. Less people are falling into Testing and are making it a career choice.

Dealing with Delays Impacting Software Testing

We have all been there, on a project where the inevitable has happened. You guessed it, the code is late into testing, delivered at the 11th hour. Entry criteria has come under threat and possibly been ignored completely. Your test environment has been delivered but remains unproven as the code has not been available. The Project Manager has been harassed by the business stakeholders for the development delay and is not interested in your problems. IT Management are pressing to see the project delivered on time and to top it all, Marketing have arranged for a campaign launch to occur on the prescribed delivery date. As the Test Manager, you are now the primary obstacle to go live; the success of the project is resting on your shoulders. Oh yes, and your six week testing window has been reduced to four.

This is the stress (no pun intended), of the test execution phase. Not only are you now required to think out of the box in order to complete the testing, but you must also be thinking at a far wider level than just testing. There are actions that can be taken at a project level which can make a massive difference to the work of the testers.

Let’s start by looking at this from the wider perspective. I remember a situation on a very early project I was managing. Sat in a project progress meeting and being asked how we could test with an execution window reduced from four weeks to two. I dug my heels in and refused to budge, resulting in a separate meeting immediately afterwards with the PM, being instructed that this was not the right course of action and getting some early tuition in the art of testing.

Back to our problem, there are several actions that can be taken at project level to help with the situation and it may be possible to assist the PM by making some recommendations. (1) Does the application all need to be placed live? Are there any aspects of the application that could be put live as part of a second release, reducing the scope of the testing required? (2) Can the release occur to internal users only? This minimises the risk of damage in the event of production defects, meaning that exit criteria may be reviewed. (3) Can additional development effort be applied to recover whilst in development?

Regardless of the answers at a project level, the following can be applied by the Test Manager to handle the situation from within testing:

(1) Insert the testers into the development team and increase the unit and integration in the small testing. Improve the quality before it hits testing, reducing the volume of defects found and therefore the duration.
(2) Consider taking components of the application that are finished, prior to the arranged delivery date. It is likely that there are only certain parts of the application that are causing the delay and not the entirety of the application. Bringing in some parts early, increases the testing window and enables recovery of some of the lost time.
(3) Apply a risk based testing technique. Test the high risk elements of the application first and work through the testing in order of risk. When time runs out, this should mean that the only the items of lower risk have been omitted.
(4) Increase the hours being worked by the team. Look at options around overtime and weekend working. If using offshore capabilities then look at working two days in one. (A word of warning at this point, if the testers are working, they need to be supported by the environment support staff and the developers. Testing alone will only increase the flow of defects making it harder for development to keep up and an environment issue could stop all out of hours work.)
(5) Consider overlapping some of the testing phases. E.g. If UAT is being run as a separate phase, after functional testing, look at overlapping some of the UAT with the functional testing that is occurring.
(6) Ensure that there is a high focus on defect prioritisation by the business. Make sure the developers are fixing what needs to be fixed first. (Don’t ignore severity at this point.)
(7) Monitor defect turnaround. If the development has arrived late it is indicative of problems and a slow defect turnaround will cripple the project and whilst testing may complete, whilst the exit criteria will have been corrupted.
(8) Can more environments be made available? There is likely to already be a requirement for multiple environments, but if you begin overlapping test phases, functional with non-functional, with UAT, then the volume of environments needed may increase.
(9) Carry out a review of the exit criteria for the project. Bear in mind that this was set prior to the problems occurring and although they are the desired outcome, some compromise may need to be reached. Work out what is acceptable and don’t forget that if coverage is reduced, the numbers of defects are indicative of only a percentage of the testing. i.e. 80% coverage, means that you have possibly only have discovered 80% of the defects and it is good to assume that 20% remain unfound.

Actions (2), (3) and (5) above, all increase risk in some manner. Ensure that the project and stakeholders have agreed to these risks and that where possible mitigating actions are in place.

To summarise, there are many actions that can be taken within testing to deal with a slippage, whilst maintaining the original delivery date. Don't forget the project elements that can make a difference. I am sure there are more but do ensure that all aspects are being looked at and only then start to compromise on the testing.

Static Testing - Do you have a requirement?

Static Testing is the first form of testing that can be applied during the software development life-cycle. Once the requirement document has been produced it should be checked against the 8 points identified below. This will ensure the removal of defects prior to other departments or resources becoming involved and therefore minimise costs.

60% of all defects are attributable to the requirement specification.

The 8 point check applied in order to perform static testing against a requirement document is as follows:

-Singular: One requirement that does not refer to others or use words like "and".
-Unambiguous: Not open to more than one interpretation. Clear and easy to understand.
-Measurable: Avoids use of words like instant, approximately. Specifies given units, such as hours minutes and seconds.
-Complete: The requirement is not lacking information or supporting data.
-Developable: The developers will be able to implement the requirement.
-Testable: The testers will be able to test the requirement.
-Achievement Driven: A benefit is associated, which is tangible.
-Business Owned: A member of the business owns each requirement, providing a point of reference and approval.

Within the testing discipline, 60% of defects are believed to be attributable to the requirement specification. This means that one of the largest contributors to poor software quality can be remedied at project initiation. It is also widely accepted that each phase later in the software development life-cycle that a defect is found, it costs 10 times more to resolve. In the simplest form, this means that for every £1 it costs to resolve a defect at the point of the requirement being defined, if missed until the resulting application is live, will cost 10,000 time more. This is due to the effort that will have been spent designing, developing and testing something which was wrong, but the real cost comes when a problem in live causes an outage affecting large volumes of users and perhaps even stopping them working. Not only does this impact the individual, but dependent on what they are doing, could stop income to an organisation's other departments: Sales, Billing, Payroll. All of these become disasters and if in the public eye can lead to bad press for the organisation.

It is 10,000 times cheaper to fix a defect in the requirement compared to live!

We are trying to make static testing for organisations as easy and affordable to achieve as possible to achieve for organisations. Without complexities of billing and non-disclosure, the document can be e-mailed to either of the contacts detailed below. Our pricing is fixed so there are no surprises along the way.

Grant.Obermaier@TransitionConsulting.in or Mick.Morey@TransitionConsulting.in

Testing will be scheduled and will take between 5 and 30 working days. If a faster turn around is required please specify any deadlines and we will advise on achievability.

Early Defect Detection

They say that it is the early bird that catches the worm. The same is true in software testing.

The process starts with a requirement and from this point onwards, there is gradually more and more effort being applied to the creation or modification of an application. More people become involved and the cost of the solution increases. This is the natural course of events. It is therefore a natural conclusion, that if a mistake is made in the requirement, the cheapest time to fix the problem, is as the requirement is being defined, or immediately afterwards. This keeps the cost to a minimum. The same logic applies to design, development and test. Yes, even testing has the ability to insert defects, with badly written scenarios, scripts or reporting defects that are not real.

Defects are introduced throughout the software development lifecycle and the art of testing is to find as many of them as possible when they are inserted. It is widely recognised that there is a parabolic curve of defect insertion. The starting point is the requirement specification which begins by inserting 60% of the defects. The curve terminates with live or production, where the intended result is to find 0 defects. The tester should report on completion of the project, the defect detection efficiency. This looks at understanding for each defect, where it was inserted and where it was detected. A perfect test process would look to identify each defect as it is inserted. This is highly unlikely and the reality is that some defects will be found in later phases of testing, or indeed in live.

It is important to understand that projects testing budgets can double when critical steps such as static testing are not performed. This forces 60% of the defects into later phases, incurring as a minimum the cost of the design. Projects also often suffer from slippages and with test execution occurring at the end of the process, is often squeezed or compromised. The project must also realise that reducing the testing duration or coverage, is likely to increase the risk of a defect being found in live, where in comparison to it being found and removed in the requirements phase, will cost 10,000 times more to resolve, at best 10 times more. The reality here is that a defect undetected until live, can be business impacting, brand damaging and could cause a business to fail.

It is therefore important that testing is involved in the project from the outset, not as something that is included if there is the time, the budget and the inclination. Only by the application of systematic testing throughout the project is the quality level going to be understood and the opportunity to remedy problems in a timely manner presented.

Components of a Bug

Legs, body, head, eyes, mouth; or are we talking about software bugs? The bug, error, defect; it little matters what it is called, but some are more touchy about the subject than others and like to use more politically correct statements, like issue or observation. The reality is that during the testing of an application or piece of software, something is identified as not performing in the expected manner, as driven by experience or indeed by the requirement specification, functional specification or design documents.

Call it what you will, we will use the term defect. Ensuring that you capture the right information at this point is critical. From a testing and development perspective, it is essential to be able to reproduce the defect at a later date, either as part of the remedial process or proving that a remedy has worked. TCL India would recommend that the following points are recorded for each defect:

· Unique defect identifier – a unique reference by which the defect can be identified.
· The application name or identifier and version – the software in which the defect was located.
· Environment specifics – the environment that was being used to test the application.
· Test script identifier (if applicable) – A unique reference from the test script, enabling each defect to be identified by script.
· Defect synopsis – A brief description of the defect that has been encountered.
· Detailed description of Defect – Full definition of the defect, enabling it to be reproduced by developers or those trying to resolve it.
· Test Steps Executed – from the test script, details of the steps executed and the detail of the step that has failed
· Expected Result for Test Step – The expected results for the step, e.g. the displayed logo should have been blue and white
· Actual Result for Test Step – The actual results encountered, e.g. the displayed logo was red and white
· Evidence of defect – Additional information showing the defect, such as screen shots
· Severity estimate – The impact of the defect on the ability of the tester to complete testing
· Priority – The importance of resolving the defect to the organisation
· Tester name – The name of the tester who has reported the defect
· Test date – The date that the tester located the defect
· Defect reporting date – The date that the defect is located. This should be immediate
· Defect assigned to – Who is being asked to resolve the defect

Each time responsibility for a defect is passed to another party, that fact should be recorded to form a history. The value becomes obvious when investigating why certain defects may not have been resolved in a timely manner. Investigation may be required as to why defects are bouncing backwards and forwards between different people, or those which are with an individual or team for a long period of time, impacting on the defect fix turn around time, potentially breaking SLAs.

In some more advanced tools used for defect management you will be able to prescribe the life-cycle of a defect. Such as:

-Raised by the tester

-Passed to the business analyst to prioritise

-Passed to the environment manager for specification of the environment and ensuring the defect was not caused by environmental issues

-Then onto development to fix code
-Back to the tester for re-testing

-Then onto the test manager or defect manager for closure.

The key then is to report on the defects on a daily basis during test execution, keeping the project informed of the number of defects raised and closed that day and the resulting cumulative totals. This enables the Project Manager to make decisions on the project's likelihood to go live on time, whether to apply more resources to certain aspects of the projects, whether overtime is required etc.

Care should be taken not to raise the same defect over and over again. When volumes of defects are high it is worth employing a defect manager to ensure that defects are not being repeatedly raised, wasting the time of resources. The defect manager should also ensure that all defects are being progressed. Sometimes defects that have been remedied are re-introduced erroneously: this is usually symptomatic of poor configuration management.