SOFTWARE TESTING GLOSSARY (Update)

This glossary is a living post – so will be edited as we come across terminology that is not included. If you have any suggestions or disagree with an explanation, drop us an e-mail and let us know. TCL India offer this as a means of establishing glossaries of your own or as a point of reference. This update includes 10 new definitions.

A…………………………………………………………………………………………………

Accessibility: (Testing)Looks at an application to ensure that those with disabilities will be able to use the application as an able individual.

Agile: A development method, involving the creation of software, with the contributing parties, including testing, all working on the same item at the same time.

Alpha: (Testing)Testing of an application which is in production, but not available for general use. This is normally performed by users internal to the business.

Analyst: Person who interacts with the business in order to understand and document their requirements for an application or change to an existing one.

Analyst (Test): Person responsible for the preparation and execution of test scripts, recording and progressing of defects, reporting into the Test Team Leader or Test Manager.

Automation: The process of developing a script or program which can be run by software rather than a Test Engineer, to test an aspect of an application. Automation is used to increase the speed of test execution.

Automation Centre: A tool used in the automation of software testing. Seehttps://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-127-24%5E1074_4000_100__&jumpid=reg_R1002_USEN

B……………………………………………………………………………………………………

Beta: (Testing)Testing of an application which is in production, but not available for general use. This is normally performed by a select group of friendly external testers.

Black Box: The process of testing without understanding the internal workings of an application, relying on inputs and outputs only.

Bug: See DefectBusiness Requirement Specification: See Requirement Specification

C……………………………………………………………………………………………………

Case: See ScenarioCode: The software that has been produced by development and is being subjected to testing.

Completion Report: A document produced during the testing and presented to the project manager on completion of testing activity. The document should detail the tests that have been performed along with the associated results and defects. Some reports may include a recommendation for the applications suitability for release to production.

Component: The smallest item that is testable or producible. This often refers to a single file of code.

Configuration Management: The means of managing the components required to make a larger item. Applicable to files, documents, data, hardware, software, tools. Understanding of the versions of each component required in order to be able to re-build the larger item.

Criteria: See Entry Criteria and Exit Criteria

D……………………………………………………………………………………………………

Data: Information pertaining to the use of a system or recorded through the use of a system. Data is used in order to test a system, potentially after data cleansing if personal information is involved.

Data Generator: A tool used to generate high volumes of data in order to be able to test many permutations, or to load test an item.

Data Protection Act (DPA): The act that determines how personal data should be used and protected from abuse.

Data Scrambling: The process of altering personal information from data to be used for testing purposes.

Defect: Where the item under test has been found to inaccurate, resulting from testing. Primarily used in associated with software, but equally valid for static testing of documentation.

Defect Detection: The means of identifying a defect. Can be a metric used to predict the volume of defects expected during the course of a project, or as a means of looking back at a project to understand where testing needs to be concentrated in future projects of a similar nature.

Defect Removal Efficiency: A metric used to assess the ability of testing to remove defects as they are introduced, during the software development life-cycle, keeping the cost of testing later phases to a minimum.

Defect Turnaround: The time taken from the identification of a defect, through to the point of resolution. Different levels of granularity may be used. e.g. A focus on the time taken by development.

Developer: A person responsible for the development of an application.

Development: A process of producing an application by production of low level design, code, unit testing and integration in the small testing.

Dynamic: Testing which occurs on the right hand side of the V-model, with the application present.

E……………………………………………………………………………………………………

Environment: The combination of hardware, software and data as used in development, testing and production. The platform/s upon which the testing occurs.

Entry Criteria: The criteria that must be met prior to a deliverable entering the next phase of testing to another. This is normally associated with documented test assets and pre-agreed volumes of defects.

Error: See Defect

Exit Criteria: The criteria that must be met prior to a deliverable leaving the current phase of testing. This is normally associated with documented test assets and pre-agreed volumes of defects.

Execution: The process of working through a script on the application under test, in the testing environment.

F……………………………………………………………………………………………………

Functional: (Testing) The testing of a products function, against requirements and design.

Functional Specification: A document which extracts all of the functional requirements from the requirement specification.

G……………………………………………………………………………………………………

Glass Box: See White Box

Grey Box: Testing performed by testers with some knowledge of the internal workings of an application. See also Black Box and White Box testing.

H……………………………………………………………………………………………………

High Level Design: A design showing the major components and how these will interface to each other, defining the hardware to be used and the software that will be impacted.

I…………………………………………………………………………………………………… 


Integration in the Large: Where the application or applications that have been developed are brought together along with those which have remained unchanged, building a production like system around the application/s. Testing is then applied looking at the communication between the different applications.

Integration in the Small: Where the components of the application that have been developed are brought together along with those which have remained unchanged, building the application or major component of a single application. Testing is then applied looking at the communication between the different components.Integration: The act of bringing many parts together to form a whole.

ISEB: Information Systems Examination Board. This was historically the board that was used to certify test professionals at either Foundation (Entry Level) or Practitioner Level (Advanced). See:http://www.bcs.org/server.php?show=nav.5732

ISTQB: International Software Testing Qualifications Board. See:http://www.istqb.org/

J…………………………………………………………………………………………………….
K……………………………………………………………………………………………………

Key Performance Indicator: A mechanism built on one or metrics, which determines a band of acceptable performance, which over time is often targeted towards improvement.

L……………………………………………………………………………………………………

Live: See Production

Load: One of the types of performance testing, this looks at testing for the maximum load on the system.

Load Runner: Tool used to performance test one or many applications, to understand how it handles increases in load. See:https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-15-17%5e8_4000_100__

Low Level Design: Definition of exactly how the application/s will be modified or produced in order to meet the requirements and the high level design. This can extend in some examples to elements of pseudo code being defined.

M……………………………………………………………………………………………………

Metric: A measure of an attribute or occurrence in connection with an organisation, department or functions performance.

N……………………………………………………………………………………………………..

Non-functional: How an application performs, rather than how it does it.

Non-functional Specification: A document which details the non-functional requirements such as performance, security, operational elements.

O……………………………………………………………………………………………………
P……………………………………………………………………………………………………

Performance Centre: A tool used for measuring the performance of an application or series of applications. See:https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-126-17_4000_100__

Performance: Used to describe many types of testing relating to the speed of an application. See Volume, Load and Stress.

Plan: A document produced for a type of testing defining how the testing will be performed, when the testing will be performed and who will be performing it. The test plan lists the test scenarios that will be scripted.

Preparation: The process of generating the test scripts.

Prince2: A project management methodology that is used amongst a lot of blue chip organizations to bring discipline to the software development life cycle.

Priority: The importance of fixing a defect from a business perspective. Defined by business representatives.

Production: The area or a computer network that contains applications which are in use by real users and contains real data. The area that applications are released to on completion of a project.

Q……………………………………………………………………………………………………

Quality: The suitability of an item for its intended purpose and how well it achieves this.

Quality Centre: Tool used to assist with the management of testing, recording and tracking scripts, logging and tracking defects and more. See:https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-127-24_4000_100__

R……………………………………………………………………………………………………

Regression: Retesting of a previously tested program following modification to ensure that faults have not been introduced or uncovered as a result of the changes made.

Requirement Specification: A document normally produced by a business analyst, capturing the needs of the individual in a manner which means that they can be translated into a software solution.

Re-Test: Taking a defect which has failed and executing the associated test script again.

S……………………………………………………………………………………………………

Safety Critical: Used to identify something for which the use has an impact on personal safety, such as medicinal applications or those used by rescue services.

Scenario: A high level description of how a requirement is going to be tested.

Script: Referable back to the scenario, the script defines each step that must be passed through in order to perform a test, along with the expected results. As the script is executed, results are recorded and if they match the expected result are marked as passed, otherwise as failed. A script containing a failure should have a resultant defect raised.

Schedule: A document, similar to a project plan, but detailing the activities associated with testing, when they are due to occur and who will be performing them.

Severity: The importance of a defect to testing and the application. Defined by Testers

Special Interest Group In Software Testing (SIGIST): A group that has been set up by the British Computer Society, made up of people interested in the subject of software testing.

Smoke: A process of proving an application or environment is ready to commence full testing, by running a sample set of scripts testing the primary functionality and/or connectivity.

Software Development Life Cycle (SDLC): The process of taking a requirement and translating it into a fully working application.

Static: The process of testing without the presence of software. Normally refers to the testing of documentation.

Strategy: A document produced at project or programme level, defining what testing is to be performed.

Stress: A form of performance testing, whereby the volume of testing is increased to a point at which the application is deemed to be failing to perform, either due to failure to meet performance criteria or system breakdown.

T……………………………………………………………………………………………………

Technical Architecture Document: Definition of how the business requirements should be translated into a technical solution at the highest level.

Test Bed: A cumulative term used for all of the test environments used by the test department.

Test Director: A tool used for test management, capture of requirements, scripts and defects, with the ability to manage defects through to resolution. Now replaced by Quality Centre.

Test Maturity Model (TMM): A means of measuring a testing department’s level of maturity in a similar manner to that of the capability maturity model (CMM).

Testing: The process of reviewing an item for accuracy and its ability to complete the desired objective.

U……………………………………………………………………………………………………

Unit: The testing of the application by the developers at a component or modular level, looking at whether the code is performing as expected.

User Acceptance: The means of testing applied, normally by members of the business or recipients of the system, whilst still in the test environment. This looks to ensure that business requirements have been understood by all involved in the application production as well as providing an opportunity to check the business processes that the application will be used in conjunction with.

V……………………………………………………………………………………………………

V-Model: A testing focussed software development life-cycle where the process of creativity occurs down the left side of the V and the process of testing up the right side of the V, where each step down the V has a corresponding point of testing up the right of the V. Begins with a requirement and terminates with User Acceptance Testing.

Version: An alpha numeric means of identifying a particular build of software, or a component thereof. The version changes incrementally to reflect each modification of the software, component, document etc.

Volume: (Testing) A type of performance testing which increases the volume of users on a system, in each cycle, taking the volume up to a prescribed limit, normally exceeding optimum load.

W……………………………………………………………………………………………………

W3C: World Wide Web Consortium – body creating web standards and guidelines. http://www.w3.org/

WAI: Web Accessibility Initiative. See: http://www.w3.org/WAI/

Waterfall: Project Management Method whereby each element of the software development life-cycle occurs sequentially.

Win Runner: A tool that was historically used for test automation. See Automation Centre.

White Box: Testing normally performed by developers, looking at the code and ensuring that each model of code is performing as expected. See Unit

X……………………………………………………………………………………………………

Y…………………………………………………………………………………………………….
Z…………………………………………………………………………………………………….

How to Annoy a Test Manager

A little light relief from all the serious blogging, but some of these do really get me angry.

The coding has finished and we need to test – should we be talking to you now?

I know you were expecting the code today, but when do you really need it?

Can you please stop raising defects as we don’t have time to fix that many problems?

We have only got half the time we thought we had for testing so how are you going to get all of your work done?

We know the quality is bad, but we are releasing anyway?

UAT has been dropped because the business can’t free any resources.

The fact that this problem was fixed in an earlier release is not indicative of poor configuration management.

Why do we need developers, support and environment resources to work the weekend as well as testers?

The environment is nearly the same as live, we just don’t know how nearly!

Testing is supposed to make the software perfect. Why hasn’t it worked?

We have spent 20% of our budget on analysis, 60% on design and development and 10% on other items. You have 10% left, what can you do for us?

We are running out of money because the test execution is taking too long, so we are going to let the testers go and the developers can finish the job.

Don’t worry about getting testers. We have lots of people in the business that we can get to test at lunchtimes.

The supplier was not asked to perform Unit testing and we can’t afford to have it done now as it was not in the budget.

Testing is a service and as such we will let you know when we need you.

I know you asked for three environments, but we can only get one. Don’t worry. I am sure everyone can get on it at the same time.

The testing is occurring on the development environment. That way the developers can change things whenever they need to.

The problems in live are fixed out of hours so we can’t test them.

If we automate all of the testing how much time will it save us?

Why do we need testers on board before the testing starts?

The user spilt coffee on their system and it crashed. I bet you didn’t test for that did you?

We have too many high severity defects, so we are going to hold a meeting and see how many we can downgrade?

We need to do some performance testing. There are no non-functional requirements, so let’s just assume that everything must happen in under a second.

Testing is easy. Anyone can do it.

It’s not a bug. It’s a design feature.

The requirements were captured at the start of the project. But don’t read that it’s way out of date.

Please feel free to add your own by way of comments.

Reinventing the Testing Wheel (Object Oriented Testing)

I have been thinking about this one for some time and wondered what other views existed. For some time developers have taken a more modular approach to programming, applying techniques such as Object Oriented Design, in order to improve efficiency. Why re-invent the wheel each time a certain piece of code is required? Take something like a calculation of VAT. Once this has been written the first time, that should be enough and if stored and indexed correctly, each time someone else needs a VAT calculation they call a pre-written routine from a library. Not only does this mean that the developer does not spend valuable time writing the same code again, but also that the module should already have been tested and therefore be reliable, reducing the likelihood of errors.

This was seen as a major break through in the development world, but one which has not been replicated to my knowledge in the testing arena. When we write a manual test script, we write it in a manner which is long hand. So how many times have we written scripts to test VAT calculations? Why are we not creating a library of scripts that are re-usable by all testers? Why do we have to sit and think each time that we test a field: to understand what all the permutations are that we need to deal with, how to test it positively and negatively? There are hundreds of thousands of testers all over the world that are repeating the same thought processes, writing the same scripts, duplicating their efforts.

How many times has an input of a person's age been tested? What are all of the parameters? How old before they are at school, how old before they can hold a credit card, bank account, retire? All of these questions have been thought about so often, but we still religiously recreate them each time. So testers are accused of being too slow or not moving with the times, being overly expensive. Well I suppose when you look at something like this, the answer has to be, "Yes".

So when we are asked what the future may hold for testers, I put on my rose-tinted spectacles and imagine…..

I imagine a world where we are sharing our knowledge and expertise, generating a global testing community and capability. One where we are seen as slick operators, contributing massively to technology programmes the world over, with the minimum of input for the maximum return.

How does this happen? If there were a central repository for scripts to which every tester contributed, where we went as a first port of call to locate our script components, or store those components we have to create, we would quickly generate a wealth of stored expertise. A Wikipedia of test scripts – wow! We would need to be able to locate scripts in a fast and efficient manner and the database used to store the script components or objects would need to be referenced in a multitude of ways. But this is not new technology. Developers have been doing a lot of this for years and we can no doubt learn from the mistakes that they made along the way.

I would welcome comment on this. Is it feasible? How could it work? Am I barking mad? If you are a developer and have a story about OOD then share it and let us know if you think this is feasible. Drop me a line and let me know.

Test Script Storage

Storage of anything should be in a manner whereby retrieval of the item is easy. If we look at a library, everyone is familiar with the process. Find the section dealing with your subject matter and then search for the book by author. It has a proven track record and we are all able to go into a library and find what we are looking for (as long as it is there).

Test scripts are no different. As testers we generate vast volumes of scripts and need to be able to re-use them. It strikes me as odd therefore, that scripts are so often stored against a project. There may be subdivisions within the project structure for each of the applications being tested, perhaps even a definition of which version the scripts have been written against. Indeed If the testers are a constant from one phase of the project to the next, the scripts are likely to be re-used.

But for each project there is normally a primary application that is being tested and the others are interfacing systems. The scripts that are generated dealing with the interface do not necessarily cover the entire application. The first problem occurs when someone outside of the project needs to test the primary application. If they are unaware of the project name, the chances are that they will produce their own set of scripts. We now have a duplication of effort occurring. Each new script that is written is potentially wasting effort.

The second problem is that the scripts that were written to test a change to an interfacing system, will not be stored in a way that is usable or locatable by those testing it as a primary application. This generates two problems, the first being that the test suite for the interfacing system is now out of date. The second problem is that the regression pack has not been updated to reflect the changes and therefore future changes may introduce problems into the new code which will not be identified. At best the difference in version may alert the testers to the fact that a change has occurred and time will be spent trying to re-locate the scripts, or more likely, they will find out what the change was and generate their own scripts.

I believe the way to avoid this is to ensure that scripts are not stored by project, but always by application. The application should then be divided into a series of suites, one for each version. Lastly a regression pack should exist as a permanent fixture. This allows the storage of scripts as deltas to the original suite and then as each new version of the application is developed, the scripts can be reviewed for those which would best augment the regression pack. It may be prudent to have each version contain all scripts, as some will have been modified or removed from the test suite.

This means that any one on any project, can look at the applications they are working with and know without doubt, where to look for the scripts. I can only see one argument against this, which is around end-to-end tests and UAT, where the focus on the application is not present and the focus is more around the business process. In such instances these scripts may need to be stored against the project.

To my way of thinking, this is symptomatic of testing being seen as a service to a project. Therefore the need is to satisfy the project’s needs by storing everything in a very project-centric manner. The reality is that this is costing organisations a vast amount of time and money as scripts are permanently re-generated in order to meet the current requirement.
The situation becomes worse where large volumes of external resources are involved. A lack of consistency and knowledge leaving the client organisation becomes an issue and the likelihood of finding a previously written set of scripts lowers dramatically.

Lastly, the regression pack is key to ensure the stability of the production platform and anything which compromises the regression pack undermines the efficiency of the business.

So – to summarise, ensure that test scripts are written and stored by application wherever possible. Breakdown the scripts under the application folder by version and have a generic regression pack sub-folder either at application level or version level.