SOFTWARE TESTING GLOSSARY (Update)

This glossary is a living post – so will be edited as we come across terminology that is not included. If you have any suggestions or disagree with an explanation, drop us an e-mail and let us know. TCL India offer this as a means of establishing glossaries of your own or as a point of reference. This update includes 10 new definitions.

A…………………………………………………………………………………………………

Accessibility: (Testing)Looks at an application to ensure that those with disabilities will be able to use the application as an able individual.

Agile: A development method, involving the creation of software, with the contributing parties, including testing, all working on the same item at the same time.

Alpha: (Testing)Testing of an application which is in production, but not available for general use. This is normally performed by users internal to the business.

Analyst: Person who interacts with the business in order to understand and document their requirements for an application or change to an existing one.

Analyst (Test): Person responsible for the preparation and execution of test scripts, recording and progressing of defects, reporting into the Test Team Leader or Test Manager.

Automation: The process of developing a script or program which can be run by software rather than a Test Engineer, to test an aspect of an application. Automation is used to increase the speed of test execution.

Automation Centre: A tool used in the automation of software testing. Seehttps://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-127-24%5E1074_4000_100__&jumpid=reg_R1002_USEN

B……………………………………………………………………………………………………

Beta: (Testing)Testing of an application which is in production, but not available for general use. This is normally performed by a select group of friendly external testers.

Black Box: The process of testing without understanding the internal workings of an application, relying on inputs and outputs only.

Bug: See DefectBusiness Requirement Specification: See Requirement Specification

C……………………………………………………………………………………………………

Case: See ScenarioCode: The software that has been produced by development and is being subjected to testing.

Completion Report: A document produced during the testing and presented to the project manager on completion of testing activity. The document should detail the tests that have been performed along with the associated results and defects. Some reports may include a recommendation for the applications suitability for release to production.

Component: The smallest item that is testable or producible. This often refers to a single file of code.

Configuration Management: The means of managing the components required to make a larger item. Applicable to files, documents, data, hardware, software, tools. Understanding of the versions of each component required in order to be able to re-build the larger item.

Criteria: See Entry Criteria and Exit Criteria

D……………………………………………………………………………………………………

Data: Information pertaining to the use of a system or recorded through the use of a system. Data is used in order to test a system, potentially after data cleansing if personal information is involved.

Data Generator: A tool used to generate high volumes of data in order to be able to test many permutations, or to load test an item.

Data Protection Act (DPA): The act that determines how personal data should be used and protected from abuse.

Data Scrambling: The process of altering personal information from data to be used for testing purposes.

Defect: Where the item under test has been found to inaccurate, resulting from testing. Primarily used in associated with software, but equally valid for static testing of documentation.

Defect Detection: The means of identifying a defect. Can be a metric used to predict the volume of defects expected during the course of a project, or as a means of looking back at a project to understand where testing needs to be concentrated in future projects of a similar nature.

Defect Removal Efficiency: A metric used to assess the ability of testing to remove defects as they are introduced, during the software development life-cycle, keeping the cost of testing later phases to a minimum.

Defect Turnaround: The time taken from the identification of a defect, through to the point of resolution. Different levels of granularity may be used. e.g. A focus on the time taken by development.

Developer: A person responsible for the development of an application.

Development: A process of producing an application by production of low level design, code, unit testing and integration in the small testing.

Dynamic: Testing which occurs on the right hand side of the V-model, with the application present.

E……………………………………………………………………………………………………

Environment: The combination of hardware, software and data as used in development, testing and production. The platform/s upon which the testing occurs.

Entry Criteria: The criteria that must be met prior to a deliverable entering the next phase of testing to another. This is normally associated with documented test assets and pre-agreed volumes of defects.

Error: See Defect

Exit Criteria: The criteria that must be met prior to a deliverable leaving the current phase of testing. This is normally associated with documented test assets and pre-agreed volumes of defects.

Execution: The process of working through a script on the application under test, in the testing environment.

F……………………………………………………………………………………………………

Functional: (Testing) The testing of a products function, against requirements and design.

Functional Specification: A document which extracts all of the functional requirements from the requirement specification.

G……………………………………………………………………………………………………

Glass Box: See White Box

Grey Box: Testing performed by testers with some knowledge of the internal workings of an application. See also Black Box and White Box testing.

H……………………………………………………………………………………………………

High Level Design: A design showing the major components and how these will interface to each other, defining the hardware to be used and the software that will be impacted.

I…………………………………………………………………………………………………… 


Integration in the Large: Where the application or applications that have been developed are brought together along with those which have remained unchanged, building a production like system around the application/s. Testing is then applied looking at the communication between the different applications.

Integration in the Small: Where the components of the application that have been developed are brought together along with those which have remained unchanged, building the application or major component of a single application. Testing is then applied looking at the communication between the different components.Integration: The act of bringing many parts together to form a whole.

ISEB: Information Systems Examination Board. This was historically the board that was used to certify test professionals at either Foundation (Entry Level) or Practitioner Level (Advanced). See:http://www.bcs.org/server.php?show=nav.5732

ISTQB: International Software Testing Qualifications Board. See:http://www.istqb.org/

J…………………………………………………………………………………………………….
K……………………………………………………………………………………………………

Key Performance Indicator: A mechanism built on one or metrics, which determines a band of acceptable performance, which over time is often targeted towards improvement.

L……………………………………………………………………………………………………

Live: See Production

Load: One of the types of performance testing, this looks at testing for the maximum load on the system.

Load Runner: Tool used to performance test one or many applications, to understand how it handles increases in load. See:https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-15-17%5e8_4000_100__

Low Level Design: Definition of exactly how the application/s will be modified or produced in order to meet the requirements and the high level design. This can extend in some examples to elements of pseudo code being defined.

M……………………………………………………………………………………………………

Metric: A measure of an attribute or occurrence in connection with an organisation, department or functions performance.

N……………………………………………………………………………………………………..

Non-functional: How an application performs, rather than how it does it.

Non-functional Specification: A document which details the non-functional requirements such as performance, security, operational elements.

O……………………………………………………………………………………………………
P……………………………………………………………………………………………………

Performance Centre: A tool used for measuring the performance of an application or series of applications. See:https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-126-17_4000_100__

Performance: Used to describe many types of testing relating to the speed of an application. See Volume, Load and Stress.

Plan: A document produced for a type of testing defining how the testing will be performed, when the testing will be performed and who will be performing it. The test plan lists the test scenarios that will be scripted.

Preparation: The process of generating the test scripts.

Prince2: A project management methodology that is used amongst a lot of blue chip organizations to bring discipline to the software development life cycle.

Priority: The importance of fixing a defect from a business perspective. Defined by business representatives.

Production: The area or a computer network that contains applications which are in use by real users and contains real data. The area that applications are released to on completion of a project.

Q……………………………………………………………………………………………………

Quality: The suitability of an item for its intended purpose and how well it achieves this.

Quality Centre: Tool used to assist with the management of testing, recording and tracking scripts, logging and tracking defects and more. See:https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-127-24_4000_100__

R……………………………………………………………………………………………………

Regression: Retesting of a previously tested program following modification to ensure that faults have not been introduced or uncovered as a result of the changes made.

Requirement Specification: A document normally produced by a business analyst, capturing the needs of the individual in a manner which means that they can be translated into a software solution.

Re-Test: Taking a defect which has failed and executing the associated test script again.

S……………………………………………………………………………………………………

Safety Critical: Used to identify something for which the use has an impact on personal safety, such as medicinal applications or those used by rescue services.

Scenario: A high level description of how a requirement is going to be tested.

Script: Referable back to the scenario, the script defines each step that must be passed through in order to perform a test, along with the expected results. As the script is executed, results are recorded and if they match the expected result are marked as passed, otherwise as failed. A script containing a failure should have a resultant defect raised.

Schedule: A document, similar to a project plan, but detailing the activities associated with testing, when they are due to occur and who will be performing them.

Severity: The importance of a defect to testing and the application. Defined by Testers

Special Interest Group In Software Testing (SIGIST): A group that has been set up by the British Computer Society, made up of people interested in the subject of software testing.

Smoke: A process of proving an application or environment is ready to commence full testing, by running a sample set of scripts testing the primary functionality and/or connectivity.

Software Development Life Cycle (SDLC): The process of taking a requirement and translating it into a fully working application.

Static: The process of testing without the presence of software. Normally refers to the testing of documentation.

Strategy: A document produced at project or programme level, defining what testing is to be performed.

Stress: A form of performance testing, whereby the volume of testing is increased to a point at which the application is deemed to be failing to perform, either due to failure to meet performance criteria or system breakdown.

T……………………………………………………………………………………………………

Technical Architecture Document: Definition of how the business requirements should be translated into a technical solution at the highest level.

Test Bed: A cumulative term used for all of the test environments used by the test department.

Test Director: A tool used for test management, capture of requirements, scripts and defects, with the ability to manage defects through to resolution. Now replaced by Quality Centre.

Test Maturity Model (TMM): A means of measuring a testing department’s level of maturity in a similar manner to that of the capability maturity model (CMM).

Testing: The process of reviewing an item for accuracy and its ability to complete the desired objective.

U……………………………………………………………………………………………………

Unit: The testing of the application by the developers at a component or modular level, looking at whether the code is performing as expected.

User Acceptance: The means of testing applied, normally by members of the business or recipients of the system, whilst still in the test environment. This looks to ensure that business requirements have been understood by all involved in the application production as well as providing an opportunity to check the business processes that the application will be used in conjunction with.

V……………………………………………………………………………………………………

V-Model: A testing focussed software development life-cycle where the process of creativity occurs down the left side of the V and the process of testing up the right side of the V, where each step down the V has a corresponding point of testing up the right of the V. Begins with a requirement and terminates with User Acceptance Testing.

Version: An alpha numeric means of identifying a particular build of software, or a component thereof. The version changes incrementally to reflect each modification of the software, component, document etc.

Volume: (Testing) A type of performance testing which increases the volume of users on a system, in each cycle, taking the volume up to a prescribed limit, normally exceeding optimum load.

W……………………………………………………………………………………………………

W3C: World Wide Web Consortium – body creating web standards and guidelines. http://www.w3.org/

WAI: Web Accessibility Initiative. See: http://www.w3.org/WAI/

Waterfall: Project Management Method whereby each element of the software development life-cycle occurs sequentially.

Win Runner: A tool that was historically used for test automation. See Automation Centre.

White Box: Testing normally performed by developers, looking at the code and ensuring that each model of code is performing as expected. See Unit

X……………………………………………………………………………………………………

Y…………………………………………………………………………………………………….
Z…………………………………………………………………………………………………….

How to Annoy a Test Manager

A little light relief from all the serious blogging, but some of these do really get me angry.

The coding has finished and we need to test – should we be talking to you now?

I know you were expecting the code today, but when do you really need it?

Can you please stop raising defects as we don’t have time to fix that many problems?

We have only got half the time we thought we had for testing so how are you going to get all of your work done?

We know the quality is bad, but we are releasing anyway?

UAT has been dropped because the business can’t free any resources.

The fact that this problem was fixed in an earlier release is not indicative of poor configuration management.

Why do we need developers, support and environment resources to work the weekend as well as testers?

The environment is nearly the same as live, we just don’t know how nearly!

Testing is supposed to make the software perfect. Why hasn’t it worked?

We have spent 20% of our budget on analysis, 60% on design and development and 10% on other items. You have 10% left, what can you do for us?

We are running out of money because the test execution is taking too long, so we are going to let the testers go and the developers can finish the job.

Don’t worry about getting testers. We have lots of people in the business that we can get to test at lunchtimes.

The supplier was not asked to perform Unit testing and we can’t afford to have it done now as it was not in the budget.

Testing is a service and as such we will let you know when we need you.

I know you asked for three environments, but we can only get one. Don’t worry. I am sure everyone can get on it at the same time.

The testing is occurring on the development environment. That way the developers can change things whenever they need to.

The problems in live are fixed out of hours so we can’t test them.

If we automate all of the testing how much time will it save us?

Why do we need testers on board before the testing starts?

The user spilt coffee on their system and it crashed. I bet you didn’t test for that did you?

We have too many high severity defects, so we are going to hold a meeting and see how many we can downgrade?

We need to do some performance testing. There are no non-functional requirements, so let’s just assume that everything must happen in under a second.

Testing is easy. Anyone can do it.

It’s not a bug. It’s a design feature.

The requirements were captured at the start of the project. But don’t read that it’s way out of date.

Please feel free to add your own by way of comments.

Role of the Environment Manager

One area that I have regularly come across which needs further clarification is the role of the Test Environment Manager. I have produced this article for those people looking to understand what exactly should be expected of an Environment Manager and could be used as the basis for a job description or role profile.

Test Environments are required in order to enable the test execution to occur. The Test Environment Manager is responsible for the provision of the test environment, its configuration status, its stability and maintenance. They also provide administrative activities such as backups and purging on request of the project, monitor usage of environments, supplying support throughout the test execution, including out of hours working if required. They will be expected to manage change to the test environments and keep the programme and testing informed of status. They must be able to give advance warning of downtime of any component of the test environment, which may cover multiple platforms. Duties are predominantly based around the organisation’s infrastructure but can extend to cover interfacing with 3rd party organisations which are offering equipment to form part of the project environment.

The environment manager is responsible for ensuring the provision of suitable test environments for all aspects of the testing of the core system. A suitable test environment is one that meets the requirements for the phase of testing and those requirements will be documented and provided to the Environment Manager by the Test Manager.

They shall ensure that the interfacing systems test environments are available and of a known and controlled configuration. For the core system, the environment should be available two weeks prior to test execution commencement, so giving time for any difficulties to be resolved. The environment must remain available for the duration of the project. As many peripheral systems should be delivered on similar timescales.

Thought must be given to the possibility of different environments being required, where phases of testing overlap, or where the requirements of the testing phase are such that the provided environment needs to be changed or upgraded. Each time a new environment is made available, this should also occur two weeks prior to test execution occurring in the new environment. The Environments will be placed under change control and will always be of a known state of configuration.

No upgrade of hardware or software may occur to a test environment without the approval of the Environment Manager. The Environment Manager must place change control over the environments and maintain that level of control. This may be a formalised mechanism, but as a minimum, requires that the Environment Managers approval has been received prior to the change occurring. All changes must be communicated to the following people as a minimum:

1. Development Manager
2. Testing Manager
3. UAT Manager
4. Project Manager

It is expected that this list will grow as time passes and the individual users of the system become known. It is felt that the above will form a minimum conduit for such communications until such time as all individuals are known.

A record of the configuration should be maintained for each environment or element of an environment if multiple environments are required in order to make a whole. The record should detail the hardware build that has been provided by Technical Operations, including versions of operating systems and other utilities provided as part of the basic provision. Over and above this, each item of software placed onto the environment for test, shall have its version recorded, so enabling a clear picture of the configuration at any given time. Any change to any aspect of the configuration should be approved under the change control mechanism and logged in the Configuration Record.

Should the software under test be configurable, a document with its own version control shall be supplied to the Environment Manager so that they can use this as a state of software configuration. If changes are required to that configuration, these should be recorded in an upgraded version of the document and the Configuration Record updated to reflect that this has happened.

Ad hoc changes to the configuration, be that hardware, software or software configuration, are not permissible. Alteration to the configuration must be controlled and the configuration must always be maintained in a known and managed state.

The Environment manager, will ensure that there is support for the system during all times of test execution, be this late at night or over weekends and bank holidays. They will make known a list of all the different elements of the environments and the contact names and numbers of the people that they support.

A plan shall be maintained by the environment manager of who is using what environment for what purpose at what time. This needs to be maintained so as to ensure that changes in environment usage are managed and controlled and so that if multiple environments are required, this can be kept to a minimum by the intelligent employment of the environment resources available.

The Environment Manager, as a result of the change control mechanism, will be able to notify the users of the test environments of any outages. These changes shall be recorded in an outage log, so that upon project completion, the details of outages can be formally reported. It will also enable monitoring of any problems that may occur in this area and be used to aid resolution of repetitive issues.

As part of the standard weekly output, the environment manager will distribute the following information.

1. Planned usage for the week.
2. Planned outages for the week.
3. Planned changes for the week.
4. The statement of configuration for each environment.
5. Planned Backups

The last item on the list is backups. These shall be available to all testers upon request and shall be recorded and requested by the Environment Manager.

I hope that this is found to be useful and informative.

Importance of Accurate Test Environments

Look - my new test bed. What a brilliant image! How many of us go here for our next test environment.

Do people realise the importance of the environment to the tester? I don’t think they can. I have spent most of my testing life working with make-do arrangements and being expected to test to high standards. I had a conversation with a fellow tester today, who is currently working as an Environment Manager and he and I were both moaning about the problem.

So what happens? Firstly all of the investment on a project is in the production equipment. As the project is creating application X for the first time, the to-be production environment can be used for testing purposes. It is ideal in this situation because it means that we can test on the most suitable infrastructure possible – what it will run on in live.

The problem comes the next time the application is updated. There was no investment in test infrastructure as part of the original project – they didn’t need it. So now the update project has to justify a massive expenditure in test equipment, which could potentially jeopardise the projects financial benefit. This step is therefore avoided and the hunt begins. We need to find a piece of kit that can be used to test the application. Something which is similar to that in production would do. Alright something which resembles the production kit, if you squint at it whilst wearing dark glasses. The good news comes through. Kit has been found and once the rust is scraped off, it is found to be an earlier version of the kit in production. The RAM is reduced, the processor is 2 years earlier than that which is in production and the operating system has fallen out of support. Never mind, it is all we have. Then the test manager explains that he needs three environments to be able to complete the testing on schedule. Don’t worry we can whack in some more RAM and increase the disk space. It’s still not half as good as production but we can split it into three for you.

This is seen as a pragmatic solution by all involved and the test manager must now do his best. (Agggghhhhh!!!!!!!). Let me explain my frustration. As soon as you compromise the test environment, you compromise the testing. So many times defects are raised which are directly attributable to the environment and if that environment is not the same as production, time is going to be spent resolving defects which will not exist in production and potentially defects which will occur due to the production environment will not be located during testing. Certainly in terms of release testing, the process is likely to be different and effectively the results during testing are false and cannot be relied upon. The lack of adequate test environments becomes a project risk in its’ own right and should be logged in the project risk register accordingly.

It is critical that the testing environment is as close to production as possible and that any deviations are fully understood. Only with the differences between production and testing environments being known and measurable can you rely on the testing results.

Take performance testing. If you are running with less RAM, less disk space, less process power, how will this manifest itself in the results? As for security testing, the operating system version becomes paramount as the level of inherent security is likely to change. Know and document the differences and if this cannot be achieved be careful about signing off the product as fit for release.

As a recommendation to all project managers, ensure that you leave an inheritance of test infrastructure for the following projects. Placing a new piece of equipment live is not sufficient. Consideration must be given to ongoing support, testing, development and training and what environments are going to be used for these.