SOFTWARE TESTING GLOSSARY (Update)

This glossary is a living post – so will be edited as we come across terminology that is not included. If you have any suggestions or disagree with an explanation, drop us an e-mail and let us know. TCL India offer this as a means of establishing glossaries of your own or as a point of reference. This update includes 10 new definitions.

A…………………………………………………………………………………………………

Accessibility: (Testing)Looks at an application to ensure that those with disabilities will be able to use the application as an able individual.

Agile: A development method, involving the creation of software, with the contributing parties, including testing, all working on the same item at the same time.

Alpha: (Testing)Testing of an application which is in production, but not available for general use. This is normally performed by users internal to the business.

Analyst: Person who interacts with the business in order to understand and document their requirements for an application or change to an existing one.

Analyst (Test): Person responsible for the preparation and execution of test scripts, recording and progressing of defects, reporting into the Test Team Leader or Test Manager.

Automation: The process of developing a script or program which can be run by software rather than a Test Engineer, to test an aspect of an application. Automation is used to increase the speed of test execution.

Automation Centre: A tool used in the automation of software testing. Seehttps://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-127-24%5E1074_4000_100__&jumpid=reg_R1002_USEN

B……………………………………………………………………………………………………

Beta: (Testing)Testing of an application which is in production, but not available for general use. This is normally performed by a select group of friendly external testers.

Black Box: The process of testing without understanding the internal workings of an application, relying on inputs and outputs only.

Bug: See DefectBusiness Requirement Specification: See Requirement Specification

C……………………………………………………………………………………………………

Case: See ScenarioCode: The software that has been produced by development and is being subjected to testing.

Completion Report: A document produced during the testing and presented to the project manager on completion of testing activity. The document should detail the tests that have been performed along with the associated results and defects. Some reports may include a recommendation for the applications suitability for release to production.

Component: The smallest item that is testable or producible. This often refers to a single file of code.

Configuration Management: The means of managing the components required to make a larger item. Applicable to files, documents, data, hardware, software, tools. Understanding of the versions of each component required in order to be able to re-build the larger item.

Criteria: See Entry Criteria and Exit Criteria

D……………………………………………………………………………………………………

Data: Information pertaining to the use of a system or recorded through the use of a system. Data is used in order to test a system, potentially after data cleansing if personal information is involved.

Data Generator: A tool used to generate high volumes of data in order to be able to test many permutations, or to load test an item.

Data Protection Act (DPA): The act that determines how personal data should be used and protected from abuse.

Data Scrambling: The process of altering personal information from data to be used for testing purposes.

Defect: Where the item under test has been found to inaccurate, resulting from testing. Primarily used in associated with software, but equally valid for static testing of documentation.

Defect Detection: The means of identifying a defect. Can be a metric used to predict the volume of defects expected during the course of a project, or as a means of looking back at a project to understand where testing needs to be concentrated in future projects of a similar nature.

Defect Removal Efficiency: A metric used to assess the ability of testing to remove defects as they are introduced, during the software development life-cycle, keeping the cost of testing later phases to a minimum.

Defect Turnaround: The time taken from the identification of a defect, through to the point of resolution. Different levels of granularity may be used. e.g. A focus on the time taken by development.

Developer: A person responsible for the development of an application.

Development: A process of producing an application by production of low level design, code, unit testing and integration in the small testing.

Dynamic: Testing which occurs on the right hand side of the V-model, with the application present.

E……………………………………………………………………………………………………

Environment: The combination of hardware, software and data as used in development, testing and production. The platform/s upon which the testing occurs.

Entry Criteria: The criteria that must be met prior to a deliverable entering the next phase of testing to another. This is normally associated with documented test assets and pre-agreed volumes of defects.

Error: See Defect

Exit Criteria: The criteria that must be met prior to a deliverable leaving the current phase of testing. This is normally associated with documented test assets and pre-agreed volumes of defects.

Execution: The process of working through a script on the application under test, in the testing environment.

F……………………………………………………………………………………………………

Functional: (Testing) The testing of a products function, against requirements and design.

Functional Specification: A document which extracts all of the functional requirements from the requirement specification.

G……………………………………………………………………………………………………

Glass Box: See White Box

Grey Box: Testing performed by testers with some knowledge of the internal workings of an application. See also Black Box and White Box testing.

H……………………………………………………………………………………………………

High Level Design: A design showing the major components and how these will interface to each other, defining the hardware to be used and the software that will be impacted.

I…………………………………………………………………………………………………… 


Integration in the Large: Where the application or applications that have been developed are brought together along with those which have remained unchanged, building a production like system around the application/s. Testing is then applied looking at the communication between the different applications.

Integration in the Small: Where the components of the application that have been developed are brought together along with those which have remained unchanged, building the application or major component of a single application. Testing is then applied looking at the communication between the different components.Integration: The act of bringing many parts together to form a whole.

ISEB: Information Systems Examination Board. This was historically the board that was used to certify test professionals at either Foundation (Entry Level) or Practitioner Level (Advanced). See:http://www.bcs.org/server.php?show=nav.5732

ISTQB: International Software Testing Qualifications Board. See:http://www.istqb.org/

J…………………………………………………………………………………………………….
K……………………………………………………………………………………………………

Key Performance Indicator: A mechanism built on one or metrics, which determines a band of acceptable performance, which over time is often targeted towards improvement.

L……………………………………………………………………………………………………

Live: See Production

Load: One of the types of performance testing, this looks at testing for the maximum load on the system.

Load Runner: Tool used to performance test one or many applications, to understand how it handles increases in load. See:https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-15-17%5e8_4000_100__

Low Level Design: Definition of exactly how the application/s will be modified or produced in order to meet the requirements and the high level design. This can extend in some examples to elements of pseudo code being defined.

M……………………………………………………………………………………………………

Metric: A measure of an attribute or occurrence in connection with an organisation, department or functions performance.

N……………………………………………………………………………………………………..

Non-functional: How an application performs, rather than how it does it.

Non-functional Specification: A document which details the non-functional requirements such as performance, security, operational elements.

O……………………………………………………………………………………………………
P……………………………………………………………………………………………………

Performance Centre: A tool used for measuring the performance of an application or series of applications. See:https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-126-17_4000_100__

Performance: Used to describe many types of testing relating to the speed of an application. See Volume, Load and Stress.

Plan: A document produced for a type of testing defining how the testing will be performed, when the testing will be performed and who will be performing it. The test plan lists the test scenarios that will be scripted.

Preparation: The process of generating the test scripts.

Prince2: A project management methodology that is used amongst a lot of blue chip organizations to bring discipline to the software development life cycle.

Priority: The importance of fixing a defect from a business perspective. Defined by business representatives.

Production: The area or a computer network that contains applications which are in use by real users and contains real data. The area that applications are released to on completion of a project.

Q……………………………………………………………………………………………………

Quality: The suitability of an item for its intended purpose and how well it achieves this.

Quality Centre: Tool used to assist with the management of testing, recording and tracking scripts, logging and tracking defects and more. See:https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-127-24_4000_100__

R……………………………………………………………………………………………………

Regression: Retesting of a previously tested program following modification to ensure that faults have not been introduced or uncovered as a result of the changes made.

Requirement Specification: A document normally produced by a business analyst, capturing the needs of the individual in a manner which means that they can be translated into a software solution.

Re-Test: Taking a defect which has failed and executing the associated test script again.

S……………………………………………………………………………………………………

Safety Critical: Used to identify something for which the use has an impact on personal safety, such as medicinal applications or those used by rescue services.

Scenario: A high level description of how a requirement is going to be tested.

Script: Referable back to the scenario, the script defines each step that must be passed through in order to perform a test, along with the expected results. As the script is executed, results are recorded and if they match the expected result are marked as passed, otherwise as failed. A script containing a failure should have a resultant defect raised.

Schedule: A document, similar to a project plan, but detailing the activities associated with testing, when they are due to occur and who will be performing them.

Severity: The importance of a defect to testing and the application. Defined by Testers

Special Interest Group In Software Testing (SIGIST): A group that has been set up by the British Computer Society, made up of people interested in the subject of software testing.

Smoke: A process of proving an application or environment is ready to commence full testing, by running a sample set of scripts testing the primary functionality and/or connectivity.

Software Development Life Cycle (SDLC): The process of taking a requirement and translating it into a fully working application.

Static: The process of testing without the presence of software. Normally refers to the testing of documentation.

Strategy: A document produced at project or programme level, defining what testing is to be performed.

Stress: A form of performance testing, whereby the volume of testing is increased to a point at which the application is deemed to be failing to perform, either due to failure to meet performance criteria or system breakdown.

T……………………………………………………………………………………………………

Technical Architecture Document: Definition of how the business requirements should be translated into a technical solution at the highest level.

Test Bed: A cumulative term used for all of the test environments used by the test department.

Test Director: A tool used for test management, capture of requirements, scripts and defects, with the ability to manage defects through to resolution. Now replaced by Quality Centre.

Test Maturity Model (TMM): A means of measuring a testing department’s level of maturity in a similar manner to that of the capability maturity model (CMM).

Testing: The process of reviewing an item for accuracy and its ability to complete the desired objective.

U……………………………………………………………………………………………………

Unit: The testing of the application by the developers at a component or modular level, looking at whether the code is performing as expected.

User Acceptance: The means of testing applied, normally by members of the business or recipients of the system, whilst still in the test environment. This looks to ensure that business requirements have been understood by all involved in the application production as well as providing an opportunity to check the business processes that the application will be used in conjunction with.

V……………………………………………………………………………………………………

V-Model: A testing focussed software development life-cycle where the process of creativity occurs down the left side of the V and the process of testing up the right side of the V, where each step down the V has a corresponding point of testing up the right of the V. Begins with a requirement and terminates with User Acceptance Testing.

Version: An alpha numeric means of identifying a particular build of software, or a component thereof. The version changes incrementally to reflect each modification of the software, component, document etc.

Volume: (Testing) A type of performance testing which increases the volume of users on a system, in each cycle, taking the volume up to a prescribed limit, normally exceeding optimum load.

W……………………………………………………………………………………………………

W3C: World Wide Web Consortium – body creating web standards and guidelines. http://www.w3.org/

WAI: Web Accessibility Initiative. See: http://www.w3.org/WAI/

Waterfall: Project Management Method whereby each element of the software development life-cycle occurs sequentially.

Win Runner: A tool that was historically used for test automation. See Automation Centre.

White Box: Testing normally performed by developers, looking at the code and ensuring that each model of code is performing as expected. See Unit

X……………………………………………………………………………………………………

Y…………………………………………………………………………………………………….
Z…………………………………………………………………………………………………….

Test Script Storage

Storage of anything should be in a manner whereby retrieval of the item is easy. If we look at a library, everyone is familiar with the process. Find the section dealing with your subject matter and then search for the book by author. It has a proven track record and we are all able to go into a library and find what we are looking for (as long as it is there).

Test scripts are no different. As testers we generate vast volumes of scripts and need to be able to re-use them. It strikes me as odd therefore, that scripts are so often stored against a project. There may be subdivisions within the project structure for each of the applications being tested, perhaps even a definition of which version the scripts have been written against. Indeed If the testers are a constant from one phase of the project to the next, the scripts are likely to be re-used.

But for each project there is normally a primary application that is being tested and the others are interfacing systems. The scripts that are generated dealing with the interface do not necessarily cover the entire application. The first problem occurs when someone outside of the project needs to test the primary application. If they are unaware of the project name, the chances are that they will produce their own set of scripts. We now have a duplication of effort occurring. Each new script that is written is potentially wasting effort.

The second problem is that the scripts that were written to test a change to an interfacing system, will not be stored in a way that is usable or locatable by those testing it as a primary application. This generates two problems, the first being that the test suite for the interfacing system is now out of date. The second problem is that the regression pack has not been updated to reflect the changes and therefore future changes may introduce problems into the new code which will not be identified. At best the difference in version may alert the testers to the fact that a change has occurred and time will be spent trying to re-locate the scripts, or more likely, they will find out what the change was and generate their own scripts.

I believe the way to avoid this is to ensure that scripts are not stored by project, but always by application. The application should then be divided into a series of suites, one for each version. Lastly a regression pack should exist as a permanent fixture. This allows the storage of scripts as deltas to the original suite and then as each new version of the application is developed, the scripts can be reviewed for those which would best augment the regression pack. It may be prudent to have each version contain all scripts, as some will have been modified or removed from the test suite.

This means that any one on any project, can look at the applications they are working with and know without doubt, where to look for the scripts. I can only see one argument against this, which is around end-to-end tests and UAT, where the focus on the application is not present and the focus is more around the business process. In such instances these scripts may need to be stored against the project.

To my way of thinking, this is symptomatic of testing being seen as a service to a project. Therefore the need is to satisfy the project’s needs by storing everything in a very project-centric manner. The reality is that this is costing organisations a vast amount of time and money as scripts are permanently re-generated in order to meet the current requirement.
The situation becomes worse where large volumes of external resources are involved. A lack of consistency and knowledge leaving the client organisation becomes an issue and the likelihood of finding a previously written set of scripts lowers dramatically.

Lastly, the regression pack is key to ensure the stability of the production platform and anything which compromises the regression pack undermines the efficiency of the business.

So – to summarise, ensure that test scripts are written and stored by application wherever possible. Breakdown the scripts under the application folder by version and have a generic regression pack sub-folder either at application level or version level.

Middleware Regression Testing

I have been working for a client on an interesting problem recently, which I thought I would share on the blog. The problem has been around the regression testing of an item which is intrinsic by nature, to many other projects. The item is classed as middleware, which provides a translation between two different systems, but is being relied upon increasingly, with in excess of 50 systems now interfacing. Most projects now require the middleware to be updated and the volume of change is placing the regression testing under increasing pressure as more releases occur.

The problem is amplified as the projects are working individually and the middleware changes are independent as a result. A group of projects are collated to form a release and once normal testing is completed the middleware changes are converged to form a single drop of code, which is then passed into the regression testing phase.

The first mistake has been made. The projects are not taking accountability for testing the converged code and in some instances have even been disbanded prior to this testing occurring. At this point the release has lost the testers with the understanding of the end to end function and more pressure is applied to the regression test team, to trace the scripts from the project, interpret them and generate new scripts. It is important that The testing of the converged code is now taking up large amounts of the already pressured regression testing window. This needs to be pushed back to the projects and the convergence owned by them.

By removing the testing of the converged code from the regression process, more time is now allowed and full regression can be performed, but this does not provide a future proof solution. The same window is now being fully utilised and as the volume of utilisation increases over time, the regression pack will increase and the window will become too small again.

This is an ideal opportunity to make the most of an automated solution. If we look to industry standards, automation makes a test 7 times faster to execute but 7 times longer to write. This means that the testing duration, once automated can be reduced to 1.5 days. Now we have a regression test that is only performing regression; that can be executed swiftly and provides scope for future growth.

The testing environments were another level of complexity. Whilst the project completed it’s testing with all the required components/systems/applications, the regression team, within their two week window had to acquire their own equivalent. Again this was another drag on time, but more importantly the regression was sometimes being performed on an environment that was incomplete. The result has been to make the regression middleware environment available to the projects for the convergence testing, but to utilise the projects interfacing systems. In order to future proof, the regression testing will be moved onto a pre-production environment.

There were other aspects to this problem, but those listed were particularly pertinent. The following feedback was received from two sources as a result of the work and presentation of the findings:

“I was delighted that your slides prompted so much discussion, even if it made your presentation difficult, and that by all accounts we have got the main individuals on board.”

“I was very appreciative of the clear way you presented where we are and your proposals for going forward, together with a clear capture of issues that are spoken about in lots of quarters but never pulled together into a single picture. Great.”

In summary – regression testing should encompass the entire system.
Projects should retain responsibility past the point of release to production.
Automation is ideally suited to regression testing.
Testing should always occur on as complete an environment as possible.

Regression Testing

The SIGIST “Special Interest Group In Software Testing”, have a glossary of terms located at http://www.testingstandards.co.uk/living_glossary.htm

Here we find regression testing defined as: “Retesting of a previously tested program following modification to ensure that faults have not been introduced or uncovered as a result of the changes made”

Regression Testing is a key component of any project and should be the final test performed prior to release to the production platform.

So what comprises a regression test?

The best way of describing this, is that an application which has been previously released, will be in a known state. Tests will have been previously written and executed against this and a selection of them will have been extracted to form the regression test pack. The tests would look to exercise as a minimum the core functionality of the application, in a positive manner. In this way, whenever a change is made to the application in the future, the regression pack can be run to ensure that faults have not been introduced or uncovered in functionality that was previously deemed to be working correctly. The regression pack should be updated as each new version of the application is released, building in new functionality to the suite of regression tests and maintaining it in line with the applications capabilities.

Some debate occurs as to what a regression test should comprise and it is fair to say that the larger the regression suite, the more confidence will exist in the new version of the application, but there is a balance to be achieved. The more tests, the longer they take to execute and the higher the cost to change. Pragmatism must be applied and rather than testing every permutation, coverage should ensure that the functionality of the application has not been impacted by change. Tests which apply different data to the same function, access the same function from a different point or negatively test the function (trying to break it), do not need to be part of the regression suite.

Experience shows that there is a distinct need for regression testing. Problems with Configuration Management or code control are often identified during regression, where previously fixed problems are re-introduced. Code which is common to different modules within the application is also a primary area of difficulty, as a change for the benefit of one module may adversely impact another.

It is key from a regression point of view, that no test is performed that has not been run successfully on a previous occasion. In a similar manner, no code should be regression tested which has not been previously tested.

A recent experience we had with webMethods regression testing was where, due to the nature of webMethods, several projects were being released together and the webMethods code was being converged and tested for the first time during the regression test. Because the convergence was seen as something over and above the need of the projects, it was passed onto a different team to build and a different test team to regression test. Not only was this an inaccurate use of regression testing, but the testers with the most pertinent knowledge around what was expected, were not involved.

As testers we must protect the regression testing activity. Ensure that it is full, up-to-date and complete. Failure to run regression properly increases the risk of faults in production.