Look - my new test bed. What a brilliant image! How many of us go here for our next test environment.
Do people realise the importance of the environment to the tester? I don’t think they can. I have spent most of my testing life working with make-do arrangements and being expected to test to high standards. I had a conversation with a fellow tester today, who is currently working as an Environment Manager and he and I were both moaning about the problem.
So what happens? Firstly all of the investment on a project is in the production equipment. As the project is creating application X for the first time, the to-be production environment can be used for testing purposes. It is ideal in this situation because it means that we can test on the most suitable infrastructure possible – what it will run on in live.
The problem comes the next time the application is updated. There was no investment in test infrastructure as part of the original project – they didn’t need it. So now the update project has to justify a massive expenditure in test equipment, which could potentially jeopardise the projects financial benefit. This step is therefore avoided and the hunt begins. We need to find a piece of kit that can be used to test the application. Something which is similar to that in production would do. Alright something which resembles the production kit, if you squint at it whilst wearing dark glasses. The good news comes through. Kit has been found and once the rust is scraped off, it is found to be an earlier version of the kit in production. The RAM is reduced, the processor is 2 years earlier than that which is in production and the operating system has fallen out of support. Never mind, it is all we have. Then the test manager explains that he needs three environments to be able to complete the testing on schedule. Don’t worry we can whack in some more RAM and increase the disk space. It’s still not half as good as production but we can split it into three for you.
This is seen as a pragmatic solution by all involved and the test manager must now do his best. (Agggghhhhh!!!!!!!). Let me explain my frustration. As soon as you compromise the test environment, you compromise the testing. So many times defects are raised which are directly attributable to the environment and if that environment is not the same as production, time is going to be spent resolving defects which will not exist in production and potentially defects which will occur due to the production environment will not be located during testing. Certainly in terms of release testing, the process is likely to be different and effectively the results during testing are false and cannot be relied upon. The lack of adequate test environments becomes a project risk in its’ own right and should be logged in the project risk register accordingly.
It is critical that the testing environment is as close to production as possible and that any deviations are fully understood. Only with the differences between production and testing environments being known and measurable can you rely on the testing results.
Take performance testing. If you are running with less RAM, less disk space, less process power, how will this manifest itself in the results? As for security testing, the operating system version becomes paramount as the level of inherent security is likely to change. Know and document the differences and if this cannot be achieved be careful about signing off the product as fit for release.
As a recommendation to all project managers, ensure that you leave an inheritance of test infrastructure for the following projects. Placing a new piece of equipment live is not sufficient. Consideration must be given to ongoing support, testing, development and training and what environments are going to be used for these.
So what happens? Firstly all of the investment on a project is in the production equipment. As the project is creating application X for the first time, the to-be production environment can be used for testing purposes. It is ideal in this situation because it means that we can test on the most suitable infrastructure possible – what it will run on in live.
The problem comes the next time the application is updated. There was no investment in test infrastructure as part of the original project – they didn’t need it. So now the update project has to justify a massive expenditure in test equipment, which could potentially jeopardise the projects financial benefit. This step is therefore avoided and the hunt begins. We need to find a piece of kit that can be used to test the application. Something which is similar to that in production would do. Alright something which resembles the production kit, if you squint at it whilst wearing dark glasses. The good news comes through. Kit has been found and once the rust is scraped off, it is found to be an earlier version of the kit in production. The RAM is reduced, the processor is 2 years earlier than that which is in production and the operating system has fallen out of support. Never mind, it is all we have. Then the test manager explains that he needs three environments to be able to complete the testing on schedule. Don’t worry we can whack in some more RAM and increase the disk space. It’s still not half as good as production but we can split it into three for you.
This is seen as a pragmatic solution by all involved and the test manager must now do his best. (Agggghhhhh!!!!!!!). Let me explain my frustration. As soon as you compromise the test environment, you compromise the testing. So many times defects are raised which are directly attributable to the environment and if that environment is not the same as production, time is going to be spent resolving defects which will not exist in production and potentially defects which will occur due to the production environment will not be located during testing. Certainly in terms of release testing, the process is likely to be different and effectively the results during testing are false and cannot be relied upon. The lack of adequate test environments becomes a project risk in its’ own right and should be logged in the project risk register accordingly.
It is critical that the testing environment is as close to production as possible and that any deviations are fully understood. Only with the differences between production and testing environments being known and measurable can you rely on the testing results.
Take performance testing. If you are running with less RAM, less disk space, less process power, how will this manifest itself in the results? As for security testing, the operating system version becomes paramount as the level of inherent security is likely to change. Know and document the differences and if this cannot be achieved be careful about signing off the product as fit for release.
As a recommendation to all project managers, ensure that you leave an inheritance of test infrastructure for the following projects. Placing a new piece of equipment live is not sufficient. Consideration must be given to ongoing support, testing, development and training and what environments are going to be used for these.
No comments:
Post a Comment
Hi, we would like to hear your feedback.