Testing in Live – Why you should not!

As a tester I am often asked why we can’t test in live. There are so many positives that can be gained from it that it surely makes sense. After all, the environment is exactly the same as live, the data is real and the volume of users and traffic is real.

Production is designed to serve a business in operation. It is being used in anger every minute of the day. It may be internally used, it may be used by high street shops or it may be open to public use such as the internet.

The first problem is that the code that needs to be tested is not fit for live because its quality is not known. So by placing the code live you are compromising the production platform. You can cause code which is running perfectly in production to fail, as common routines are modified.

Testing is designed to break things. We need to understand the points of vulnerability and testing tries to identify these. This means that the application could be caused to crash, which could result in a domino effect and take out other elements of infrastructure or code.

In testing terms, the point at which resolving a defect is highest, is in production. One of the primary reasons for this is that outages which affect the production platform can be disastrous. Imagine 1000 customer services personnel being unable to work for an hour. A hourly pay rate of £10.00 per hour and the problem has already cost £10,000.00 But what if you have an on-line presence and you lose that for an hour. You have now lost sales revenue for that period of time and this you cannot recover. Perhaps more damaging is that perception of your organisation has taken some public damage. The potential new customer went somewhere else and you have now lost their trade for the foreseeable future. The existing client who was returning has now been forced elsewhere.

Think also of the press that can be received through public outages. Organisations that experience loss of service will often find this being reported in the media and this causes immeasurable brand damage.

Another consideration is that the data which exists on your production system is real. The testing therefore cannot modify, add or delete information without careful consideration of the consequences. Corruption of data exposes an organisation to the data protection act, but worse, may invalidate information which is crucial to your business. What happens if a change of address occurs which is a test – the client is potentially lost along with their business?

A final point is that your system is in all likelihood linked to other organisations. When I first became involved in development, I had a story relayed where a developer had unit tested his code in a live environment. His code was a credit checking piece of software and he used his own details each time he ran the test. As your credit score is negatively impacted each time a check is performed, the developer managed to take his credit rating from positive to negative in a matter of a day. He did not realise he had done this until such time has he tried to make a credit purchase and was refused. Fortunately, after some discussion with the credit reference agency, his rating was restored. But any kind of financial transaction must be performed in a test environment, linking to other test systems. Imagine the damage that re-running a BACS transaction could cause, or payroll being sent out at the wrong time.

Production is very much live. You would not expect a doctor to test a new drug on a human without going through a certain degree of process first. Even then you would move into a clinical trial under strict process.

What about Beta testing I hear you ask. Yes, there are examples when the software is deemed to have no negative impact on production, where sufficient testing has already been performed as to be confident of its capabilities, that the software may be released with a “Health Warning” attached. It may be that a select group of users will be asked to perform a trial. But in these instances, the application will have been put through its’ paces in a test environment.

It is important that users of systems have good experiences. Failure to achieve this results in loss of the users from the system, negative impact to the reputation of the department and individuals making the release, plus the potential to damage the organisation's brand and revenue-earning potential should make this a course of action that is avoided.