When companies begin to implement development processes such as DevOps or Continuous Integration and Delivery, testing in production can become an important piece of the equation. Especially when it comes to cloud or web-based apps, instead of finding bugs, the job of a tester is to be the guardian and responsible for the entire development process, ensuring that defects are identified and removed before they get into the production environment.
TiP or Testing in Production is so important, although people may think it takes the place of traditional testing, but rather it enhances it with a set of test procedures that makes it more conducive to do it in the production environment. As many know, there are a class of tests that don’t make sense to execute anywhere other than the production environment.
The Risks can be overcome with a process that is air tight:
Among the risks are:
• Exposing potential vulnerabilities to the public
• Loss of data
• Poor user experience that damages the reputation of the organization
• Relying on users to report defects and vulnerabilities. Many will simply leave the app and not use it again, rather than filing a bug report with the organization
According to an article by on SDTimes by Madison Moore : “The best way to mitigate the risks is by taking the right approach and by thinking about the outcomes,” said Margo Visitacion, vice president and principal analyst at Forrester Research. “It’s not about building the features and not just did you deliver on the user end, but did you consider performance and data integrity and test processes?”
The structure that TiP provides is a conducting tests using the live site and real users – You are ensured meaningful results when you run the tests like this. TiP comes in many forms, most software testers should know them. A summary of the most important ones are below:
A/B Split Testing
A/B Testing can be the go-to tool you have in your arsenal. This type of testing is very common in web-based apps and it’s a great way to use behavioral data to enhance decision making. With this tool, you develop two or more experiences- “A” and “B”, next you expose an equal set of users to each experience. Measuring results are easier because you have two “experiences” to compare. This type of testing ensures beneficial results.
Canary birds were very good for testing the environment of coal miners. Ensuring the miners were safe from poisonous gases. But, in the process of testing new code before it goes into high production, deploying a small subset of production machine before releasing it widely, it’s like having the canary in the mine.. if the code does well and doesn’t die the likelihood the full scale release will also go well.
This is a tried and true method of ensuring that the production code is ready for prime-time. This may be an unsettling process for some, create a problem to see how the production environment handles the challenge. “Injecting a fault” into the system with the understanding that you are protected from specific errors.
This type of intentional error inducing process can drastically improve the reliability of the application, and the added benefit of keeping the developers and testing teams on their toes.
Synthetic User Testing
Creating and monitoring fake users who interact with the real site. These users are tasked with executing various functions and transactions within the web app. This team would have knowledge of the potential weaknesses of the site, and try to exploit them, they would also put the site through what would be called “normal usage” tests.
With the end user in mind, and what type of user experience they are having.
Wanting to know that your application and organization can recover from a challenge. Putting your application through the necessary tests and gauging how well your system recovers from the errors such as things that are normal in the application development process like backups and dealing with secondary sites.
These type of problems would likely to showcase how the system and support personnel handle problems that are likely to happen. This could go a long way to shoring up the confidence in all the teams involved in the SDLC.
Why Should I Add TiP to my Testing Strategy?
To have a testing in production strategy in your QA arsenal can take your QA testing to a new level. Yes, there are inherent risks of testing with live users, but the risks of NOT testing with live users is bigger, and your results when not being live, are never comparable to the live user data. The staging environment and the real environment are never the same thing.
The SD Times Article continues; “Testing has long belonged to testers and QA. With agile, that has changed: Developers are now often asked to perform more of the testing, and testing can occur on a weekly, daily or hourly basis. In an agile environment, companies could test on every change on code, so every time the software evolves, every change of code is tested, according to Alon Girmonsky, CEO of performance management company BlazeMeter.