As the agile development philosophy and DevOps software become more sophisticated, the KPIs that are used to measure success must follow suit. Forrester’s study The Definitive Software Quality Metrics for Agile+DevOps“ found that contextual metrics far surpass so called “counting metrics” when measuring the success of a program.
The Move to Contextualize KPIs
Counting KPIs is certainly better than leaving metrics to chance. However, setting those KPIs in place with each other gives another level of insight into DevOps. It also measures success far more precisely, as the Forrester report found.
Of the 603 companies that Forrester surveyed, those focused on the software’s holistic utility to provide a positive user experience were more successful (by far). These companies were also much better at measuring success and better able to create end to end automated processes.
Companies that did not look to contextualize their KPIs all seemed to have bottlenecks in their processes. Many of these businesses could not identify what those bottlenecks were; however, they all shared the same characteristic when surveyed for DevOps. Many respondents to the study did place blame on manual testing.
Best Practices for Agile & DevOps
There are other practices that are employed to a high degree in the companies that place their KPIs in context. These best practices include the following –
- An appropriate amount of money set aside in the budget focused on testing and upgrading skills
- A philosophy of continuous improvement for testing to stay alongside new release frequency demands
- Including testers in delivery teams that are fully integrated
- Using this testing mechanism as early in development as possible
The Risks of Automation
Although there are many advantages to automating the testing process, there are also some risks.
Forrester and other reports agree that the development of an automated software life cycle is vital to speed up the speed of releases. The report speaks to the current inability to accurately measure quality throughout the whole cycle, however. Full automation in this type of delivery pipeline actually increases the risk of errors during the process. Under these circumstances, companies must develop automation slowly, along with testing practices.
Many companies are actually delivering products that are not within an acceptable business risk. This metric is always becoming more narrow as competition becomes greater. Only the companies with the right kind of agile and DevOps software quality measures are able to keep up with the moving goalpost. Forrester found that many companies are vastly overestimating their capacity to assess risk.
Correctly Defining Software Quality
There is a gap between the definition of quality as related to business and quality as defined by technical experts. The mismatch is the cause of the huge overestimation that many companies have in assessing their own risk profile.
If this gap occurs on the aspects of business that are less important, you may get away with it. However, critical functions with this problem may cause huge bottlenecks or complete failure of a department, as in the case of security authorizations. In most cases, a company will not find out about the gap until it is too late.
The scenario above can be avoided by understanding when a process is sophisticated enough to be automated. This is exactly why counting metrics are becoming less and less useful – the question has completely changed when it comes to modern DevOps. Are your tests accurately assessing the appropriate risk factors – the ones that are critical to your business? Counting does not matter. What is important is the decisions that you can make based on the tests that you run.