When most people think about software quality metrics, they think about defect related metrics such as:
- Total defects
- Defects at delivery
- Mean time to defect
- Defect burn rate
- Mean time to failure
- Mean time between failures
While examining defects is a start to developing quality metrics, it’s just a start. When examining the total product lifecycle from sales, to product management, to development and testing, product release, and customer service, there are many areas where ‘defects’ could creep into the cycle. After the product is released and in the hands of end users, how can you measure quality? How do you know how satisfied they are? One alternative way of measuring quality from this point of view is measuring Quality in Use (QinU). Quality in use is the effect of the using the software, rather than the quality of software itself. In other words, how does it support the user to get their job done or how effective is the user when using the system to perform their task?
So, metrics from the QinU point of view must be developed in context. For instance, the user, and task for accounting software is much different than that for software used to sell automobiles and therefore, must not only be designed differently, but also must have different measures applied when measuring its quality. Until recently, measuring QinU, or quality from an end user perspective has been difficult. However, with web-based applications we can now measure productivity and efficiency by tracking user activity. Assuming that productive and efficient users are also satisfied users leads us to develop metrics for productivity and efficiency. Of course there are many other factors that influence whether or not a user is satisfied. But this gives us a start. Some metrics that come to mind include:
- Total errors/user
- Time to complete a task
- Mean time to user error
- Mean time between user errors
- User task improvement (either in time, or in reduced errors) over time
Defects in the software or lack of defects will definitely influence your customers’ perception of quality and satisfaction. However, many other factors come into play; Quality in Use is one of them.