As I prepare for my talk in Kuala Lumpur at Softec 2014, I started thinking about our own projects at XBOSoft and the software defect metrics that we use internally to see how we are doing for our clients. There are the normal ones such as defect escape or leakage rates, and defect fix time, technical debt reduction – refactoring. But from a ‘pure qa’ point of view and in particular XBOSoft, we want to reduce the work for our clients while improving the quality of their software. Some of the key metrics we track include:
- Defect reject rate: This applies not only to ourselves but all test organizations, and all dev organizations (agile too). The last thing you want is to waste someone’s time examining a defect that doesn’t exist. The main issue here is to track why defects are not defects or why they get rejected. Of course, we want this number to continue to go downward over time. There could be a number of reasons for rejected defects including:
- Not in the requirements
- Not understood by the developer
- Cannot reproduce
The main point here is that those fixing the defects must document the fix/or do not fix, status and cause, so that learning can occur going forward.
- Defects found by clients: Of course we want to find all the defects rather than our clients, but sometimes this is not possible as they may have access to the software or certain functions before we do and therefore file a defect before we even get a chance. Naturally we want this number or ratio to be as low as possible and to continually decrease over time, but how to make it decrease over time?
- As we learn more about the application, we should gather expertise and be able to spot defects faster just because we know the app well.
- As we work closer with our clients, we should be able to get access to features and functions even prior to official ‘testing’.
- With deeper and deeper domain expertise, we should find those defects that are very deep such that only our clients’ customers can find these defects.
Software defect metrics can be lots of fun if you use them to improve rather than point fingers. No one likes to get on the scale and be yelled at, even if they did eat chocolate cake the night before.