Just mention software testing metrics and there is ample discussion on Defect Removal Efficiency (DRE), so let’s examine it closer.
Back in the days of waterfall, if we found 100 defects during the testing phase (which we fixed pre-production) and then later, say within 90 days after software release (in production), found five defects, then the DRE would be what we were able to remove (fix) versus what was left and found by users:
100/(100+5) = 95.2%
This 95% has been referred to by Capers Jones, a well-known software measurement colleague, as being a good number to shoot for.
However, many organizations are now either using agile or migrating towards agile, whereas much of Capers’ research was done using larger/traditional organizations that are still using waterfall methods. In an agile environment, the numbers get a little tainted for several reasons:
- With agile, we are releasing so often that we don’t have a full 90 days to collect data (defects found). The main point here is to be consistent, so this needs to be much shorter, perhaps two weeks to a month.
- With release cycles so fast, we may decide not to fix defects that we find, both pre- and post-production. This can be either a conscious decision where we let lower priority defects enter production, or even higher severity defects, given our knowledge of risk and chance of occurrence.
Regarding point #1 above, we simply have to be consistent and recognize that this time span is going to be shorter and therefore perhaps less statistically reliable due to less data and less time for users to find the defects related to that release. They may find a defect five weeks later from that release N, but we are already in release N + 3 by that point.
Regarding point #2 above, in agile we may consciously let defects into production either because we make an assessment that the risk is worth it (combination of severity and probability of the defect being found) or because we simply don’t have the bandwidth to fix it and prioritize feature release over defect fixing. In this case then, the defects removed (DRE) could be low, yet the defects found much higher. This is where you need another metric, Defect Detection Efficiency (DDE). In other words, DRE is low because you chose not to fix defects, yet your Defect Detection Efficiency could be high.
On the other hand, what if DRE is high and DDE is low? In this case, we have a high rate of defects that are fixed, yet we are not good at finding them? What would that look like?
As you can see, simply going for 95% defect removal efficiency is not as straightforward as it sounds and you can’t just rely on the numbers with so many other factors to consider.