Have you implemented a software quality metrics program? If so, then you know its not easy and that metrics programs often fail. Let’s take a look at some common reasons that metrics fail:

  • Metrics don’t measure the right thing – Often times, we get an idea for a metric from a person, company or article and begin using it without thinking ‘What am I trying to measure and why?” In the end, we sometimes get measurements that don’t matter relative to our goal.
  • Metric becomes the goal – It is possible for organizations to be so concentrated on the metric, that they don’t understand the metric’s relationship to the goal or objective. If you are measuring software quality, certainly defect counts matter. But there is a lot more to it, and defect counts need to be incorporated into an overall valuation.
  • Metrics don’t have consistent context so they are unreliable – Just as in weighing yourself, it doesn’t make sense to drink 2 gallons one day and weigh in, and go jogging 10 miles the next day and weigh in. Context needs to be defined and then maintained for measurements to be meaningful. This can be difficult in today’s environment with changing test platforms and other contextual factors.
  • Measurements used are not consistent – Just as context needs to be consistent, so do the measurements and the methods that you collect the measurements and calculate the metrics.
  • Metrics don’t answer the questions you had to begin with – You run off collecting measurements and calculating metrics without thinking what answers will I get after getting this information?
  • Metrics have no audience – As a corollary to the previous factor, if there is no question to be answered, then their will also be no audience for the metric. Metrics need to have an audience in order to have meaning. How many of the metrics and reports that you generate are read?
  • Measurements are too hard to get – If you end up designing the right metric to answer the right question, it doesn’t matter if it takes several man days to get the data and do the calculations. Unless the value and decisions made from these metrics have considerable value, they’ll soon be abandoned.
  • Metrics have no indicators so cannot evaluate – You collect mounds of data but then what? How do you determine what is ‘good’ or ‘bad’? Before you get started collecting and calculating you need to put together a way to evaluate the numbers you get with meaningful indicators that can be used as benchmarks as your metrics program matures.

So we’ve discussed a few of the reasons why software quality metrics programs fail. What are the elements of a successful metrics program and what metrics can guide you down the road to improving your software quality? If you are looking for these answers, stay tuned!