Software Test Metrics – Do’s and Don’ts Webinar Questions and Answers

In the recent webinar with QAI on software test metrics, the audience had several questions that we were not able to answer during the webinar. Here are the first few questions and related answers.

Any suggestions for measuring resource allocation?

By resource allocation, I assume you are meaning labor. I’m not clear about allocating where, but most clients always ask about the ratio of developers and testers thinking that there is a hard rule. The literature will tell you about 3 or 2 developers to 1 tester, but that is for testing, and not QA. QA usually runs at a much smaller ratio around 5%.  All of this depends on many contextual factors like development methodology, skills of the developers and so on. If you have poor development skills, then obviously you need more testing.

Can you also please share some thoughts on gathering soft information such as lost business opportunity, etc.

These kinds of soft information are best collected in correlation, and remember that is it not necessarily cause and effect. For instance, you can just collect sales as one of the metrics downstream. And collect other metrics upstream such as defects. Do some multi-variate correlation analysis and see if there is a relationship. We like to recommend collecting numbers on the front line, such as those in technical support. For instance, the length and number of tech support calls.

Do you have any recommendations for leading indicators of quality during software development. Most of the ones on the slides appeared to be lag indicators.

You are right. We only had 1 hour, but there are many upstream metrics that you can use to do predictive analysis as well. Just look into your process. For instance, your user stories. You can examine them and see how complete and accurate they are. By inspection, you can develop metrics for those and quantify ‘defects’ in user stories. Defects in user stories will then influence defects later in the production process. Just as if you had a defect in a mold for a plastic factory.