Last time, I blogged on some general flows of defects and ‘buckets’ of measurements that I thought were possible. Then I thought that I needed to take a step back and think at a higher level about software testing measurement in general. Software testing measurements can help us in a few areas including:
Planning: Cost estimation, training and resource planning, budgeting and scheduling
Controlling: Track testing activities for progress and compliance to the plan
Improving: Benchmarking processes and identifying where efforts should be focused and then measure the effects.
There are many other areas that measurement can help us, but the main point is to think about WHY we are collecting these measurements and how to use them, i.e. improve, control, compare, understand.
Just some basic definitions with some examples in a testing context:
Measurement: An direct observation usually quantified by a number, such as number of defects found. Keep in mind that number of defects found means almost nothing by itself unless you put it in context with other information.
Metric: A calculable quantity derived from a measurement or other metrics (indirect and direct). We could take the same measurement, number of defects found (in the testing prior to production), and divide by defects found in operations to arrive at a metric: defects found pre/post production. Let’s call it DFPP for short. Still this has some meaning but limited.
Indicator: A representation or interpretation of the metric according to a defined scale. When we apply an indicator, then the metric begins to have meaning. If we take the same example DFPP, it begins to have real meaning if we compare it over time, or from release to release. What this does is give us context. Then we can apply an indicator such as (change of -5% and less = good, no change to 2% increase = average, and 2.1% – 5% increase = poor, and >5% increase = failing). If we do that, then we can really interpret the metrics in our own business environment to have real meaning and thus help us improve.
Software Test Measurement Usage
Software test measurement can be used in many different ways for different types of measurements including:
Progress: How well are we performing compared to schedule, as well as predict future time and corresponding effort required to finish?
Effort: How much effort (hours or expertise needed) are we putting in?
Cost: How much money or other resources are we using versus estimated? as well as how to predict future costs?
Customer Satisfaction: How are our customers responding to our quality?
My next blog will be about these different types of measurements in the software QA environment and how to use them.