Pairwise Testing Isn't Easy, That's Why We Have Tools

Pairwise Testing Isn’t Easy, That’s Why We Have Tools (from Wikipedia)

After my talk on Implementing Pairwise Testing at the Atlanta Quality Assurance Association last week, I had a chance to sit down and think about the discussion. From what I gathered about the audience:

  • Great interest in tools. Everyone wanted to know how can I do it! Of course we all do, and I hope that I gave them plenty of references and sources for this.
  • Limited knowledge of a much larger field called combinatorial testing (of which pairwise testing is just a subset).
  • Lack of awareness that combinatorial testing, and pairwise testing, is a difficult math problem! See Wikipedia showing the formula.
  • Limited knowledge on where and when to use pairwise testing which also coincides with the weaknesses and strengths of pairwise testing.

I closed the discussion by talking about combinatorial testing, and in particular, T-Way testing. I think the audience got a good idea of what T-Way testing is and what tools they can use for it, but I don’t think I did a good job (I kind of left it as an open question for them to ponder on) explaining how to combine pairwise and T-way together.

First of all, T can be replaced with 3, 4, 5, etc… depending on how many combinations you want and how much confidence you want to have that you’ve developed sufficient test cases to find some (i.e. 75%) or all (99.9%) of the defects that exist. The higher you go with T, the more work there is in test cases (combinations of T parameters), but also more confidence that you’ve found all the defects. Obviously we’d like to go with T equals a big a number as possible but that is a lot of work. The way I’d recommend using pairwise together with T-way is:

Firstly use pairwise, that’s the minimum to get to a 70+ % confidence that your test cases will find most defects.

As pundits of pairwise say, it won’t catch all. So for mission critical software we need to increase T. Increase it for those areas of your software that you consider critical. For instance, if you think the function of generating an income statement is more critical than others, increase T in that part of your software to enumerate more test cases in that area. Then you can have a higher confidence level that this particular function has been more thoroughly tested.

You can use pairwise more creatively as well. Go through your defects and examine what functions and variable deserve more attention, parameterize those test cases and variables, and increase T in those areas to enumerate more test cases there. Perhaps you already have them, or just maybe some of the software tools I showed will come up with test cases that can help you take the exploratory out of exploratory and make your testing more thorough and consistent. I think we all want that!

I’ll stop here. If you have a particular question about pairwise or combinatorial testing, let me know: philip.lew@xbosoft.com.