Using Pairwise Testing - Versus NWise Testing

Different effects when choosing Pairwise (n=2) versus other combinations

I’ve been invited to give a talk on Pairwise Testing Test Case Design in Atlanta on August 11 for the Atlanta Quality Assurance Association. My preparation though involves much more than just explaining pairwise testing.  First I had to think of the Why’s. When using pairwise testing as a combinatorial method for test case selection, you need to think about much more than choosing what tool to use or how it works, but rather when and where.

First, it’s important to note that pairwise testing is a term for testing the combination of two variables and testing the inputs of unique pairs rather than every variable where you could have up to ‘n’ inputs. The reason we go after n = 2 versus n = 3, 4, 5, etc… is that there is more bang for the buck when you go from n=1 to 2 versus n=3, and n=4. The work done by Kuhn and Reilly shows this in the figure above. Other than this study, I have not seen any other empirical work to show WHY the software testing community seems to prefer choosing n=2.  As you can see, by choosing n=2, you can catch up to 76% of the defects. On the other hand, if you go for n=3, you get another 20% gain in effectiveness. I’d guess that most of you want to get this extra 20%?

The problem is that until recently few tools could generate combinations with n>2, and most that did required long computation times because the generation process is mathematically complex. Hence, pairwise testing has become the most common approach to combinatorial testing because it is reasonably effective. However, with computers that now fit in our hands that are more powerful than a Cray computer from a decade ago, should we not change our thinking?

So in preparation for my talk in Using Pairwise Testing  I started thinking, how about 3-wise testing? Why be satisfied with 70%?