When you think of test automation coverage, what do you mean? Many prospective clients we talk to want to have 80% of tests automated or even more. When they throw out a number like that (to me, what seems arbitrary), we always ask where they got that target? More often than not, they have reasons from a business perspective, meaning they want to get stuff done faster, but no quantitative background behind these percentage goals. But if they were to achieve that magic number, it would just make them feel good inside that they have that level of coverage that helps them sleep at night. I get that. Given that, my team and I had a discussion on automation test coverage and how to determine the optimal coverage. Not surprisingly, we all had different ideas on automation test coverage is. I’ve listed out a few of them:
  • Functional coverage: Has each function in the program been automated?
  • Statement coverage: Has each line of the source code been executed?
  • Condition coverage: Has each evaluation point (such as a true/false decision, or options 1-10, etc.) been executed?
  • Path coverage: Has every possible route through a given part of the code been executed?
  • Entry/exit coverage: Has every possible call and return of the function been executed?
  • User story coverage: Has every user story and its negative aspects been executed?
  • Menu coverage: Has every possible menu option and combination thereof, been executed?
  • API coverage: Has every possible API call and parameters for each call been executed?
  • Data coverage: Has every possible data type been edited, deleted and added?
As you can see, some of these coverage types are white box oriented, while others may pertain to what you originally thought of as test automation coverage. Most think of white box testing for code coverage but for black box, UI, or API type automation, what is code coverage? From the list above, you can see that we could have all the user stories coverage in a set of automation test suites. But does that mean we have 100% test automation coverage? If we use a UI type automation tool such as Selenium and access all the menus and input data into each field, does that mean we have 100% coverage? I don’t think so. The point is that before we set an arbitrary goal of percent coverage, we need to really think about what type of automation test coverage. If we say 100 percent coverage, and use a tool to exercise every line of source code, I’m not sure that this will even come close to helping us feel comfortable that we are delivering a quality product.