Clients often ask why a test automation framework is so crucial. We often draw the analogy that when you build a house, you need to first do the framing. But not just any framework will do – in order for your test automation to be effective for you the framework itself needs to be designed for you. You don’t want a single story house framework when you actually need two stories! Any properly designed framework should be able to easily handle:
- Results Reporting – Think from the stakeholder’s point of view. What do you think they want to examine? How often? What levels of aggregation? Different stakeholders will be interested in different types of information and levels of aggregation both in time and entity. This includes data on the software under test, and information about the scripts’ success and failure rate (and why they are failing).
- Debugging – Your framework should debug easily and be able to determine if a failed script is due to the script or due to the software. It should also be able to determine where the script failed and when executing which part of the software under test. Report in a clear manner so the user or stakeholder knows what’s going on.
- Scalability – What purpose and what software under test is your framework for? Can it scale to 100 test scripts? 1,000? How about 10,000? Are the scripts dependent upon each other? What if you have to expand to multiple software under test?
- Maintenance – How easy or difficult is the framework to maintain? Think about whether other scripts are impacted if you have an error in a script and need to make a change. Do you reuse code effectively so that you can make a change in one place and have it work throughout your entire set of scripts, or do you have to make the same change in many places?
- Expertise – Do you need a coding expert to develop scripts? How about maintaining or updating them? What if you need to edit, add or reorganize scripts in different orders or grouping? What if you bring in new personnel? Will they be able to jump in and write scripts immediately?
- Flexibility in Execution – Does the framework execute smoke/regression/customized test suites? Can test scripts be picked out and grouped as desired? Is it easy to filter and sort scripts according to functions that you want to test or to different types of regression? How will those running the framework be able to easily pick out the scripts they want to run?
These are just a few of the basic test automation framework requirements that we consider when building a framework. Some of our clients won’t care much about a high level of expertise because they have the skills in-house, while others might consider maintenance a bigger issue because they may have the expertise, but only one resource. Of course, the primary requirement is reporting, and without reporting, it’s hard to show the true value of test automation at all.