Regression Testing – When to Automate Check List

Automated regression testing has long been an integral part of QA. In many cases, automation has a proven track record in reducing the costs of testing, increasing test coverage and effectiveness, and shortening testing cycles. There are also various intangible benefits associated with automation such as reaching the market faster and improving tester morale (and productivity) through the elimination of tedious tasks.

Automation, however, is not a cure-all for testing problems and can not eliminate the need for manual testing. For most applications, there is an optimum balance between automated testing and manual testing. The dilemma faced by small development shops with a QA & Testing staff of one or two (or none) is how to determine this balance. Deciding on the types of automation, the areas that can potentially be automated, and the tools and skill levels of the testing resources that will be required is not a simple black and white proposition.

So, what is an organization already buried in hundreds or thousands of test cases to do? At XBOSoft, we have developed a simple high-level “Assessment for When to Automate.” Its purpose is to provide a basis and framework for addressing the question “Should we automate some of our testing?” It doesn’t directly answer the question. The intention is to give people a better idea of deciding if spending more time and effort on the “automation question” is warranted.

The Automation Assessment is a simple 3-step process:

Step 1 – Pre-Assessment

These are simple “yes or no” questions. A “yes” answer to any one of them indicates moving to the next step is warranted. It should be noted that for some of the questions below, test automation is no silver bullet in and of itself. Automation, however, may be part of the solution.

1. Is your testing mix more than 90% manual?
2. Has the testing activity become deskilled?
3. Is there frustration by the length of time it takes to finish projects?
4. Is there a need to hire additional testing resources, but no budget?
5. Is manual testing activity tedious and repetitive?

Step 2 – Automation Assessment Questionnaire

The Questionnaire is attached below. The questions selected reflect many “rules of thumb” in the industry. In general, the more “yes” answers, the more potential benefits there are to be realized in automation. In fact, six or more “yes” answers are a good indication that there is latent “automation value” and pursuing automation would be beneficial to the organization.

Step 3 – Automation Cost/Benefit Review

If the Automation Assessment indicates there is potential value in automation, then the next question is “are the benefits in automation worth the effort”?

XBO’s recommended automation cost/benefit review is not a rigorous spread-sheet-driven return on investment (ROI) analysis. Rather, before diving into the nitty-gritty details of developing financial models and business cases, it’s important to understand at a high level the four areas below. By understanding these elements, you’ll have a clearer idea of the context driving your more detailed analysis for your organization.

Current Testing – the idea here is to establish a baseline of current testing costs. Our recommended approach has two parts: Baseline = actual costs + opportunity costs. “Actual costs” would be the salaries, expenses, and overhead of the current testing. This should be a good solid number. What most people often overlook, however, is “opportunity cost”! Estimating opportunity costs is a little tricky and entails attaching some value to things such as better product quality and faster testing cycles. What does it cost you to have defects found in production? What are the risks of users that are dissatisfied? How do you quantify the financial loss of a user who walks away to a competitor? What benefits could you derive if you were to reduce your end-of-cycle regression by 25%?

Future Estimated Testing Demand – the objective is to try and determine a “delta” in testing between the current year’s (should be known) and the next 2-4 years. This would be based on information in company business plans, product development plans, etc… Is the company growing such that new QA & Testing specialists will have to be hired? If there are big requirements for hiring, then perhaps a mix of these should be test automation specialists and incorporated into your costing estimates. Also, to be considered are the platforms covered and whether or not you’ll need to invest in mobile devices or mobile device farm subscriptions.

Automation Benefits – this involves estimating the “direct” benefits that can be realized through automation. First, estimate the percentage of test cases that can be considered for automation. For practical purposes, this should generally be between 10% – 50% of the testing program. Second, adjust the future estimated testing demand by the percentage of test cases that can be automated. This should give a ball-park estimate of the direct automation benefits in terms of time you should be able to conservatively save by automating rather than doing manually. However, take into account that automation will also entail covering additional platforms and increasing coverage, so the actual time saved may be ‘used up’ or allocated towards better quality software.

It should be noted that automation benefits will also come through intangible factors such as enhanced product quality, faster execution of the tests in multiple environments, and more time to focus exploratory and manual testing on new features to name a few.

Costs to Automate – The key components in estimating the costs to automate are:

1. Test automation tool selection and cost.
2. Time, and expertise level required for:
   • Customizing the automation tool
   • Building reusable modular components
   • Designing automation frameworks
   • Developing automation scripts
   • Maintaining scripts
   • Developing a reporting framework
   • Analyzing script execution results
   • Integrating automation into the CI/CD pipeline
3. Hourly cost per test automation resource

Provided the testing team has the experience to scope out the automation tool(s) to be purchased, assigning a cost is straightforward. The challenge is estimating the resources needed in #2 above.

XBO Check List for Test Automation