Regression Testing – Developing a Plan
To create an overall system-level regression test plan, begin by defining the entry and exit criteria and time boundary for each type of regression test. The time boundary is important because one type of testing may be dependent on another, and of course, with no time boundary, testing time can naturally expand.
- Entry criteria may include that all software changes have passed unit testing, a green build has passed smoke and sanity testing, and the regression test suite is prepared.
- Exit criteria may include that all priority test cases have been executed, all defects have been reported, all defects have either been resolved or deferred for future resolution, and the product owner or customer has signed off.
Identify the test cases or test types that will be part of the regression test suite. Keep in mind that you may have different regression test suites for different purposes. Include test cases for recently-changed code and for defects that were recently found and fixed. Exclude test cases for known defects that have not yet been resolved. Finally, automate test cases that are reusable for coverage of core features.
In the plan, be sure to allow time for exploratory testing as exploratory testing tends to find more issues than existing test cases and test automation. Also, allow time to conduct post-testing reporting and analysis. Otherwise, you’ll never be able to improve. Maintenance is also a critical activity as some tests become invalid while others may shift in priority or become broken-blocked and need updating. You should have a version control system for the regression tests themselves, whether they are automated or manual test cases.
Regression Testing – Reporting and Analysis
What information do the stakeholders need to make “quality” assessments and what information do you have readily available? The reporting should include whatever metrics are deemed necessary, no more, and no less. The last thing you want to do is produce reports that no one reads. Additionally, as your regression testing process changes and adapts over time, you may change your focus and therefore, use different metrics to support your focus. Some examples of questions that proper reporting can help answer:
- Where to test more or less
- What tests should be executed or not
- What tests are effective
- Should more tests be added
- Where is QA spending the most time
- What features or functions need refactoring
- Overall product quality assessment
A key value in having a process in place for evaluating “product quality” is that it can provide warning signs when a change in regression testing techniques may be warranted.