Deciding between manual testing vs automation testing can seem like a cornelian dilemma. At least, that’s how it transpires when dealing with many of our clients. So we thought we’d help you make the decision between manual testing vs automation testing, and when and where they should be applied.
Manual testing vs automation testing – The dilemma
The crux of the problem in deciding between manual testing vs automation testing is that most higher-level managers want to automate 100 percent and fixate on % automation, Right? Who hasn’t run into this paradox? Firstly, we know that everything can’t and should not be automated. During this session, we will discuss the challenges of creating well-formed test cases and criteria for identifying suitable candidates for automation scripts.
Remember this is part of our 7 episodes on Agile in QA
By the way, this is the third episode of our 7-part series on Agile from a software quality point of view. Make sure you check-out all the episodes in their sequential order to build a powerful framework in your business.
First, let’s establish a common understanding of testing terms
A set of instructions on how to test an application, with expected results, followed by a team member who reports the results in a prescribed manner. The team will use a specified process to report the test results and associated defects. Teams commonly use a specified tool or a spreadsheet to track the test cases (assets) and their execution status.
A scripted set of instructions to exercise an application. The term automation testing can be applied simply to an individual test script or complexly to being a block of test cases that are executed by a continuous deployment and making the decision of whether or not the build can be pushed to the next level.
This is generally freeform and a good way to involve non-traditional testers in the test process. In practice, it is best to time-box this effort and provide a free-form, central place to report observations and bugs (defects / identified deficiencies/incidents). In an Agile team, this can be paired with the retrospective ceremony. Exploratory testing is a form of Manual testing.
Targets segments of the software. It generally exercises the code to validate the expected positive and error conditions within the code. Unit testing can be either manual or automated, depending on the team.
What are the elements of a well-formed test case?
We can think about a test case creation somewhat in the same way we write an agile story using the If, When, Then format.
Tests code/functionality for values and notes when those values come back as negative or false. For example, the test may assert that the value of an expression should be 45, and if we get anything other than 45, the result returns false.
Each test case should have at least one verification point. The test case can have more than one verification point. Using the agile mindset, the goal is to have the test case complex enough to cover self-contained but simple enough so the results from the test are not dependent upon too many verification points, causing a high failure rate.
Reporting test findings and test status. Generally, a test case’s status is deemed as not started, in progress, passed (no defects), failed (defect found), or blocked (test cannot be executed due to a dependency). When a test passes, generally, there is not a lot of documentation due to the expected and actual test results being in agreement. When a test case is blocked or fails, notes need to be added to document how the actual results vary from the expected results.
Also known as incidents (defects in production) or bugs, when product does not perform as expected.
*Note: On an Agile team, testing tasks are built into stories, and defects may be identified and resolved outside of a formal defect tracking process as part of the story completion process.
Going into the next part of our discussion, we need to pause to have a slightly theoretical conversation. An easy way to start this discussion is by asking what would cause a test case to be not fit for automation:
Single use test case
Edge case test case
A test case which has many dependencies
Automation test case development is generally reserved for repeatable, well understood tests which will be executed multiple times.
When the point of the test case is to test the validity of data, the test case may be a framework, which iterates through the data row by row. In this case the test designer decides if the test results will be reported as a test case per row or as the complete data set.
How tests are grouped, common terms, and test execution strategies
A test (manual or automated) with an individual objective and at least one verification point
An automated test with an individual objective and at least one verification point.
A group of tests, generally executed together, targeting specific functionality and reported on as a test set.
Usually manual or exploratory testing. Manual functional test cases generally contain the following parts:
Objective – why the test case is important. This can also be referred to as an assertion.
Detailed Instructions on how to complete the test, and specified verification points.
Expected Results of the correct state of the validation points after the test has been completed.
Actual Results: after the test has been completed, documentation of the actual results. If a verification point does not end in the expected state, recording details of the actual end state of the verification point. The defect tracking information will be included if a defect is written up.
An example test case for a login scenario for a web page may contain the following information:
Objective: Log in to XYZ application using a predetermined username and password to access the application’s Homepage.
Detailed Instructions: (Note: The instructions may be broken into steps, with a verification point on each step)
Verification Point: The login page will load with username and password input fields and a Login/Cancel button. Login page will accept username and password, and will load Homepage when the login button is pressed.
Now that we understand some basic terms and the general format of a test case, let’s talk about test execution strategies. If the team is following a test drive design approach, Unit tests will be written and executed against the code while it is being created. Once the unit tests pass, the code should be ready for integration.
Test execution strategy
Are executed against the delivered code. For many teams the first round of testing against a code base is manual while they gain experience, knowledge, and expertise. Once the functional testing on delivered code is well understood, teams generally identify candidates for automation.
The functionality of the full application. It may be a combination of manual testing and automated testing. The advantage is the team has assurance the full application is working as expected. On the other hand, the disadvantage is the length of time to run the tests.
Is when the regression is targeted toward new functionality and base level of application functionality. The reason behind this is to shorten the test cycle, especially if the application has a large library of tests.
Is a small set of tests that are executed to determine the general health of the application. Generally, the length of the time the tests will take to be executed is time boxed to a short duration. If automated, a smoke test suite may last 3-15 minutes, while if it’s done manually, it will generally take 30 minutes or less. An automated smoke test suite may be part of the quality gate in a continuous development/continuous integration process to determine if the code can be promoted to the target environment.
So, when should we use manual testing vs automation testing?
The question we set out to answer at the beginning of this session is when to use manual testing vs automation testing. Based on the information provided in our discussion, we use automation testing in places where the effort and costs to create the automation is targeting critical functionality. Furthermore, we use automation testing for tests which will be executed frequently, and tests that require the least amount of effort to automate, and maintain.
Whether you’re automating unit tests, functional, regression or smoke, these rules apply.
Targeting frequently used tests provide the highest ‘bang for the buck’ for your automation efforts.
Automation tests, like the underlying code, have to be maintained, have valid data inputs, and be modified when the underlying functionality is modified. Therefore, you’d also want to ensure that you automate those parts of the software that are less apt to be changed. This way, you can avoid going back and modifying your automation scripts. Of course, there are ways to develop your automation scripts to be more adaptable by creating reusable objects or using parameters so the functions or data changes can be accommodated quickly, with minimal impact and maintenance to the existing automation code maintenance.
Dispelling manual testing vs automation testing mentality
As usual, the many factors to decide if automation is a viable solution have to be balanced with common sense. The graphic below illustrates this best. At the end of the day, it’s easy to see this does not result in a 100% automation test strategy, as there will always be some manual testing.
Detailed exploration of the topic
Of course, there are many tools, some AI or machine learning based, that boast automatic object recognition and self-learning. But in the end, you still have to understand the application and the domain deeply and be able to think like an end user in order to develop meaningful test automation scripts. At XBOSoft, we firmly believe that “You can’t automate what you don’t understand”. We could be wrong, but in the last 15 years, we’ve developed a strong case for both manual testing and test automation based on understanding the domain and application. As such, our team has deep expertise on:
XBOSoft offers a unique blend of agile testing expertise for companies that are either currently in waterfall and converting their development methodologies to Agile, as well as those with various hybrid and popular Agile testing methodologies such as Scrum and Kanban.
Discover the challenges of Agile sizing, Epics, Stories, Tasks, and Spikes. We will start the discussion by defining the Agile components and then discuss the different sizing processes used and the thought process behind the philosophies.
Jimmy Florent is a results-driven leader with more than 16 years of experience in traditional and digital sales & marketing for multinational corporations. He has successfully built and implemented global strategies within consumer durables, retail, and service industries for companies like Black&Decker, Sony, Wickes, Tivoli China. Jimmy is able to bring together different teams in complex matrix structures to achieve common goals. He’s successfully led teams in China, UK, France while operating on a global platform (EMEA, APAC, China).