Sometimes defect verification can make you insane. Eliminating non-reproducible defects can save everyone time and energy, so we thought we’d list out some of the common reasons behind ‘cannot reproduce’ as a guideline for fellow testers trying to verify defects.
Generally, a defect is described with conditions and steps for reproduction. With the same conditions and steps, you’d expect the same result. When a defect cannot be reproduced, and you’re sure the step-by-step descriptions are clear and unambiguous, the cause is usually in the conditions. Conditions should contain all environment information when the defect occurs, including information on the hardware, software, network, baseline data and starting point in the application, and previous behaviors with the system under test. Based on our experience, four of the most common reasons for ‘cannot reproduce’ include:
- Different software or different version of software used
When defects are documented, you usually include:
1) Operating system
2) Browser information
3) Installed application, especially firewall and anti-virus software
4) Target software version
Non-reproducible defects usually occur because a different version of software to reproduce and the defect is fixed when fixing other problems. In this case, it is better to check the version (or submit date) information first when reproducing a defect and trying to restore a system of that version.
- Different hardware profile
Once a laptop related bug required several hours to reproduce using several different laptops. Finally, we found out that the feature was not compatible with just any wireless LAN card. So documenting the specific testing devices is critical to prevent such non-reproducible defects.
- Network conditions
More and more non-reproducible defects are caused by network conditions especially with SaaS and mobile-based software. It is difficult to figure out if a defect is related to network status. A tester should be careful when the reaction from target system is slow and returns the result. Try to reproduce the problem with many different network conditions that you ‘think’ are abnormal.
- System status
Most “cannot reproduce” situations are due to different system status’, known as preconditions. In most cases, the tester will provide all the information step by step. But sometimes the defect still cannot be reproduced. This may be caused by lacking some specific data stored in the test database, or different browser cache status, or any other previous operation which causes a different system status.
When doing defect verification, its best to investigate the defect first and figure out the possible causes before reporting the defect. Make sure you characterize all the conditions carefully. This not only saves time for fellow testers but developers as well.