How does one begin to thoughtfully build a mobile testing strategy for ensuring quality in this constantly shifting mobile landscape? For starters, building out a QA “dimensional” framework provides a great strategic foundation that can meet the immediate and future testing needs in this dynamic environment.
Critical dimensions in a mobile QA and test strategy include:
- Platform Compatibility (OS/Devices)
- Network Variances
- Test automation
- Usability and User Experience
If your intertested in a PDF white paper with more details on developing your mobile testing strategy, go here. Our recommended mobile test strategy dimensions are summarized below:
Mobile Functional Testing
When you approach mobile functional testing, you need to start with the type(s) of mobile application (Pre-Installed Mobile Applications, Native Mobile Applications, Mobile Hybrid, Mobile WebApp, Mobile Website) you’ll be working on and then build out the important characteristics to be checked: such as proper navigation and rendering with various screen resolutions, user interface (design and rendering), security, performance, and browser compatibility among others.
Note that mobile applications have particular characteristics both from the user and platform point of view that require special attention, such as Interrupt and Ad hoc testing. There are so many notifications and interrupts that you can’t enumerate them all in test cases. This is where exploratory and Ad hoc testing with experienced testers who understand the domain and user is critical.
As with testing traditional software applications, there are many tools and utilities to supplement your mobile efforts. Most SDKs provide tools, but there are many additional tools that can help you. A good source is TestingTools.com’s Mobile Testing Tools.
Mobile Platform Compatibility
Because a mobile application must be compatible with native device features and not interfere or inhibit common functions, mobile QA needs to incorporate the capabilities and limitations of different device/OS platforms in order to differentiate defects from platform limitations. With so many permutations and combinations to test, it’s critical to have a strategy for prioritizing the combinations to be tested. To help manage the potentially large number of test combos, simulators and remote devices provide reasonable alternatives. In the end, you may require a combination of real, emulated, and remote devices.
Mobile Device Emulators
Though testing with a real device is preferred as it represents the real situation from both user and device standpoints, emulators can be used to complement real devices especially for basic functionality testing. Some good sites for emulator resources can be found at Browser Stack and MobiReady .
Using Remote Device Access Services
It is not economically feasible to buy all the mobile devices that could use your mobile application. On the other hand, simulators cannot be counted on completely for testing and launching a mobile application. Remote Device Access (RDA) services can complement your handset inventory to give you a sense of comfort that your application is ready for release. RDA services provide access to a live device over the Internet enabling you to load your application and test it on the device remotely. RDA Service providers include for instance Browserstack, DeviceAnywhere, MobileLabs, Perfecto Mobile and Amazon Web Services.
One of the special aspects of mobile application QA is that it involves more than the mobile phone or application. Network configurations can significantly impact your application’s performance and impact your test environment. Network variations including 2G, 3G, 4G, LTE, CDMA, GPRS, GSM, HSCSD, SIM, WAP should make up a critical part of your test matrix depending on your target market.
Performance is much more complicated with mobile applications as the parts can be disjointed and not well integrated. There are three main components; server, network, and mobile device with applications. You need to be cognizant of each, while determining your strategy based on the type of mobile application discussed above. In addition, you should take into account mobile application behavior (on device) under stressed resource conditions (memory, network) with various permutations of stress on the network and server components. Some recommendations include:
- Test in a production environment
- Anticipate for and test for peak load, benchmark progress as new versions are released
- Make sure that your performance tests accurately model the actual behavior and environment of your mobile users.
- Test across the network geographies of your users. Get out of your lab.
Mobile Test Automation
As with regular automation, test automation for mobile follows the same rules; it makes no sense to automate where it’s not economical. Don’t automate tests that aren’t run often or tests what will need to be changed frequently due to an application that is changing often.
There are many automation tools available, some open source and many commercial as the market has matured in the last few years. Each has its own strengths and limitations and you may need a combination of several to do what you need. Most of the test automation vendors now have mobile capabilities at different levels of functionality. Appium is a good open source test automation framework for use with native, hybrid and mobile web apps. It drives iOS and Android apps using the WebDriver protocol. There are many others and (FossBytes) is a good resource for details.
With the popularity of wearables, the importance of integration comes to the fore. Almost all wearables are dependent on the mobile smartphone as a communications and processing hub. Inherent in their functionality is communication with the smartphone either through Ant+, WIFI, Bluetooth or NFC, and sometimes a combination.
Most payment mechanisms will be implemented through NFC (i.e. from Apple watch to point of sale terminal) but then communication from the Apple Watch to the iPhone will be done via Bluetooth. As you can see, all of these interfaces will require a high level of interface testing and integration. Otherwise, sporadic transaction success will lead to frustrated users that don’t end up using their devices.
The usage of these interfaces and the gathering of data broadens the concerns for security and privacy.
Mobile Usability and User Experience
Getting a product out that is functional is not enough. Touch enabled and haptic interfaces make the interaction between the user and the device more direct and personal. The distance between user and device is reduced not just physically but also emotionally, and users create a relationship with the device; hence the importance of user experience in mobile applications.
Conduct thorough usability testing and don’t overlook the raw experience. Give your mobile application to a user and don’t say anything, just watch. You’re in trouble with basic functionality if they can’t even get it going, and in worse trouble if they don’t even notice some of the features that you had been working so hard on. Observe them and see what they access when the product is in use.
Remember the initial user experience should not be undervalued as it sets the tone for the entire product experience. Most mobile usability issues arise from two main problems: Direct usage of web design approaches and direct porting between different mobile platforms.
Special guides and recommendations on usability for mobile platforms have been created by various vendors/organizations such as:
- Android (https://developer.android.com)
- iOS (https://developer.apple.com/app-store/resources)
- Windows (https://developer.microsoft.com/en-us/windows/apps/design)
Developing a mobile testing strategy is critical. Here’s why. You can’t test everything and this is especially true across mobile platforms. Mobile applications integrate with almost everything requiring special attention to interfaces and user scenarios. Performance and poor user experience are app killers. Don’t let your lack of strategy and planning lead to failure.