For testing the performance of mobile application software, the testing focus is different than when testing general computer software due to the nature of embedded software systems. Embedded software is specially designed for each device, so software (including OS and application) is closely tied to the hardware. Some typical complaints for using mobile application software may be:
- Do you feel some times your mobile very slow response after clicking a button?
- When you are playing a game, can you accept an incoming call easily?
- Do you feel your mobile works slowly when you play a game for 2 hours?
All of these problems are performance related but finding exactly where the problem is can be difficult. Performance testing for mobile applications always includes testing hardware, signal (web), OS and the application itself. There are a variety of performance test tools, but each is usually for a particular test, so doing a complete mobile application performance test may require a combination of tools. For instance, some tools can be used for automation, such as Robotium, which is an emulator (but it can only emulate usage of one application at a time). Another example is J-Benchmark which can test Java’s performance. Others, such as Quadrant, can be used to measure resources, i.e. CPU usage, memory usage, disk usage, etc. These are just a few of the tools that we use.
The difficulty in doing mobile performance testing is that it is difficult to judge if errors or low performance were caused by the OS, application or hardware. After using a variety of tools, you need to collect the data, and act as a detective. One useful way that we use is to substitute environments in order to isolate problems. For example, test using the same mobile device with different OS such as iOS3 and iOS4. In this case, if the application runs smoothly in iOS4 but gets bogged down with iOS3, this points us in the direction of investigating the OS further. This is just one of many methods. Since mobile application testing tools are not that mature, you need to master several tools and understand how to interpret the resulting data.