We’ve been testing mobile applications in the field as well as in the lab for a large mobile device manufacturer. We test in the field, in different countries to determine how different carrier networks, applications, and tasks, impact the user experience. When doing mobile performance testing, there are any number of factors that can be measured and tracked. They all will vary depending on the device, it’s configuration, network, task being executed (what software and what action), etc.
Mobile Performance Testing Measurements
First look: This is the time that it takes for a user to receive the first ‘meaningful’ data/information back from the mobile application. If the application is native to the device and just needs some complementary data, this is much different than a mobile site. And a site specifically designed for mobile will be much more efficient than a normal website. Definitions of ‘meaningful’ of course vary by application and context.
Payload: This is the amount of data transmitted for the first initial screen accessing the application/website.
Complete load: This is the amount of data transmitted for an entire task.
Task completion: This is the amount of time a user takes to complete an entire task. Depending on what data that must be accessed and what screens must render, and the complexity (payload) of those screens, this will affect performance.
Many webapps and mobile browsers are specially designed to reduce payload and therefore increase performance. As with any mobile testing, its important to keep the environmental variables in control and documented so you know what caused an increase or decrease from one test to the next. Of course there are many other measurements for performance. These are just a few. We’ll keep you updated with more measurements and evaluation schemes…