From XBOSoft’s recent Quarterly Webinar by Jan Japp on Mobile Testing, we had some really great follow up questions. Q & A follows:

Q – For a startup company at the stage of beta development, what type of testing tools should be preferred? Is there any timeline with respect to the number of months the application has been around? Like in first 11 months use free tools, 12-18 months use slightly expensive, and so on…

A (Jan Japp) – For starters, I would recommend developers’ tools where possible for cost savings and flexibility. The developers already have their platform and the tools usually are good for quick testing. However, when the product becomes more mature, I suggest real testing devices (maybe a device board? That can be soooo much fun!). But it costs a lot of money, so it often comes down to commercial testing tools like SauceLabs, BrowserStack, and Perfecto Mobile. If an organization doesn’t want to pay for a commercial tool, I’d start a discussion about quality, risks, and costs.

I don’t really think there is a standard timeline like that. Every situation is unique. From my experience, most projects that had a mobile component had to deliver within a year or so. And we (my colleagues and I) didn’t have the time for growing into new toolsets.


Q – Is it preferred to have mobile testing done in-house or have it outsourced to external specialists or external firms? For a consumer-focused startup, what is suggested?

A (Jan Japp) – That fully depends on the quality of your in-house testers! Mobile testing requires extra skills (usability testing, mobile test tooling, mobile platforms, Selenium/Appium, etc.) If you don’t have these skills in-house you better hire external specialists.

Also, for some test goals, you can outsource a part of the testing. I’m talking about crowd testing. What does the public think about your app? Is it easy to use? You can do that with a user panel in case you have a controlled group of users (did that on project 1) or you can organize testing sessions with for instance employees from other departments (did that on project 2), but that takes an awful lot of time and the numbers of testers you have is still limited. Crowd testing can be a solution, but it is expensive.

When it comes to consumer focussed apps, I would surely do some testing by real users. They view and use a screen differently than a developer or professional tester.

Thanks, Jan Jaap for answering these questions.