How device coverage should influence your mobile app testing strategy ?

Brief Intro: There was a time when everything was manual, then came automation and now the device farms. I am talking in the context of Mobile app testing. When you change focus from Web to Mobile, one of the thing which ideally should get detailed attention from you is “device coverage”.
Why device coverage matters: Let me quote some numbers for Android OS first to give you a rough idea. Below is a snapshot from OpenSignal. According to the report, android device fragmentation has increased 28% over last year.

When it comes to iOS the scene is relatively better. Below is the screenshot of iOS to Android comparison in terms of fragmentation.
Probable solutions to device Coverage:  The good news is there are lot of vendors offering device farm and to name the prominent ones:
  • Xamarin Test Cloud Service
  • Cloud Test Lab by Google
  • AWS Device Farm
  • Testdroid

But the not so good news is most of these are paid service and the charges for engaging a device could go up to $240 per device per month (on AWS device farm for example). Till device farm became easily accessible and popular, folks were majorly testing on emulator and some handful of devices. Beta testing phase used to be a little longer and most of the bugs were found during that and field testing of app.
Now that device farms brings you plethora of devices but the concern is every minute of a device in a farm costs you.  What should be your strategy for mobile testing and automation given the limited testing time you have and the money to spend on?
Contextual information about the app:  A lot in your strategy would be shaped by what kind of app it is ? Is it a B2B or B2C ? Is it a hybrid app or native app? Do you know your targeted customer segment ? Do you know the age segment of those customers ? Possible usage patterns  of your app?
Once you have these info, you could potentially decide what are the typical devices which might get used.  There are other set of questions which can definitely lead you to some pointers to test strategy for mobile apps.
  • Should you invest in devices locally ?
  • What should be your emulator configuration to run test?
  • Can your functional test run in parallel ?
  • Which section of functional test you should run for device coverage?
  • What devices you should opt when using device farm ?
  • When should you run your tests on Device Farm ?
Probable approach:  We have tried to diffuse how you can help shape your mobile testing strategy in the below picture. It’s more to prepare you and open you up for discussion which should help you arrive at your device testing strategy.
Apart from the test strategy you also need to have a highly optimised code where you are handling “wait” smartly in your test code. Remember that with each minute of your test execution on actual device, there is a cost associated.
Want us to help you with your mobile testing strategy and automation, reach out to us.

Popular posts from this blog

Defect Tracking System – Wastage of Money & Time ???

Image
As a QA I have been logging bugs since 5 years. But what do we do with that ? A typical flow: If this is the only thing we need to do then why taking so much of pain to have a defect tracking system . To log a standard bug (Standard Bug= Broken Functionality), we need to write steps to reproduce, pre-requisites (if any), expected result, actual result, priority, severity, snapshot (if any), Test environment details and finally save it. This takes around roughly 3-5 minutes to log a bug. To say truth, I have never seen any report coming out of defect tracking system which is of use in analyzing the pattern of bug or finding out any trend in them. All I have seen apart from tracking usage, is the summary of bugs with count based on Priority and Severity. Is that the only purpose of Defect Tracking System? If that is the case then why not use a Google spreadsheet rather then investing money of the company and time of QA in a bug logging system. I am sure if we mine the data which i
Read More »

The 3 qualities that will always help you become better at almost anything…

Image
Have we ever noticed how excited we are when we start anything? But it’s a sad fact that 70% of people abandon what they have started with so much energy and love. It could be as small as building a hobby or starting a pet project or even an activity or a side learning we started. “ Well begun is half done .” Aristotle did say this but we are judged for how we have finished something. You separate yourself fro m  the masses when you choose to give that task a finish.  The first quality is finishing off what you started and it helps develop mental stamina or willpower.   It builds confidence and inculcates as a habit in a long run. You don’t need to sign up for the mammoth activity to build this habit, even finishing mundane day-to-day routine tasks will help. Diligence — the quality of working carefully and with a lot of effort Diligence is a quality that almost never will get unnoticed. It’s one of the underlying traits of great works or success. The key to mediocrity on the o
Read More »

Test Case Paths : Happy, Sad & Bad

Image
I came back from the background of writing test cases which were categorized in either Positive or Negative test cases. Positive test cases were business flows which may or may not give any result but they will certainly not give any exception or error. Negative test cases are those test cases which were bound to result into some exception or better expressed as error causing scenarios.  Then those positive test cases were prioritized as High/Medium/Low . Finally it used to result in quite a comprehensive test case document. Then it will go through the process of review and few more updates or change of priority etc. Certainly it is a tedious process. Here in ThoughtWorks I got introduced to different way of categorizing test cases called “ Happy/Sad/Bad ”. A test case which results you in a positive result is a Happy path , for e.g. entering proper username and password on login page. A test case which yields no result like entering invalid username and invalid password on a logi
Read More »