Are there any good strategies for A/B testing on mobile devices?

Question

Based on what I know, A/B testing frameworks are for working with websites. For example, you want to know which of blue/red color of "Buy" button will encourage user to click it.

Since website information of user interface comes from server, and browsers display it, A/B testing is possible. However, since the user interface of mobile applications are native and static, it's not possible to implement A/B testing in the same manner (if I'm not mistaken).

So, my question is how to implement an A/B testing framework from the database/services all the way through to a native mobile application.

Any suggestion would be appreciated. Thanks.

Answer 1

A/B testing on mobile is different from website in two main points:

• A website is always connected to the internet. A native mobile app is like a Windows app in the 90's: it does not always have a reliable internet connection
• The web is much more mature than the mobile. Modern web A/B testing tools can completely control all the elements on a web page.

The basic thing to do is to create a client-server architecture:

Tag manually the elements to test in your app

You will tag manually the element that you would like to A/B test and your framework will ask to the server which element to show. (The variations are stored on the server side). For example, the color of the button can be retrieved from the server and set at runtime when the button is displayed. It is not completely true that native mobile interfaces are static. They can be modified at runtime programmatically.

Send events to your server

The second step is to send an event to your server when the variation has been viewed and when the conversion succeeds.

Potential issues

By using this trivial architecture you will have the following problems:

Always show something

When the button should be displayed, the mobile device might not have an internet connection at all. You still need to show something to your user event if the internet connection is not there. The solution is to display a default value.

Store events

When a goal is performed (for example a click on a button) and if the mobile device is offline, the event will be dropped and you will end with wrong statistics. The solution is to save your events locally and to send them to the server later (when the internet connection is back).

Never wait for a variation

If you ask a variation from the server only when you need it, you application will be really slow (which is not acceptable). You need to retrieve your variation data as soon as possible in your application.

You mobile framework should be built to never block the application. That's a golden rule on mobile.

Save variation attributions

The server should save which variation has been assigned to a device to get reliable results.

Technology stack

Operating system/programming language

You can choose any technology stack. Just make sure that your server side can handle the load. If your mobile apps have many users, you will get many requests on your server. You might need to have a load balancer between your web servers. If you choose non-free technologies, watch out for licensing cost as high load means high cost.

Database

You might also have millions to database records. Generating reports might not be just easy as doing a "SELECT COUNT(*)" from you database table for performance issues. Be careful if you choose non-free technologies.

Disclaimer: I am the CTO of Arise.io

Answer 2

In another answer to this question, poiuytrez provides a nice overview of the considerations involved with rolling one's own mobile A/B testing framework. Since decorum prevents him from mentioning off-the-shelf solutions due to his involvement with a product in the space, for the sake of completeness, I'd like to suggest that anyone looking to implement mobile A/B testing should consider leveraging existing solutions, especially because the problem lends itself to reasonably natural separation of the framework from the subject app.

Commercial A/B Testing Frameworks

The ability to effectively separate A/B testing from a subject app, combined with the possibility of immediately measurable revenue improvements, has lead to a flurry of commercial activity in mobile A/B testing for apps:

• Appiterate
• Apptimize
• Arise
• Artisan Optimize
• Bestly
• Leanplum
• Optimimo
• Splitforce
• Swrve
• Vessel

Free A/B Testing Frameworks

Amazon has its Amazon A/B Testing Service for in-app experiments.

Open source A/B Testing Frameworks

If none of the above solutions are suitable for your purposes, it's still possible to avoid building entirely from scratch by leveraging an open source mobile A/B testing framework:

• Clutch.io
• iOS A/B Split Test Library
• Switchboard

Disclaimer: I have no products in mobile A/B testing, and I sponsored the bounty on this question, so I am not eligible for the award.

Answer 3

For an overall overview on running A/B tests on mobile apps here is a blog post with 10 steps: http://www.andrewthompson.co/2013/12/running-ab-tests-on-mobile-apps.html

Answer 4

Some solid posts have been written by mobile engineers at Facebook and LinkedIn on their internal A/B testing frameworks in the past month. But if you're running lean and don't feel like reinventing the wheel, check out one of the solutions listed above (Disclosure: I work for Splitforce).

How it works

For mobile apps, A/B testing basically works by replacing static, hard-coded objects with dynamic objects that can be controlled from a remote server.

This methodology raises a potential performance issue: What if the end user's device is not connected to pull configuration data for an object being tested?

We've built Splitforce to mitigate such performance risks. On first app launch, the mobile app requests configuration data from the server and then caches that data locally on the device. This is to both ensure a consistent user experience on subsequent app launches, and prevent corrupt test results by guaranteeing accurate attribution of conversion events to variations.

If the end user's connection fails or is timed-out on that first app launch, the library displays a hard-coded 'default' variation. And to make sure that everything is looking good before you go live, we've built a 'shake to preview' in debug mode that does just that :-)

Once the SDK and experiment has been integrated, non-technical product or marketing folks can setup new tests or tweak existing tests on-the-fly - without having to resubmit to the app stores or hassle engineers.

And once the app is deployed with Splitforce event data will be stored locally and sent back to the website to be displayed for each variation alongside measurements of observed improvement and statistical confidence.

How is it used?

We've seen Splitforce used to A/B test:

• UI elements + layouts (color, text, images, ad/menu placements)
• UX workflows
• Game dynamics + rules
• Prices + promotions

We've also seen the tool used to debug and reconfigure apps remotely, by essentially setting one variation of a test subject to 100%.

Answer 5

Unlike on the web, changing content on mobile apps is a more involved process. Specifically: all code updates to your app have to be approved by Apple/Google.

Enter services such as Leanplum. Leanplum allows its customers to test nearly every aspect of their app in three primary ways:

1. Visual Interface Editor: This requires no coding, and Leanplum will automatically detect the elements and allow you to change them. No engineers or app store resubmissions required.
2. Content Management System: Enables more granular change to the app.
3. Mobile Marketing Automation: A full-service solution where every message and notification sent to your users (whether in-app, push, email, or in the news feed) can be A/B tested and personalized.

For more information on best practices for mobile A/B testing, you can check out this Leanplum blog post.

To learn more, check out our site at leanplum.com. We offer free 30-day trials of our platform, so you can experiment with building better brand relationships with your customers, without any commitment.

(Disclaimer: I am an Engineer at Leanplum.)

Answer 6

Some times ago I wrote simple A/B testing library for Android applications. For gathering data you can choose mixpanel, kissmetrics or your own server. You can find more about this library here:

• BestChoice - Android A/B test library