From a Swarm of Devices to A Cloud: A Hands-On Experience With Automated Testing in A Visual Studio App Center

From a Swarm of Devices to A Cloud: A Hands-On Experience With Automated Testing in A Visual Studio App Center

We have been developing mobile apps for over ten years, and it’s totally unacceptable for us to make a product that behaves unpredictably. That is why pumping up our testing team and infrastructure is our top priority.

Hundreds of Android devices, newer and older versions of iOS, and a range of display sizes makes QA engineers catch bugs both on real mobile devices and on different operating systems. But hardly anyone can run the same manual scenario on 10, 20, or 50 devices. Challenges like this have taught us to automate testing, especially for mobile devices. But let’s keep it real — creating and maintaining the testing infrastructure, even with 20 real devices, is a big headache.

Luckily, Microsoft has us covered — their Visual Studio App Center service has a virtual swarm of devices to run QA tests on. In this article, we want to share our experience working with the App Center, and some lessons we learned.

Why Do We Even Need It in The First Place?

Currently, we are working on a shopping platform app. The project has been going on for quite a while: the customer often adds new features and we have been their only developer. The app has over ten screens: from “Buy Now” to an array of messages, push notifications, and functions like “Shop This Look.” And every now and then there is an investor demo or a new mall launch, so the sooner we release (without compromising on quality, of course), the better.

So far, we have been running autotests on a limited range of devices that we had at our disposal (about 30 Android and iOS devices). These days, the number of users has been growing rapidly, and quality is even more crucial. We were confronted with a request to use a cloud service for running automated tests on a larger number of different devices.

While developing and releasing, we like to employ all modern and effective practices, such as cross-review, continuous integration, automated functional, and modular testing on a long list of iOS and Android devices, analytics, and crashlytics. We have used each of these practices before, separately and as a whole. But with a large-scale project and our extended team, we wanted a catch-all solution, a tool that provides all of the above and offers a one-stop service to manage the development and delivery process for diverse platforms.

Each role in a team had its own requirements. Developers wanted to stick to an established process (a repository, an automated build tool, etc.) and avoid using unfamiliar and unverified tools in the production. QA engineers aspired to drop the testing load on an array of devices. Managers and the customer aimed for a convenient interface with a transparent workflow to track the progress. Could there be a tool to satisfy all of these needs?

In researching available solutions, we considered different options (for example, BrowserStack for Appium tests). Microsoft App Center had a trial period, so we gave it a try and could see what would happen if we cared more about quality and automated the whole mobile application release workflow for any platform with one service. Here’s how it turned out.

How to Set Up An iOS App

Since the trial period has no limits in functionality, let’s set up a new iOS app in Microsoft App Center:


Choosing the platform:


Adding App Center SDK to the project:

pod 'AppCenter'

After installing the pods, open the AppDelegate.swift file and add these lines under our own import command:

import AppCenter 
import AppCenterAnalytics 
import AppCenterCrashes

In the same file, adding these lines to the didFinishLaunchingWithOptions  method:

MSAppCenter.start("{Your App Secret}", withServices:[

The same process works for Objective C, see the full manual here.


Open the Build tab and select our service, svc.



When configuring the build, it is important to sign it correctly, since it generates an app format file that can be run on actual devices.



Done! Press “Build Now” button and voilà.



The process for Android apps is similar, read the manual here.

Running First iOs Tests

You can include unit- and native tests for each platform into your build (there’s a tick for that). Let’s look at how automated testing works at different devices in App Center.

Configuring a Set of iOS and Android Devices and Uploading Tests

Open the “Test” tab and select “Device Sets.” Let’s make a set of devices to run our tests on. You can choose from over 250 Android devices and over 200 iOS devices (generation version + iOS version). See the full list here. What’s a bit disappointing is the fact that the new Apple devices appear in this list in 1-2 months after the sales start (and that’s the official response).

Let’s look at how to prepare the tests for upload to App Center (example for XCUITest). App Center allows you to build the app only, so the test project has to be built locally at your computer or CI.


# Generate an XCUITest bundle and your iOS application as described above.
$ rm -rf DerivedData
$ xcrun xcodebuild build-for-testing -derivedDataPath DerivedData -scheme YOUR_APP_SCHEME

# Upload your test to App Center
$ appcenter test run xcuitest \     
--app "app center username/<app name>" \
        --devices DEVICE_SET \     
        --test-series "master" \     
        --locale "en_US" \     
        --build-dir DerivedData/Build/Products/Debug-iphoneos

Appium Tests

First, make sure that your test framework matches the ones supported by the App Center. Besides, you must use the driver provided by App Center, and that limits the framework use (for example, you can’t use Google Guice).

Building a Project for Maven Users

Step 1: Add Repository and Dependency

You will need to add the JCenter repository to your pom.xml file:


Then add a dependency for the Appium test extensions:


This code will ensure the enhanced Android and iOS drivers are available at compile time. The enhanced drivers are provided primarily to enable the label feature. See Step 4 for more detail on the label feature.

Step 2: Adding a Upload Profile

Copy this snippet into your pom.xml in the tag. If there’s no section in your pom, make it. When the profile is activated, it will pack your test classes and all dependencies into the target/upload folder, ready to be uploaded to Test Cloud.

Building a Project for Gradle Users

Step 1: Adding the Repository and Dependencies

Check that the build.gradle in the root folder has the JCenter repository activated:

allprojects {
repositories {

Add the following snippet to build.gradle in the app folder:


Starting with Gradle 3.0, androidTestCompile is obsolete and replaced by androidTestImplementation .

Step 2: Automating the pom File Generation

You will need the pom.xml file to make the uploader work. Add this snippet to build.gradle in the app folder to make the pom file generate automatically:

apply plugin: 'maven'

task createPom {
pom {
    withXml {
        def dependenciesNode = asNode().appendNode('dependencies')

        //Iterate over the compile dependencies (we don't want the test ones), adding a <dependency> node for each
        configurations.testCompile.allDependencies.each {
            def dependencyNode = dependenciesNode.appendNode('dependency')
                    dependencyNode.appendNode('version', it.version)

        def profilesNode = asNode().appendNode('profiles')
        profilesNode.append(new XmlParser().parse(''))

Changes to The Tests

Step 1: Adding Imports

Import these packages into your test classes:

import org.junit.rules.TestWatcher;
import org.junit.Rule;

Step 2: Instantiating a TestWatcher

Insert this declaration in each of your test classes:

public TestWatcher watcher = Factory.createWatcher();

Step 3: Changing the Driver Type

Replace your declaration of “AndroidDriver<MobileElement>” with “EnhancedAndroidDriver<MobileElement>” or “IOSDriver<MobileElement>” with “EnhancedIOSDriver<MobileElement>”

private static EnhancedAndroidDriver<MobileElement> driver;

Step 4: Updating Your Driver Instantiations

Replace the way you instantiate your driver so that lines in the form of:

driver = new AndroidDriver<MobileElement>(url, capabilities);

…are changed to:

driver = Factory.createAndroidDriver(url, capabilities);

Using these drivers will still allow you to run your tests locally without additional modifications, but enables you to “label” test steps in your test execution using  driver.label("text") . The text and a screenshot from the device will be visible in the test report in Test Cloud.

It is recommended to have a call to label in the @After method, which will include a screenshot of the app’s final state in the test report. A screenshot will be taken, even if a test is failing, and often provides valuable information as to why it does so. An example of the @After   method for a test could look like this code:

public void TearDown(){
    driver.label("Stopping App");

Uploading the Tests to App Center Test

Step-by-step uploading:

  • Generate an App Center Test upload command using the instructions at starting a test run.

  • Pack your test classes and all dependencies into the target/upload folder:

mvn -DskipTests -P prepare-for-upload package
  • Perform upload.

At the end, you can look at the results on each device from the list:


Result Screenshots, Logs, and Reports

At each of the devices, you can have a detailed log and a screenshot for the crashlytics report:




Also, you can see the test runs statistics for a given period:


So far, the most significant App Center disadvantage is that you can’t access the “device” to debug and inspect the results. If something goes wrong with the tests and reading logs is not enough, you have to apply to the App Center support — although it would be nice to get access to a URL and a port to connect to the device if something goes wrong, and maybe figure it out on our own. One of the popular services for automated testing on devices, BrowserStack, has this feature included in Appium.


Conveniently enough, all the application release processes, from Continuous Integration to Continuous Delivery, are now in one place: Microsoft App Center offers CI build, test, deploy to store, analytics, and crashlytics.

Having tried less than half of the functions available in the App Center, our team was impressed. The most obvious advantage is that with App Center you don’t have to support each and every device with code. Other perks include:

  • No server needed for a swarm of devices

  • No hassle connecting all this swarm to the server

  • No dealing with Android glitches when it’s been running 24/7

  • No containerizing each device and managing those containers

  • Not being limited by emulators.

Microsoft App Center is a nice fit by initial parameters: while not being demanding in terms of integration, it provides all of the functions needed and spares the support routine. We are going to use it for real development tasks as it is a great tool to ensure quality.

from DZone Cloud Zone

Sharing is caring!

Comments are closed.