Is it possible to run simultaneous runs with different device configurations?

Question

Is it possible to run multiple test runs with different test suites at the same time with an account that permits device concurrency? https://forums.xamarin.com/discussion/39831/run-ui-tests-on-multiple-devices-simultaneously In this question the answer was this When you create a test run in Xamarin Test cloud, the second page in the Test Run wizard has an option to run tests concurrently (the Parallelization drop down). If you are submitting tests at the command line, you can run tests in parallel using one of the following two command line parameters: --test-chunk to run tests in parallel by method --fixture-chunk to run tests in parallel by fixture. But can I test on different devices like in this example? Device1 - test1, test2 Device2 - test1, test3 Device3 - test4, test5

Answer 1

It is possible to run multiple tests with different test suites at the same time using device concurrency in the Xamarin Test Cloud. This is true whether or not you are using parallelization, however, parallelization does complicate the matter somewhat, because parallelization runs on multiple copies of a single device, and those copies also count against your concurrent devices.

When you select to run on parallel devices, the Test Cloud will automatically run the devices on as many copies of that device that are available.

Example Scenario

• Device Concurrency - 3
• First Test Run - 1 device selected
• Second Test Run - 2 devices selected

Without parallelization - Both tests can run as soon as devices are available, because the concurrency is the total maximum for all tests. You could similarly have three test runs each with a single device and all could start immediately. If you exceed your device concurrency, then your remaining tests will be queued up to wait for another device to be finished.

With parallelization - The first test run may use up 1, 2 or all 3 device concurrency slots; depending on how many devices are available. The slots that are used up by the first test run won't be available for the second test run until tests on them have finished.

Conclusion

Theoretically you can have multiple test runs all using parallelization at the same time; but in practice you might not have enough concurrency slots for them to actually progress concurrently.

You can think of it as a trade-off, for individual test runs on a single device, parallelization will let you get your test results much faster; but subsequent test runs will often have to wait, so it is a tradeoff. But whether you use it or not, you can still queue up more tests afterwards; so there's no "penalty" for adding extra tests beyond what your concurrency will allow to immediately run.