Running UI-tests in parallel

Some background

When the CI job approached a time limit that made it feel noticeable I started to look into how to make it faster. To no surprise to anyone the biggest amount of time was spent on running the UI tests. Outside of the .Net space there are a few ways to run tests in parallel but as far as I could find there wasn’t anything suitable that I could use for our specflow/selenium setup (I am aware of selenium grid). I decided to start easy with a poc and see if I could improve the tests times.

Site state

To make the testing slightly more complicated most of the UI-tests requires the site to be in a certain state. Some tests may require the time to be after 22.00, some other that a certain payment method is deactivated etc. State is set in the setup (via making a http request to the site) and then reset to a “normal” state in the teardown of the UI-test.

So even if we could run the tests in parallel it would not work if the tests are run against the same site since test A may have set the site in such a state that test B would fail. This won’t be a problem when the tests are run one at a time but if they are running in parallel and test a and test B are run at the same time it would lead to unexpected failures.

What I wanted to achieve

Obviously make the tests run faster in the CI build but without using too much magic. It would also be nice if I could use the same technique when running the test locally which I usually do via the resharper test runner.

A solution of sorts

First problem to solve was how to make the tests run in parallel to avoid site state to be a factor. I was already using iis express to host the site test were being run against so instead of spinning up one instance I figured I could spin up multiple sites on different ports. I started with three instances (no specific reason to start with that number, but like Jarvis Cocker I had to start somewhere, so I started there).

Project structure

Previously all the UI tests were contained in the same assembly. Before any of the tests were run iis express was started in a process and the coypu browser session is being set up. Since I needed a way to start upp three sites I decided (and this may appear slightly wonky) to split the test project into three different test project. In the startup (which correlates to [BeforeTestRun] hook in SpecFlow in my setup) an instance of iis express is started on different ports. The tests are then executed one at a time against the same site.

project

(This is fairly specific to my specflow setup but since it may interest someone I though I’d mention it. Everything regarding test logic, such as step files and general startup orchestration are still kept in the original test project. The only thing that the three new project contains are the feature files. In order to get that to work you need to tell specflow where it should look for step files since it won’t be in the same assembly as the feature file. This is done like this article explain.)

Running in resharper

This is fairly straight forward. In resharper there’s a setting where you can select how many assemblies should be run in parallel. I increased it to three, selected all three testprojects and ran them in parallel. Yay!

I'm aware that the text says 3 and the image says 4

Running on the CI-server

On the CI-side of things we have jenkins executing a psake build script. The actual execution of the tests were as simple as just running nunit console against the produced test assembly. Now I needed to to the same thing but three times in parallel. To execute these background task I used the powershell command Start-Job.

$job1 = Start-Job -Scriptblock {param($nunit_console, $dll, $resultDir) & $nunit_console /xml:$resultDir\test1.xml $dll } -Arg $nunit_console, $assembly01.FullName, $testResultDir.FullName
$job2 = Start-Job -Scriptblock {param($nunit_console, $dll, $resultDir) & $nunit_console /xml:$resultDir\test2.xml $dll } -Arg $nunit_console, $assembly02.FullName, $testResultDir.FullName
$job3 = Start-Job -Scriptblock {param($nunit_console, $dll, $resultDir) & $nunit_console /xml:$resultDir\test3.xml $dll } -Arg $nunit_console, $assembly03.FullName, $testResultDir.FullName
    
Wait-Job $job1
Wait-Job $job2
Wait-Job $job3

Receive-Job $job1
Receive-Job $job2
Receive-Job $job3

And now the tests run in parallel on the CI-server as well.

Result

The total CI-time was reduced by around 50% (from roughly 4 minutes to roughly 2 minutes). This was a good enough improvement for me to stop experimenting with tampering with more settings (increase number of projects run in parallel etc). It would also be helpful to not have to split the test projects but perhaps do a dynamic splitting on test start. But that gives me something else to tinker with for the rest of the year.

  • Fabian

    was this for a website ui test, or windows forms ui test ? i need to do that for windows forms applications. we use white framework here but the ui tests cannot run in parallel because the mouse is used for click events . ideas ?

    • http://www.popkram.com Stefan Forsberg

      This was a website ui-test. I’m afraid I have no experience at all regarding testing native applications so no ideas at the top of my head.