UI-testing (EPiServer): the need for speed

Executing our two test takes around 20 seconds. If you’ve run the test you’ve noticed that a lot of the test time is actually spent on opening a new browser window. One way to mitigate this is to use the same browser throughout all the test which opens before the first test and closes after the last test. Note that there are downsides with this since your test may now affect other tests because of the shared browser data (cookies etc.). We can try and mitigate this by clearing the state between tests but the absolutely full proof way of ensuring a clean slate is to fire up a new browser before each tests.

Sharing the browser

We need to refactor our test code a bit to make this happen. Let’s start with adding some code that runs before any tests are run and after all the tests are completed using the SetupFixture attribute.

[SetUpFixture]
public class SeleniumServer
{
    public static IWebDriver Driver { get; private set; }

    [SetUp]
    public static void Start()
    {
        Driver = new FirefoxDriver();
        Driver.Manage().Window.Maximize();
    }

    [TearDown]
    public static void Stop()
    {
        Driver.Quit();
    }

    public static void ResetBrowser()
    {
        Driver.Manage().Cookies.DeleteAllCookies();
    }
}

The TestBase class now looks like this

public class UiTestBase
{
    protected IWebDriver Driver = SeleniumServer.Driver;
    protected const string BaseUrl = "http://alloy.demo.episerver.com";

    [TearDown]
    public void After()
    {
        SeleniumServer.ResetBrowser();
    }

    public void Visit(string relativeUrl)
    {
        Driver.Navigate().GoToUrl(BaseUrl + relativeUrl);
    }
}

Bring me the head of… oh wait

A great way to speed up your tests is to use what is called a headless browser. A headless browser is basically a browser without a GUI and since it doesn’t need to show a user anything it’s considerably faster than a conventional browser. This has been missing from the .net space but now PhantomJS is available and it’s also integrated into the Selenium code! To use it we need to download the phantom js executable either via the PhantomJS page or by fetching the NuGet package

Install-Package phantomjs.exe

To use phantom we only need to adjust our SeleniumServer class to initiate a phantomjs driver instead of a firefox one. The constructor needs to know the path the phantomjs executable.

[SetUp]
public static void Start()
{
    //Driver = new FirefoxDriver();
    Driver = new PhantomJSDriver(@"(yourPathHere)\phantomjs.exe.1.8.0");
    Driver.Manage().Window.Maximize();
}

Time spent executing tests

As usual the numbers themselves are not important but rather how they compare to each other.

2 tests

  1. New browser: 20 seconds
  2. Shared browser: 13 seconds
  3. Headless browser: 7 seconds

6 tests

  1. New browser: 44 seconds
  2. Shared browser: 20 seconds
  3. Headless browser: 8 seconds

14 tests

  1. New browser: 1 minute 48 seconds
  2. Shared browser: 29 seconds
  3. Headless browser: 9 seconds
  • Mikael Lundin

    I love the thought of headless browser, but the questionis if it gives accurate results. Can you ensure that it behaves equally to webkit? Also, 7 seconds are still too slow as you easily end up with hundreds of ui tests ;)

    Loved your post. Didn’t know about phantomjs before.

    • http://ariya.ofilabs.com/ Ariya Hidayat

      Here’s my take: don’t rely only one layer of defense. For more details, check out http://ariya.ofilabs.com/2012/12/quality-code-via-multiple-layers-of-defense.html.

      Many people use PhantomJS, directly or indirectly, as part of smoke testing. It won’t be as comprehensive as a full-blown Selenium grid but it will tell you in a matter of seconds if you screw up something as basic as the login screen.

    • http://www.popkram.com Stefan Forsberg

      The point you make about ensuring equal behaviour is a fair point but it’s not a problem with just a headless browser but browsers in general (webkit vs IE, vs versions of IE etc). 

      I guess it’s a matter of trade of as always; good enough test-suite-run-speed for a high enough certainty that you can make a release. As Ariya mentions different drivers can serve different purposes, maybe developers locally run the tests with phantomjs for quick(er) feedback before comitting and the build server runs the tests with a “real” browser once every night (depending on your release cycle of course). 

      In my last project we used mechanize for ruby which is a headless browser without javascript capabilities so we used it when the page to test didn’t involve javascript and we trusted that enough to make releases on passing tests.