Automated Browser Testing on Internet Explorer with Capybara

TL;DR

  1. Install Windows in VM in VirtualBox. Remember to enable IO APIC in VM System Settings.To run headless with built in VNC server:
    <code>VBoxHeadless -s <vm name or id> -n -m <VNC port server> -o <VNC password>
    </code>
  2. Download and run Selenium Remote WebDriver on CI server:
    <code>curl -O <a href="http://selenium-release.storage.googleapis.com/2.41/selenium-server-standalone-2.41.0.jar">http://selenium-release.storage.googleapis.com/2.4...</a>
    java -jar selenium-server-standalone-2.41.0.jar -role hub
    </code>
  3. Download and run Selenium Remote WebDriver andInternetExplorerDriver on VM
    1. Install Java if necessary.
    2. Download Selenium.
    3. Download 32-bit version of InternetExplorerDriver (See gotchas at bottom about the 64-bit adapter).
    4. Set registry key if using IE 11 (Snippet is for 64-bit Windows). Paste this into reg.key file and double click
      <code>Windows Registry Editor Version 5.00
      [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Internet Explorer\MAIN\FeatureControl\FEATURE_BFCACHE]"iexplore.exe"=dword:00000000
      </code>
    5. Run Selenium Node
      <code>java -jar /path/to/selenium.jar -Dwebdriver.ie.driver=/path/to/ie_driver.exe -hub http://<CI server hostname/IP>:4444/grid/register -browser "browserName=internet explorer,version=<IE version on this VM>"
      </code>
  4. Configure Capybara to use Selenium Remote WebDriver started in step 2:
    <code>Capybara.default_driver = :remote
    url ="'http://<CI Selenium Hub>:4444/wd/hub"
    capabilities = Selenium::WebDriver::Remote::Capabilities.internet_explorer
    capabilities.version = ENV["SELENIUM_BROWSER_VERSION"]
    capabilities.javascript_enabled = true # Javascript not enabled by default, see "Gotchas" below.
    Capybara.register_driver :remote do |app|
      Capybara::Selenium::Driver.new(app,
     :browser => :remote, 
     :url => url,
     :desired_capabilities => capabilities)
    end 
    Capybara.app_host = "http://<CI server hostname IP>:#{Capybara.server_port}"
    </code>
  5. Gasp in horror at all the IE bugs you haven’t seen before.

TS;NMI (Too Short, Needs More Info)

A project we are working on has a fairly comprehensive (if not entirely stable) suite of integration tests, that we run via aJenkins whenever we make a change to the project.

Most of the team develops on OS X using Safari or Chrome, and the CI system runs the test suite on against Firefox running in an Xvfb session.

This covered most of our bases, but for a long time we’ve ignored the Elephant in the room that is Internet Explorer. We’d poke at it in a VM once new features were basically finished, but not in any systematic or automated way.

So finally we decided to make some time to invite IE to party with the other cool kids.

Our first plan was to do a quick integration with SauceLabs, but this never really worked well for us:

  1. Tests would take a loooong time to run. A test that would takea minute and a half via Saucelabs will run in 30 seconds with the new system.
  2. Low transparency. Because the browser was executing on the other side of a SaaS system, it was annoyingly difficult to debug issues, especially in light of point 1.
  3. SauceLabs Connect issues. Connect is the program that manages the tunnel between the local tests and the remote browser. We had problems with it, behaviour was different between OS X and linux, requiring us to monkeypatch the Ruby driver to behave on both problems.
  4. We ran out of minutes… I was actually at the point of clicking on the “upgrade” button the SauceLabs account page before I remembered that we had a perfectly good Linux box sitting nearby that was more than capable of running this task.

So before succumbing and pumping quarters into the SauceLabs arcade cabinet, we decided to see how far we could get with VMs and Selenium…

Bill of Materials

The first step was actually figuring out what all the moving parts were, and what we would need in order to wire everything up.

So, we needed to:

  1. be able to run IE somewhere accessible to the VM server (obviously).
  2. control IE via Selenium, since this is the system that our current test suite is invested in, via Capybara.
  3. connect the test suite up to IE and actually run our integration tests against it.

As we already had VMs for our manual testing, it was logical to for us to start at #2 first. cough.

Controlling IE via Selenium

(Note, the following is my mental model, it may not actually be correct).

Basically, Selenium is an adapter to allow programmatic control of web browsers. So you can do anything from running your test suite against a real browser; to automatically downloading bank statements every month and uploading them into your accounting system (yeah, I was bored for an hour…).

It seems like Selenium has resolved the proliferation of different versions (1, 2, RC, etc?!) into Selenium WebDriver. And there are various different drivers, the one we are interested in here is theRemote WebDriver, which is conceptually split up into several parts referred to as the Selenium Grid:

  1. The hub is the client-side interface that the test suite connects to. It provides the ability for a client to say “I want to control IE 10” or “Give me Firefox, I don’t care what version”.
  2. The actual browsers are driven by nodes, which connect to the hub and announce what browser version they can execute. Nodes then accept requests from the hub on behalf of clients to start a browser session and pass commands to the browser.
  3. Drivers are the adapters between nodes and browsers and transform the generic Selenium commands into browser-specific activity.

To actually put this into practice, we first need:

  1. Java (get it from your friendly neighbourhood dealer)
  2. Selenium Remote WebDriver. I wasn’t able to find out the difference between selenium-java vs selenium-server vsselenium-server-standalone (and I didn’t even look at thedotnet versions).
  3. The Selenium IE driver

Once you have Java installed and have downloaded the server JAR and the IE driver, run something like this:

<code>java -jar /path/to/selenium-server.jar -Dwebdriver.ie.driver=/path/to/ie_driver.exe -browser "browserName=internet explorer,version=10"
</code>

And you should end up with a hub running on port 4444 of your VM. Point a Selenium client at it asking for IE 10, and you should get a browser window pop open. If using Capybara, this should get you there:

<code>url = "http://<windows host/ip>:4444/wd/hub"
capabilities =     Selenium::WebDriver::Remote::Capabilities.internet_explorer
capabilities.version = "10"
Capybara.register_driver :remote do |app|
  Capybara::Selenium::Driver.new(app,
    :browser => :remote,
    :url => url,
    :desired_capabilities => capabilities)
end
</code>

Virtual Explorer

We wanted to run our test suite against IE 9 and up, so we needed 3 Windows 7 VMs, with each of IE 9, 10, 11 installed. First recommendation was to go with VirtualBox, and it’s proved fine so far.

There are prebuilt images, but it appears those expire after 90 days, and it was easier to just install a real version than worry about managing this.

To actaully get the Windows installer to run, we needed to enable IO APIC in the VM settings:

To begin with, we went with a host-only adapter, but this made it difficult to use the VM on an ad-hoc basis, so since we are behind a NAT anyway, running with a bridged adapter was the easiest move.

Last step was to run the VM headless, so that we didn’t need to have an X session up and running all the time:

<code>VBoxHeadless -s <vm name or id> -n -m <port of VNC server> -o <VNC password>
</code>

-n enables a VNC server, so you can easily keep an eye on the VM (or running tests).

Connect to the grid

Once the VMs were up and running, we needed to make them available for Selenium clients. Simplest way to do this is to take advantage of Selenium’s grid and split into nodes that drive IE and a single server.

To start the server, on the CI machine, using the same Selenium JAR as downloaded onto the VM earlier:

<code>java -jar /path/to/selenium-server.jar -role hub
</code>

This will start the hub listening on all interfaces on port 4444. Then start a node on each VM that connects to the node. This requires a simple change to the

<code>java -jar /path/to/selenium.jar -Dwebdriver.ie.driver=/path/to/ie_driver.exe -hub http://<CI server hostname/IP>:4444/grid/register -browser "browserName=internet explorer,version=<IE version on this VM>"
</code>

And now you can point a Capybara client at the hub, instead of directly at the VM:

<code>url = "http://<CI server hostname/ip>:4444/wd/hub"
# ... configure Capybara same as in previous section
</code>

Jenkins + The Test Suite

The last step is to get the test suite to actually run against the browser you want. Again, two parts:

  1. Configure Capybara to use IE. We saw this above, but here’s a complete configuration, with the version selected by an environment variable that we’ll get Jenkins to pass in:
    <code>Capybara.default_driver = :remote
    url ="'http://<CI Selenium Hub>:4444/wd/hub"
    capabilities = Selenium::WebDriver::Remote::Capabilities.internet_explorer
    capabilities.version = ENV["SELENIUM_BROWSER_VERSION"]
    capabilities.javascript_enabled = true # Javascript not enabled by default, see "Gotchas" below.
    Capybara.register_driver :remote do |app|
      Capybara::Selenium::Driver.new(app,
     :browser => :remote, 
     :url => url,
     :desired_capabilities => capabilities)
    end 
    </code>
    Ensure the browser knows where Capybara’s test application is running:
    <code>Capybara.app_host = "http://<CI server hostname IP>:#{Capybara.server_port}"
    </code>
  2. Add a parameter to your Jenkins job so that you can select at run time which IE version you want to run against.

Gotchas

If you’re running a 64-bit version of Windows, do not go with the 64-bit IE driver for IE 10+. This is the source of incredibly slow typing.


IE 11 requires a registry key to be set, otherwise connections to the browser process will drop out:

For IE 11 only, you will need to set a registry entry on the target computer so that the driver can maintain a connection to the instance of Internet Explorer it creates.
For 32-bit Windows installations […],HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Internet Explorer\Main\FeatureControl\FEATURE_BFCACHE.
For 64-bit Windows installations […]HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Internet Explorer\Main\FeatureControl\FEATURE_BFCACHE.
[…] create a DWORD value named iexplore.exe with the value of 0.

Here’s a registry export snippet for 64-bit Windows. Paste this into selenium-ie11-key.reg:

<code>Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Internet Explorer\MAIN\FeatureControl\FEATURE_BFCACHE]"iexplore.exe"=dword:00000000
</code>

and double click to configure the registry setting. (Or do it by hand with regedit.exe if you’re into that kind of thing…)


By default, the Selenium WebDriver capabilities for IE do not include Javascript, so add them in:

<code>capabilities = Selenium::WebDriver::Remote::Capabilities.internet_explorer
capabilities.javascript_enabled = true</code>