Automating accessibility testing with Selenium Webdriver and AxeCore

Gildas Garcia
Gildas GarciaFebruary 22, 2018
#accessibility#testing#popular

To make sure your site is accessible, you'd better add a check for accessibility in your continuous integration process. Here is how.

The Goal: Accessibility Tests in CI

I'm going to setup automatic end to end tests in a React application created with create-react-app. These tests will be run by a Continous Integration (CI) server - in my case, Travis.

At marmelab, we believe end to end tests should be ran against a production-like version of the application. This mean I'll have to build the CRA application using its dedicated command, yarn build.

The Tools: Mocha, Selenium, Axe

In order to automate my tests, I need a test runner. I'll be using mocha here as it's easier for end to end tests. I'm aware that there are solutions to write e2e tests with my favorite runner, Jest, but I haven't tried it yet.

At marmelab, we usually write our e2e tests with selenium webdriver. We already wrote some articles about it:

With the recent updates on Chrome and Chromedriver, I can even make my tests headless: I'll never see the actual browser running the tests, it will all happen without GUI. Running end to end tests with selenium requires a selenium server instance. I usually setup a testing environment with Docker and docker-compose, but I'll explore another way here, using selenium-standalone to start the selenium instance, and serve to expose the application built in production-like mode.

For accessibility, I chose to use tools from the Axe ecosystem, edited by Dequelabs. They open sourced many very useful tools!

The devtools are invaluable in the coding phase, but with axe-core and axe-webdriverjs, I can actually enforce accessibility of our apps with automated tests!

The Process: Test Setup

Now that I know what to use and why, let's install the tools:

yarn create react-app accessibility
cd accessibility
yarn add --dev babel-register babel-polyfill mocha selenium-webdriver@3.6.0 selenium-standalone serve axe-webdriverjs

I installed babel-register and babel-polyfill so that I can write our tests with es6 and async/await :)

I created an e2e folder, which will contain all the test related code.

The first thing I need to do is to setup the Selenium instance. I'll use mocha before and after hooks so that this setup runs before running any tests, and to clean up after they end.

// in e2e/helpers.js
import selenium from "selenium-standalone";

let seleniumInstance;

before(async () => {
  await new Promise((resolve, reject) => {
    // Ensure selenium and default chromedriver are installed
    selenium.install(error => (error ? reject(error) : resolve()));
  });

  seleniumInstance = await new Promise((resolve, reject) => {
    selenium.start((error, instance) =>
      error ? reject(error) : resolve(instance)
    );
  });
});

after(async () => {
  seleniumInstance && seleniumInstance.kill();
});

I also need to start a server to serve my application (which will be built using yarn build beforehand):

// in e2e/helpers.js
...
import serve from 'serve';
import path from 'path';

let server;

before(async () => {
    ...
    server = serve(path.join(__dirname, '..', 'build'), {
        port: 8080,
    });
});

after(async () => {
    ...
    server && server.stop();
});

Finally, I need a way to get a selenium webdriver instance inside the tests. This instance must be built after the selenium server instance is initialized, so I need a function to get or initialize the driver.

// in e2e/helpers.js
...
let driver;

export const getDriver = () => {
    if (driver) {
        return driver;
    }

    driver = new Builder()
        .forBrowser('chrome')
        .usingServer('http://localhost:4444/wd/hub')
        .build();

    driver
        .manage()
        .window()
        .setSize(1280, 1024);

    return driver;
};

after(async () => {
    ...
    driver && driver.quit();
});

Testing The Test Setup

Everything should be in place, let's test if it works!

// in e2e/home.test.js
import assert from "assert";
import { By, until } from "selenium-webdriver";
import { getDriver } from "./helpers";

describe("Home page", () => {
  let driver;

  before(() => {
    driver = getDriver();
  });

  it("has no accessibility issues", async () => {
    await driver.get(`http://localhost:3000`);

    // Wait until our content is visible, here we just wait for title
    await driver.wait(until.elementLocated(By.css("h1")));

    const title = await driver.findElement(By.css("h1")).getText();
    assert.equal(title, "Welcome to React");
  });
});

I can add a custom script in our package.json to run the tests:

{
    ...
    "scripts": {
        "start": "react-scripts start",
        "build": "react-scripts build",
        "test": "react-scripts test --env=jsdom",
        "eject": "react-scripts eject",
        "test-e2e": "NODE_ENV=test mocha --require babel-register --require babel-polyfill --timeout 20000 e2e/**/*.test.js"
    },
    ...
}

Now, I can run the tests with the following command:

yarn build && yarn test-e2e

Great!

Accessibility Tests With Axe

Let's create some helpers to ease accessibility testing:

// in e2e/helpers.js
...
import axe from 'axe-webdriverjs';
...

export const analyzeAccessibility = () =>
    new Promise(resolve => {
        axe(driver).analyze(results => resolve(results));
    });

I then use this helper to test accessibility in the page:

// in e2e/home.test.js
import assert from "assert";
import { By, until } from "selenium-webdriver";
import { getDriver, analyzeAccessibility } from "./helpers";

describe("Home page", () => {
  let driver;

  before(() => {
    driver = getDriver();
  });

  it("has no accessibility issues", async () => {
    await driver.get(`http://localhost:3000`);

    // Wait until our content is visible, here we just wait for the title
    await driver.wait(until.elementLocated(By.css("h1")));

    const results = await analyzeAccessibility();
    assert.equal(results.violations.length, 0);
  });
});

If I run the test command, I get this:

Ok, my tests are failing but I currently don't why.

Debugging Tests

Let's make this better:

// in e2e/home.test.js
...

describe('Home page', () => {
    ...
    it('has no accessibility issues', async () => {
        await driver.get(`http://localhost:3000`);

        // Wait until our content is visible, here we just wait for title
        await driver.wait(until.elementLocated(By.css('h1')));

        const results = await analyzeAccessibility();
        assert.equal(results.violations.length, 0, JSON.stringify(results.violations, null, 4));
    });
});

That's better than no information but it's difficult to read ! Let's make it even better:

// in e2e/helpers.js
...

export const formatAccessibilityViolations = violations => {
    const messages = violations.map(
        violation =>
            `\r\n- ${violation.help} (${violation.nodes.length} elements affected)
            \r  Help: ${violation.helpUrl}\r\n`,
    );
    return `${violations.length} violations found: ${messages.join()}`;
};

// in e2e/home.test.js
...
import {
    driver,
    analyzeAccessibility,
    formatAccessibilityViolations,
} from './helpers';
...

describe('Home page', () => {
    ...
    it('has no accessibility issues', async () => {
        await driver.get(`http://localhost:3000`);

        // Wait until our content is visible, here we just wait for title
        await driver.wait(until.elementLocated(By.css('h1')));

        const results = await analyzeAccessibility();
        assert.equal(
            results.violations.length,
            0,
            formatAccessibilityViolations(results.violations)
        );
    });
});

Which give me:

Now that's helpful without being overhelming. Great!

Fixing Accessibility Bugs

Let's fix the issue:

// in src/App.js
import React, { Component } from "react";
import logo from "./logo.svg";
import "./App.css";

class App extends Component {
  render() {
    return (
      <div className="App">
        <header className="App-header">
          <img src={logo} className="App-logo" alt="logo" />
          <h1 className="App-title">Welcome to React</h1>
        </header>

        {/* Replace the p with a main element */}
        <main className="App-intro">
          To get started, edit <code>src/App.js</code> and save to reload.
        </main>
      </div>
    );
  }
}

export default App;

And the tests are green!

Configuring The Continous Integration Server

The last thing I need to do is to configure Travis:

# in .travis.yml

# sudo is required to install selenium and the chromedriver correctly
sudo: required
language: node_js
node_js:
  - "8"

# This addon ensure we run the latest stable chrome which support headless mode
addons:
  chrome: stable

script:
  - yarn build && yarn test-e2e
cache:
  directories:
    - node_modules
    # Cache the selenium server installation (this is its default path)
    - e2e/selenium
    - .cache

Now, each time someone opens a Pull Request on that project, their code will be tested for accessibility automatically!

Configuring Accessibility Tests

Sometimes I don't want to validate all the default rules set by axe-core. For example, on one of our projects, those tests failed because of contrast issues, but it was necessary to completely review the site design to fix those issues. As it was not possible immediately, we temporarily disabled contrast related rules:

export const analyzeAccessibility = () =>
  new Promise(resolve => {
    axe(driver)
      .disableRules(["color-contrast"])
      .analyze(results => resolve(results));
  });

Moreover, you might be using some third party widgets in your app, and you probably don't want to run accessibility tests on them. Just use the exclude method, which accepts CSS selectors:

export const analyzeAccessibility = () =>
  new Promise(resolve => {
    axe(driver)
      .exclude(".disqus")
      .analyze(results => resolve(results));
  });

Conclusion

I hope this process will help you enforce accessibility in your apps. You can find the example application in this repository: https://github.com/marmelab/cra-e2e-accessibility

Please note that this is only a part of what accessibility really means. Our users may not only be impacted by disabilities but their income, their location, their reading level and many other things can impact them as well.

At the dotJS conference last year, I saw not only one, but three great talks about accessibility, by three awesome speakers. If you haven't already, I strongly encourage you to watch them:

Happy accessibility testing!

Did you like this article? Share it!