Get Rid Of Toxic Bugs On Your Apps With Detox!

During the React Europe Conference last year, I've discovered a promising E2E testing framework for mobile apps called Detox. Since then, I've kept in mind the will to give it a try. Recently, I developed an example app just for that purpose. It allowed me to see what's under the hood of Detox. Here is my feedback.

What Are End-to-End Tests?

In software development, we list 3 distinct kinds of automated tests:

• Unit Tests: As their name suggest, they test functions individually, in isolation from the rest of the codebase. They're used to prevent unexpected code changes and to ensure that functions do what they are supposed to do.
• Integration Tests (or Service Tests) are responsible for the proper connection between code parts and APIs. They test the application components altogether from a technical perspective.
• End-To-End Tests (E2E): They allow to test the application as a whole, in its execution environment, as a human could do.

According to Martin Fowler, all these kinds of tests can be classified into a Test Pyramid from the slowest / most expensive to the fastest / least expensive.

At the bottom of the pyramid, Unit-Tests must be the most common tests. Utopically, each function must be tested. Some integration tests and a bit less E2E tests are needed to ensure that the entire stack is working well.

Whereas E2E tests are very important, some people sometimes go too far with an excessive E2E test coverage. Another diagram called Ice-Cream Cone represents this anti-pattern as well.

But everyone agrees that writing and debugging E2E tests is a tedious task.

Introducing Detox

Detox was first released in 2016 by Tal Kol and Rotem Mizrachi-Meidan, 2 engineers working at Wix. Wix is a cloud-based platform which allows non-technical users to create their own website.

Detox defines itself as a Gray Box End-To-End testing automation framework for mobile apps. That means that it brings the same context aware test capabilities we're already using in browser apps through Selenium. This way, Detox allows to break off from manual Quality Insurance test processes, which are time-consuming and incomplete.

Contrary to Appium, its the main competitor, Detox uses JavaScript both on the server side and on the client side. Despite this strong requirement, Detox allows using Jest, Mocha, AVA, or any other JavaScript test runner you like.

Gray Box Testing vs Black Box Testing

As a Gray Box test framework, Detox shares both White and Black Box capabilities. Let's see together what it means.

Black Box test frameworks allows to take over an execution context (a browser, a software, mobile apps, etc) and send control commands to them.

This test methodology does not allow to access to the internal state of the application though. That's why it's necessary to manually check the existence of elements to ensure test the state after a transition.

function* navigate() {
  yield driver.navigate().to(`http://localhost/#/login`);
  yield driver.wait(until.elementLocated(By.css("#loginform")));
}

Gray Box frameworks are extending White Box test frameworks capabilities. This way, they do the same thing as Black Box frameworks, except that they access the internal state of the execution context.

Accessing the internal state of the execution context permits to know when the application is idle, and to synchronize operations adequately. That's why Detox is more powerful than most classic E2E test frameworks.

Less Flakiness

If you've already used an E2E test framework before, you've certainly encountered some strange, random and unexpected errors. These errors are therefore called "flakiness errors". As you encounter them, you feel like our good old Harold and it's not very funny.

To mitigate this behavior, we usually add some sleep (or timeouts) calls into the test suite, to ensure that the application is in an idle state before resuming the test process. Even though this "hack" works, it results in slower tests, without really solving the problem because on a slow test system, the sleep delay can sometimes be not enough.

function* login() {
  yield driver.findElement(this.elements.loginButton).click();
  yield driver.sleep(5000);
}

Thankfully, as a Grey Box Framework, Detox is able to access the application state and then to determine if the application is in an idle state or not. To achieve this idle synchronization task, Detox rely on 2 natives Grey Box drivers called EarlGrey (for iOS) and Espresso (for Android).

Because Detox runs in JavaScript, it communicates with drivers using a JSON based protocol to invoke control commands on devices.

A Concrete Use Case

As already said in the introduction, I've developed a dedicated application to give Detox a try. Since I'm a beer lover, I couldn't resist creating a simple beer registry app called beerexplorer.

Detox Setup

I've tried to setup Detox to run it on my own Android phone. Despite all my efforts, I've not been able to make it work. So I went back to an iOS emulator.

The Detox setup is relatively simple. It consists of installing the detox npm package, then calling 2 commands: detox build and detox test.

Then, Detox uses an existing configuration defined in package.json to determine which test runner and configuration it should use. All available devices configurations are stored under the "detox.configurations" key. Android and iOS device configurations can be mixed.

/* package.json */

{
  "name": "beerexplorer",
  "detox": {
    "test-runner": "jest",
    "runner-config": "e2e/config.json",
    "configurations": {
      "ios.sim.debug": {
        "binaryPath": "ios/build/Build/Products/Debug-iphonesimulator/beerexplorer.app",
        "build": "xcodebuild -project ios/beerexplorer.xcodeproj -scheme beerexplorer -configuration Debug -sdk iphonesimulator -derivedDataPath ios/build",
        "type": "ios.simulator",
        "name": "iPhone 7"
      }
    }
  }
}

When I call the detox test command, Detox looks in the runner-config configuration file for the setupTestFrameworkScriptFile to execute before running tests. I've called this file init.js.

// e2e/config.json

{
    "setupTestFrameworkScriptFile" : "./init.js"
}

Here is the test init file:

// e2e/init.js

const detox = require("detox");
const config = require("../package.json").detox;

beforeAll(async () => {
  await detox.init(config);
});

afterAll(async () => {
  await detox.cleanup();
});

Tests can either run on a local emulator, a hidden emulator, or even on a distant CI like Travis!

Detox Usage

Out of the box, Detox provides a small but powerful set of tools, that allows to control the device, select elements in the UI, and execute actions on these elements.

Device

The device object allows to control the device directly, without relying on the tested application. Here are some examples of usages from the documentation.

// Launch app with specific permissions
await device.launchApp({ permissions: { calendar: "YES" } });

// Simulate "home" button click
await device.sendToHome();

// Simulate geolocation
await device.setLocation(32.0853, 34.7818);

Some device functions are specific to a given platform, such as device.reloadReactNative for React-Native and device.shake for iOS.

Selectors / Matchers

As with other test frameworks, Detox gives the possibility to match UI elements in different ways.

The easiest (and recommended) way to match elements is to use ids. Sadly, this technique is only available on React-Native.

// id declaration
<Touchable testID="BeerListItem">...</Touchable>;

// element selection
await element(by.id("BeerListItem"));

It's also possible to match elements with other methods like text, label, type or traits. More informations on the corresponding matchers documentation.

Actions And Expectation

Once selected, it's possible to trigger actions and execute assertions on elements. As an example, here is a test suite from the homepage of the "beerexplorer" project.

describe("home", () => {
  beforeEach(async () => {
    await device.reloadReactNative();
  });

  it("should have a list of beers", async () => {
    await expect(element(by.id("BeerList"))).toBeVisible();
  });

  it("should go to detail on beer touch", async () => {
    await element(by.id("BeerListItem"))
      .atIndex(0)
      .tap();
    await expect(element(by.id("DetailBackground"))).toBeVisible();
  });

  it("should show all beers", async () => {
    await waitFor(element(by.label("Lindemans Kriek")))
      .toExist()
      .whileElement(by.id("BeerList"))
      .scroll(50, "down");
    await expect(element(by.label("Lindemans Kriek"))).toExist();
  });
});

As you can see, tests are very expressive and easy to read. There's no need to add more test about the existence of an element between transitions, thanks to the idle state synchronization.

Conclusion

Although satisfied by Detox at the end, I'm still disappointed by the difficulty of setting up an E2E test suite on Android. Because of my poor experience on mobile application tests, I don't pretend to give you the more accurate opinion. But I still think this framework (and its documentation) still to be improved for Android.

Apart from that, the developer experience with Detox is very pleasant. I never found myself in difficulty when writing tests. Also, live preview in the emulator is very empowering.

Nevertheless, If you're testing your application on iOS only, feel free to give it a try. You won't take many risks, except to be greatly satisfied by the clarity and the stability of the tests.

If you want to read more on the subject by other authors, I recommend the following:

• The Practical Test Pyramid
• Detox: Gray Box End to End Testing Framework for Mobile Apps
• Testing in React-Native - Jest & Detox
• Gray Box vs Black Box