Property Based Testing in Javascript

Pierre-Etienne Moreau
Pierre-Etienne MoreauApril 18, 2019
#testing#js

Introduction to Property Based Testing

To be more confident with your code, in particular in Javascript, a good practice consists in writing tests with a framework such as Jest for instance. But you still have to provide a set of inputs and expected results for each function you want to test. As a consequence, the quality of your test depends on the quality and the number of the inputs your considered.

Property based testing is a technique widely used in the Haskell community, using a tool such as QuickCheck. The idea consists in automatically generating inputs for testing a function.

To test a given function, the programmer provides a property that should be true for all possible inputs. QuickCheck (or a similar tool) randomly generates a huge number of inputs and checks that the given property is still satisfied.

Alternatives to Property Based Testing

For example, suppose you want to test the following function:

const maccarthy = n => (n > 100 ? n - 10 : maccarthy(maccarthy(n + 11)));

This function is also known as McCarthy 91 function because its value is always 91.

its value is always 91 is what we call a property. Indeed, in mathematics, this can be expressed by "forall x in N, maccarthy (x) = 91", where N denotes the set of natural integers (i.e. 0, 1, 2, ...).

To check that the value of maccarthy function is always 91, you have several possibilities:

Mathematical Proof

You can do a mathematical proof, using induction. This is the safest approach but also the hardest one. Indeed, for some different functions or for some other properties you may not be able to provide the proof. In addition, unless you are using a proof assistant such as Coq or Isabelle for instance, the proof is done using paper and pencil.

The problem of using paper and pencil is that the proof is related to the mathematical function, and not the JS implementation (i.e. the code). Therefore, the property can be proved correct but the implementation may still be wrong.

A Few Unit Tests

You can write several tests using a testing framework such as Jest for instance:

  test('MacCarthy is equal to 91 for n=10', () => {
      expect(maccarthy(10)).toEqual(91);
  });
  ...
  test('MacCarthy is equal to 91 for n=20', () => {
      expect(maccarthy(20)).toEqual(91);
  });

This is a very interesting approach, but to be useful, the tested values should be chosen carefully.

Testing Almost All Possible Values

You can test the output of the maccarthy function for almost all possible values:

test("MacCarthy is equal to 91 for all n", () => {
  for (let i = 0; i < 10000; i++) {
    expect(maccarthy(i)).toEqual(91);
  }
});

In the current situation, this is an interesting approach because it is very easy to enumerate a huge number of integers (using a for-loop construct).

Furthermore, this approach allows to find a bug:

MacCarthy is equal to 91 for all n

  expect(received).toEqual(expected)

  Expected value to equal:
    91
  Received:
    92

Indeed, the maccarthy function is equal to 91, not for all possible integers, but for positive integers smaller than 101.

McCarthy 91 function

Writing tests can help us to find bugs in the implementations but it can also help us to find errors in the specification (i.e. the properties that we think to be true).

In this example, this is exactly what happened: we wanted to check that

forall x in N, maccarthy(x) = 91

But this property is not true. The correct property is

forall x in N, such that x <= 101, maccarthy(x) = 91

Property Based Testing in Practice

In JavaScript, you can use a Property Based Testing library such as fast-check:

test("MacCarthy is equal to 91 forall n", () => {
  fc.assert(
    fc.property(fc.nat(), x => {
      expect(maccarthy(x)).toEqual(91);
    })
  );
});

This library provides several constructs to express a mathematical property directly in JS:

  • fc.nat() is used to randomly generate positive integers. This corresponds to the forall x in N part.
  • fc.property is used to encode the property to check.

Running this code also allows to find the counter-example to our property:

MacCarthy is equal to 91 for all n

  Property failed after 1 tests
  { seed: 364196543, path: "0:1:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:3:4" }
  Counterexample: [102]
  Shrunk 26 time(s)
  Got error: Error: expect(received).toEqual(expected)

  Expected value to equal:
    91
  Received:
    92

The system automatically generated several integers, and found a counter-example, which has been shrunk to provide a simple counter-example: 102.

What Did We Learn So Far?

In order to check the correctness of an implementation, it is obvious that we have to express the properties which have to be satisfied.

But, the previous example showed us that expressing such properties is not easy: properties may be incorrectly specified.

Property

A Property Based Testing using library such as fast-check is an interesting alternative to exhaustive enumeration. It allows to find bugs both in the implementation code and in the specification of the property we want to check. In both cases this helps the developer to get confident in the product he is developing.

Existing Libraries for JS

I have found 5 libraries that offer primitives to perform Property Based Testing:

  • fast-check, developed by Nicolas Dubien since 2017
  • jsverify, developed by Oleg Grenrus since 2014
  • testcheck, developed by Lee Byron since 2014
  • jscheck, developed by Douglas Crockford since 2013
  • quick_check.js, developed by Jakub Hampl since 2014

According to npmtrends jsverify is the most popular today but testcheck used to be very popular in 2017.

After reading the documentations I decided to start with testcheck which has a nice documentation and appeared to be easy to use. The first experiment went smoothly, but to test functions which take objects as arguments I had the need of generating randomly built objects. For this purpose testcheck provides a gen() function. For instance, the expression gen({ x: 'foo', y: gen.int, z: gen.boolean }) should generate objects such as { x: 'foo', y: -4, z: false }. Unfortunately I did not manage to use this function gen(). Digging GitHub, I found the following issue: "From what I can tell, the documentation on the website seems to describe a version that hasn't been formally released yet."

So I decided to give a try to jsverify. It is also quite easy to use. Unfortunately while I was preparing this article, I wrote the following piece of code and I got a surprise: no counter example has been found for MacCarthy function!

it("MacCarthy is equal to 91 for all n", () => {
  expect(jsc.checkForall(jsc.integer(), x => maccarthy(x) === 91)).toBeTruthy();
});

The function jsc.integer() generates both positive and negative integers, but it generates small integers such that none of them is higher than 100, and thus no counter-example has been found. If you replace jsc.integer() by jsc.integer(200), then it finds a counter-example:

 console.error node_modules/jsverify/lib/jsverify.js:325
      Failed after 1 tests and 2 shrinks. rngState: 86816de7f52acb0493; Counterexample: 102;  [ 102 ]

But this is a pity that the default function (i.e. without an upper limit) does not generate large integers.

The last library I experimented was fast-check. As illustrated in the first section of this article, it immediately found the error in the specification:

MacCarthy is equal to 91 for all n

  Property failed after 1 tests
  { seed: 364196543, path: "0:1:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:3:4" }
  Counterexample: [102]
  Shrunk 26 time(s)
  Got error: Error: expect(received).toEqual(expected)

  Expected value to equal:
    91
  Received:
    92

Using Property Based Testing In Frontend Development

As a web developer I tried to use fast-check to test React components. My idea was to use fast-check to generalize the tests we usually do when writing React applications.

For example, in the context of React Admin, an Open Source frontend framework developed by Marmelab, several unit-tests are written using enzyme. Let us consider CreateController.spec.tsx for instance. In this example, we test the CreateController component by setting a search prop and checking that a record object has been constructed during the rendering phase:

it("should return location search when set", () => {
  const childrenMock = jest.fn();
  const props = {
    ...defaultProps,
    children: childrenMock,
    location: {
      ...defaultProps.location,
      search: "?foo=baz&array[]=1&array[]=2",
    },
  };

  shallow(<CreateController {...props} />);
  expect(childrenMock).toHaveBeenCalledWith(
    expect.objectContaining({
      record: { foo: "baz", array: ["1", "2"] },
    })
  );
});

Using fast-check, we can generalize this test by automatically generating a huge number of values for the search prop.

For that, we have to generate strings which conform to a given syntax: they should correspond to a query string (i.e. the part of a URL which assigns values to specified parameters). Unfortunately this is not so easy to generate strings of the form ?foo=baz&array[]=1&array[]=2 unless you have a generative grammar for them.

A simpler approach consists in generating the objects expected as results (i.e. a record of the form { foo: 'baz', array: ['1', '2'] }) and converting them back to strings using the stringify function.

This is exactly what I have done in the following test:

it("should return location search when set", () => {
  const childrenMock = jest.fn();
  fc.assert(
    fc.property(queryParamsArbitrary, params => {
      const searchString = stringify(params, {
        arrayFormat: "bracket",
      });

      const props = {
        ...defaultProps,
        children: childrenMock,
        location: {
          ...defaultProps.location,
          search: searchString,
        },
      };

      shallow(<CreateController {...props} />);
      expect(childrenMock).toHaveBeenCalledWith(
        expect.objectContaining({
          record: params,
        })
      );
    })
  );
});

As you can see, the generation of records is hidden behind the queryParamsArbitrary expression. Indeed, in addition to predefined generators such as fc.nat(), fc.string(), etc., fast-check offers the possibility to define custom generators, called arbitraries in fast-check terminology.

This is exactly what we did for record objects.

const queryParamsArbitrary = fc.dictionary(
  fc.fullUnicodeString(1, 10),
  fc.oneof(
    fc.asciiString(),
    fc.constant(null),
    fc.array(fc.oneof(fc.asciiString()), 2, 10)
  )
);

This arbitrary generates inputs such as: {}, { '񳡭': null }, or { '򗚚񶐇򪀐􂉰񇧎': '"%$!\n\u0014# ', '򀶪ᦝ񜏣󐄧򎧲𛌆𷀨󜑘𷮵狜': ':!' }.

But more complex objects with arrays may also be generated:

 { '𘐨򙼙񌏪':
         [ 'jT0CNO\u000e+cJ',
           '~\u001d6\b',
           '\u0005M{$f',
           '\u000e',
           'D\u0012\u0010C\u000f\u0000`',
           'x\rNP',
           '$\u00191\u0015\u001c\n',
           '\u0004s~\u0017',
           '\u0017\u001a\u00108' ],
        '󆯳': null,
        '򷎒䘾􎐷': null,
        '񼉖𢄠􉥷򥘂񋏠':
         [ '\u0019?',
           'y\n\u0013',
           '\u00159K\u0018\'BYn',
           '',
           '\n{>G#e\u0014W\u0015\\',
           '',
           '\t' ],
        '򯹳񫝄񢋃򊲫𞉖򪃙򌨓񆔬🈏񮳉':
         [ '\u0001\u0010+',
           '*s\u001bcVJ\u000f',
           '\u0015,giy`p3q',
           'J{^,U} ',
           'b\u0013e*',
           '\u000b\u0014^\u001e',
           'v9Tm+\u00061V',
           'c\rW!#-',
           ')\u001em*tH\f9B' ],
        '򜫿􃃘󐱢󟪦򰘌𡡶򌴙󖋣򚼑񊇎': 'i\u001b' }

Unfortunately the use of fast-check did not help us to find any bug in the React component. But this is a positive result since we are now more confident in our code, thanks to the large number of complex inputs which have been tested.

Fast-check has been successfully used on another project (Limonade), and it helped us find bugs which would have been quite difficult to find without such an exhaustive approach.

Conclusion

Fast-check is a nice library which is very easy to setup in a JavaScript project. It can be effectively used to find bugs in the implementation or the specification. The provided arbitraries and shrunk mechanisms are quite effective in practice since they help the programmer, when a bug is found, by providing a minimal example.

However, for web development, and especially for frontend development, the code is more presentational than algorithmic, and therefore the benefit of Property Based Testing as compared to classic unit testing is probably less obvious. We have probably not yet found the use cases where it would make a big difference, but we still intend to use it from time to time in the future.

Did you like this article? Share it!