Modern web applications usually contain many moving parts and third-party dependencies. While refactoring, adding or changing the functionality we can break existing use-case scenarios and stability in some browsers.

In order to detect and prevent these situations and perform continuous integration one needs to perform functional testing. In this article we’ll talk about two open-source solutions:

• https://devexpress.github.io/testcafe
• http://nightwatchjs.org

At the first glance these solutions provide us the same set of features:

• Automated functional testing with serial and parallel execution of tests, their grouping and integration with Gulp and Mocha.
• Support for the most of desktop and mobile browsers; interaction with real DOM API (web-application opens in browser’s tab, to make sure that testing environment is very close to the real-world conditions).
• Different ways to select DOM-elements: document.querySelector, CSS-selectors and even XPath; the combination of these variants helps us to make functional tests resistant to markup changes.
• visual interactions (click on the button, input to the text field); useful and simple tools to handle async-operations.
• iframe-support. One can easily change the scope (DOMWindow) for functional tests in any desired moment.
• Enhancing abilities. Both tools provides us different possibilities to add custom browsers or to extend base functionality, including adding new commands.

Example of a functional test

Our patient for testing will be a single-page application (similar to TodoMVC) with React + Redux for the frontend and NodeJS for the backend. To approximate testing conditions to reality every Redux action contains a small delay to emulate network interactions with back-end.

Out testing web application is running at http://localhost:4000. The functional test is quite simple and includes the following scenarios:

• adding a new todo-item
• editing an existing todo-item
• do/undo operations
• deleting a todo-item

In both frameworks we use JS (ES2016 for TestCafe and ES5 for NightWatch), but it doesn’t mean that we can’t test web applications written in different languages. If you have not been using JS for a long time you should consider that current versions went far away from the old ES3 and include convenient tools to write object-oriented or functional code.

TestCafe

The installation process of TestCafe consists of the single command npm install -g testcafe. To start the tests we need to run testcafe <browser-name> <tests-directory>/ in the corresponding directory.

The source code for a functional test, which verifies the scenarios outlined above, could look like this:

import { expect } from 'chai';
import { Selector } from 'testcafe';

const MAX_TIME_AJAX_WAIT = 2500; // 2.5 seconds by default
const querySelector = Selector((val) => document.querySelector(val), {
  timeout: MAX_TIME_AJAX_WAIT
});

const querySelectorCondition = Selector((val, checkFunc) => {
  const foundElems = document.querySelectorAll(val);
  if(!foundElems) return null;
  for(let i=0; i < foundElems.length; i++) {
    if(checkFunc(foundElems[i])) return foundElems[i];
  }
  return null;
}, {
  timeout: MAX_TIME_AJAX_WAIT
});

fixture `Example page`
  .page `http://localhost:4000/`;

test('Emulate user actions and perform a verification', async t => {
  await t.setNativeDialogHandler(() => true);
  const inputNewTodo = await querySelector('header.header input.new-todo');
  await t.typeText(inputNewTodo, 'New TODO element\r\n');

  const addedTodoElement = await querySelectorCondition(
    'section.main label',
    (elm) => (elm.innerText === 'New TODO element')
  );

  await t.doubleClick(addedTodoElement);

  const addedTodoEditInput = await querySelectorCondition(
    'section.main input[type=text]',
    (elm) => (elm.value === 'New TODO element')
  );

  await t.typeText(addedTodoEditInput, ' changed\r\n');

  const addedTodoCheckboxAC = await querySelectorCondition(
    'section.main input[type=checkbox]:not([checked])',
    (elm) => (elm.nextSibling.innerText === 'New TODO element changed')
  );

  await t.click(addedTodoCheckboxAC);

  const addedTodoCheckboxBC = await querySelectorCondition(
    'section.main input[type=checkbox]',
    (elm) => (elm.nextSibling.innerText === 'New TODO element changed')
  );

  await t.click(addedTodoCheckboxBC);

  const addedTodoDelBtn = await querySelectorCondition(
    'button.destroy',
    (elm) => (elm.previousSibling.innerText === 'New TODO element changed')
  );

  await t.click(addedTodoDelBtn);
});

The address of the web page under test is defined in the fixture section followed by the functional tests. When all tests are finished the web page is automatically restored to initial state. TestCafe provides a set of specific selectors (wrappers) to search for DOM-elements on the page that wrap native functions to access DOM.

In the first example, the selector function is a wrapper for document.querySelector and in the second one it’s a wrapper for document.querySelectorAll with a callback function that helps us to pick up the correct item from the list. Every wrapper takes few parameters, one of them is a maximum time for TestCafe to wait for a specific element in DOM.

A functional test is a set of asynchronous calls of selectors and actions of test-controller (defined as variable t). Most of the methods are really simple to understand (click, typeText etc …). As for t.setNavitDialogHandler, it is used to prevent showing potential dangerous native windows which could ‘hang up’ the test, for instance: alert, confirm etc …

NightWatch

The installation process of NightWatch also starts with a simple command npm install -g nightwatch. But to run our tests we also need to install Selenium server (you can install it here) and web drivers for each browser, in which we want to run the tests.

Before running the tests we need to create a configuration file nightwatch.json and specify the location of tests, path and settings for the Selenium server, web drivers etc. When it’s done, we can run the test with a simple command nightwatch in the current directory. nightwatch.json could look like this for the Microsoft Web Driver (Edge):

{
  "src_folders" : ["nightwatch-tests"],
  "output_folder" : "reports",
  "custom_commands_path" : "",
  "custom_assertions_path" : "",
  "page_objects_path" : "",
  "globals_path" : "",
  "selenium" : {
    "start_process" : true,
    "server_path" : "nightwatch-bin/selenium-server-standalone-3.0.1.jar",
    "log_path" : "",
    "port" : 4444,
    "cli_args" : {
      "webdriver.edge.driver" : "nightwatch-bin/MicrosoftWebDriver.exe"
    }
  },
  "test_settings" : {
    "default" : {
      "selenium_port"  : 4444,
      "selenium_host"  : "localhost",
      "desiredCapabilities": {
        "browserName": "MicrosoftEdge",
        "acceptSslCerts": true
      }
    }
  }
}

With NightWatch the same functional test could be defined as follows:

module.exports = {
  'Demo test' : function (client) {
    client.url('http://localhost:4000')
      .waitForElementVisible('body', 1000)

      // May use CSS selectors for element search
      .waitForElementVisible('header.header input.new-todo', 1000)
      .setValue('header.header input.new-todo', ['New TODO element', client.Keys.ENTER])

      // Or use Xpath - it's more powerful tool
      .useXpath()
      .waitForElementVisible('//section[@class=\'main\']//label[text()=\'New TODO element\']', 2000)
      .execute(function() {
        // To dispatch a double click - NightWatch doesn't support it by default
        var evt = new MouseEvent('dblclick', {'view': window, 'bubbles': true,'cancelable': true});
        var foundElems = document.querySelectorAll('section.main label');

        if(!foundElems) return;
        var elm = null;

        for(var i=0; i < foundElems.length; i++) {
          if(foundElems[i].innerText === 'New TODO element') {
            elm = foundElems[i];
            break;
          }
        }

        elm.dispatchEvent(evt);
      })

      .waitForElementVisible('//section[@class=\'main\']//input[@type=\'text\']', 2000)
      .clearValue('//section[@class=\'main\']//input[@type=\'text\']')
      .setValue('//section[@class=\'main\']//input[@type=\'text\']',
        ['New TODO element changed', client.Keys.ENTER]
      )

      .waitForElementVisible(
        '//section[@class=\'main\']//label[text()=\'New TODO element changed\']' +
          '/preceding-sibling::input[@type=\'checkbox\' and not(@checked)]',
        2000
      )
      .click(
        '//section[@class=\'main\']//label[text()=\'New TODO element changed\']' +
          '/preceding-sibling::input[@type=\'checkbox\' and not(@checked)]'
      )

      .waitForElementVisible(
        '//section[@class=\'main\']//label[text()=\'New TODO element changed\']' +
          '/preceding-sibling::input[@type=\'checkbox\']',
        2000
      )
      .click(
        '//section[@class=\'main\']//label[text()=\'New TODO element changed\']' +
          '/preceding-sibling::input[@type=\'checkbox\']'
      )

      .waitForElementVisible(
        '//section[@class=\'main\']//label[text()=\'New TODO element changed\']' +
          '/following-sibling::button[@class=\'destroy\']',
        2000
      )
      .click(
        '//section[@class=\'main\']//label[text()=\'New TODO element changed\']' +
          '/following-sibling::button[@class=\'destroy\']'
      )

      .waitForElementNotPresent(
        '//section[@class=\'main\']//label[text()=\'New TODO element changed\']',
        2000
      )

      .pause(2000)
      .end();
    }
}

Among all the peculiarities of code you can notice that Nightwatch doesn’t support an emulation of double clicks; instead, you should dispatch an event to the target elements using dispatchEvent.

A great advantage of NightWatch is the support of XPath expressions which provide significantly broader abilities to select DOM-elements similarly to CSS-selectors (for example, finding an element by its text content).

Comparing TestCafe vs. NightWatch

Installation

[T]estCafe: One command npm install -g testcafe is enough to start writing and running the tests. [N]ightwatch: Even considering the fact that installation of the npm module is a simple npm install -g nightwatch, you still need to install a lot of additional software (selenium-standalone-server, web driver, Java) and manually create a configuration file.

Selecting DOM-elements

[T]: You’re given selector functions (wrappers over the native ones).
[N]: You have an option to choose between CSS-selectors and XPath.

Language

[T]: Out of the box, you have an ability to write tests using ES2016 that allows writing the tests in a simple and readable manner in the same language as your application code probably is. This is useful when you need to import a specific module from the application.
[N]: The obsolete ES5-syntax and module.exports-constructions in the source code of functional tests (with tricks one can still use ES6).

Async operations support

[T]: All API functions are using Promises, so you can easily describe an asynchronous logic or integrate native functions from NodeJS. Thanks to ES2016 support this code could be written in the sequential style using async / await keywords.
[N]: In the test you can implement a sequence of commands to analyze and manage the content of a web page, but they will be added to internal event queue, and integration of your own asynchronous functions could be a bit problematic.

Adding client-side code on the fly

[T]: One can fairly simply import new code modules using provided API. There is a support to create and execute custom functions, inject code or replace existing functions in the web page. In addition to that, you can execute custom functions in callbacks of NodeJS native functions with an ability to access the test controller.
[N]: There is a functionality to execute JS code on the client-side, or even to inject the whole script-block, but there is no way to integrate it with the test controller. It’s suitable for simple synchronous code, but in general the integration is problematic.

Assertions and expectations

[T]: By default, it’s chai/expect, but one can use any other compatible assertion library.
[N]: Extended language of assert and expect, including the options to describe:

• the state of the page
• presence of the element
• focus on the element
• whether or not an element belongs to specific CSS-class etc.
  It looks convenient but it’s primarily caused by the lack of full support for asynchronous operations in the test.

Managing the web page under testing

[T]: Allows mocking “dangerous” native functions that could block a web page, e.g: alert, confirm and so on, with an ability to specify the returning value. TestCafe supports complex DOM-manipulations and emulation of user interaction with controls.
[N]: Supports commands to control both BOM and DOM. Integration with client scripts is complicated.

Mouse

[T]: The virtual cursor is available, by means of which we can trigger hover, click and drag events on elements. During the test you can monitor the movement of the cursor and perform tests.
[N]: In order to work with the mouse cursor you need to call a function from the web driver’s API, but it’s way too complicated to emulate something more than just left mouse click (take at least doubleclick event as an example).

Interaction with the browser

NightWatch is based on the well-known, in some way, even traditional Selenium web driver library with the following benefits:

• well-established API
• high documentation coverage
• an extensive Q&A on the Internet
• universal and fairly low-level access to the browser

Interacting with browsers is made possible by using web drivers (which are browser-specific) with the following drawbacks:

• web drivers don’t exist for all browsers
• a web driver requires a separate update once a browser gets updated.
• brings an external dependency (Java)

More details about web drivers could be found on http://www.w3.org/TR/webdriver

TestCafe is based on another approach where the interaction with browser is cut to a minimum - it uses only the functionality to open a window / tab, resize, change the URL and a few others. Interaction with the web-page is done through a web-proxy server testcafe-hammerhead, that performs loading of a remote web page and modifications on the original JS code in a way that it can run the functional tests. https://github.com/DevExpress/testcafe-hammerhead

TestCafe’s method is more versatile because it can potentially work with any browser that supports HTML5 and ES5+, whereas NightWatch requires an appropriate web driver. Also, it allows running tests in any browser on any device without having to install additional software.

Testing our web application on Android is presented in this video: https://youtu.be/2na5jkqvUx0

Unfortunately, testcafe-hammerhead has a few potential disadvantages:

• an overhead for the analysis and modification of the application code
• potentially incorrect behavior of the web page that is being tested if the source code has been proxied with a mistake.

For example, one can bypass the replacement of the native alert popup in TestCafe and unintentionally or on purpose “hang up” test’s execution using the following code: http://pastebin.com/p6gLWA75.

Conclusions

Selenium-webdriver which is used by NightWatch is a popular and well-known solution with a standard and well-established API. In some case, for instance, when one needs to automate given website in a specific browser (writing a bot for remote website) - selenium-webdriver is much more suited.

However, in order to implement functional tests for a web application under active development TestCafe is undoubtedly ahead of its competitor because of a wide variety of reasons:

1. Running the tests in any browser on any device in a batch manner without installing 3rd-party software on the devices.
2. The ease of writing tests with ES2016 including async / await keywords, ability to import JS modules of the project under the test and the import/export of functions to/from the client etc. These are great opportunities to integrate and manage your web application.
3. The broad support for DOM-selectors, easy interaction with the DOM, virtual mouse cursor and an ability to perform complex user interactions on the page.

Thus, among existing open-source solutions for functional testing of web-applications TestCafe a is very attractive option, with a combination of lightness and functionality.

Source code of the website and tests is available here.