Testing

Introduction

Testing allows you to ensure your application works the way you think it does, especially as your codebase changes over time. If you have good tests, you can refactor and rewrite code with confidence. Tests are also the most concrete form of documentation of expected behavior, since other developers can figure out how to use your code by reading the tests.

Automated testing is critical because it allows you to run a far greater set of tests much more often than you could manually, allowing you to catch regression errors immediately.

Types of tests

Entire books have been written on the subject of testing, so we will simply touch on some basics of testing here. The important thing to consider when writing a test is what part of the application you are trying to test, and how you are verifying the behavior works.

Challenges of testing in Meteor

In most ways, testing a Meteor app is no different from testing any other full stack JavaScript application. However, compared to more traditional backend or front-end focused frameworks, two factors can make testing a little more challenging:

The ‘meteor test’ command

The primary way to test your application in Meteor is the meteor test command.

This loads your application in a special “test mode”. What this does is:

  1. Doesn’t eagerly load any of our application code as Meteor normally would.
  2. Does eagerly load any file in our application (including in imports/ folders) that look like *.test[s].*, or *.spec[s].*
  3. Sets the Meteor.isTest flag to be true.
  4. Starts up the test driver package (see below).

The Meteor build tool and the meteor test command ignore any files located in any tests/ directory. This allows you to put tests in this directory that you can run using a test runner outside of Meteor’s built-in test tools and still not have those files loaded in your application. See Meteor’s default file load order rules.

What this means is that you can write tests in files with a certain filename pattern and know they’ll not be included in normal builds of your app. When your app runs in test mode, those files will be loaded (and nothing else will), and they can import the modules you want to test. As we’ll see this is ideal for unit tests and simple integration tests.

Additionally, Meteor offers a “full application” test mode. You can run this with meteor test --full-app.

This is similar to test mode, with key differences:

  1. It loads test files matching *.app-test[s].* and *.app-spec[s].*.
  2. It does eagerly load our application code as Meteor normally would.

This means that the entirety of your application (including for instance the web server and client side router) is loaded and will run as normal. This enables you to write much more complex integration tests and also load additional files for acceptance tests.

Note that there is another test command in the Meteor tool; meteor test-packages is a way of testing Atmosphere packages, which we’ll discuss in more detail in an upcoming article about writing packages.

Driver packages

When you run a meteor test command, you must provide a --driver-package argument. A test driver is a mini-application that runs in place of your app and runs each of your defined tests, whilst reporting the results in some kind of user interface.

There are two main kinds of test driver packages:

Recommended: Mocha

In this article, we’ll use the popular Mocha test runner alongside the Chai assertion library to test our application. In order to write tests in Mocha, we can add the practicalmeteor:mocha package to our app.

1
meteor add practicalmeteor:mocha

This package also doesn’t do anything in development or production mode (in fact it declares itself testOnly so it is not even included in those modes), but when our app is run in test mode, it takes over, executing test code on both the client and server, and rendering results to the browser.

Test files themselves (for example a file named todos-item.test.js or routing.app-specs.coffee) can register themselves to be run by the test driver in the usual way for that testing library. For Mocha, that’s by using describe and it:

1
2
3
4
5
6
7
8
// Note: Arrow function use with Mocha is discouraged.
// (see http://mochajs.org/#arrow-functions)
describe('my module', function () {
it('does something that should be tested', function () {
// This code will be executed by the test driver when the app is started
// in the correct mode
})
})

Test data

When your app is run in test mode, it is initialized with a clean test database.

If you are running a test that relies on using the database, and specifically the content of the database, you’ll need to perform some setup steps in your test to ensure the database is in the state you expect. There are some tools you can use to do this.

To ensure the database is clean, the xolvio:cleaner package is useful. You can use it to reset the database in a beforeEach block:

1
2
3
4
5
6
7
import { resetDatabase } from 'meteor/xolvio:cleaner';

describe('my module', function () {
beforeEach(function () {
resetDatabase();
});
});

This technique will only work on the server. If you need to reset the database from a client test, you can use a method to do so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import { resetDatabase } from 'meteor/xolvio:cleaner';

// NOTE: Before writing a method like this you'll want to double check
// that this file is only going to be loaded in test mode!!
Meteor.methods({
'test.resetDatabase': () => resetDatabase();
});

describe('my module', function (done) {
beforeEach(function (done) {
// We need to wait until the method call is done before moving on, so we
// use Mocha's async mechanism (calling a done callback)
Meteor.call('test.resetDatabase', done);
});
});

As we’ve placed the code above in a test file, it will not load in normal development or production mode (which would be an incredibly bad thing!). If you create a Atmosphere package with a similar feature, you should mark it as testOnly and it will similarly only load in test mode.

Generating test data

Often it’s sensible to create a set of data to run your test against. You can use standard insert() calls against your collections to do this, but often it’s easier to create factories which help encode random test data. A great package to use to do this is dburles:factory.

In the Todos example app, we define a factory to describe how to create a test todo item, using the faker npm package:

1
2
3
4
5
6
7
import faker from 'faker';

Factory.define('todo', Todos, {
listId: () => Factory.get('list'),
text: () => faker.lorem.sentence(),
createdAt: () => new Date(),
});

To use the factory in a test, we simply call Factory.create:

1
2
3
4
5
6
// This creates a todo and a list in the database and returns the todo.
const todo = Factory.create('todo');

// If we have a list already, we can pass in the id and avoid creating another:
const list = Factory.create('list');
const todoInList = Factory.create('todo', { listId: list._id });

Mocking the database

As Factory.create directly inserts documents into the collection that’s passed into the Factory.define function, it can be a problem to use it on the client. However there’s a neat isolation trick that you can do to replace the server-backed Todos client collection with a mocked out local collection, that’s encoded in the stub-collections package (currently a local package in the Todos example application).

1
2
3
4
5
6
7
8
9
10
11
import { StubCollections } from 'meteor/stub-collections';
import { Todos } from 'path/to/todos.js';

StubCollections.stub(Todos);

// Now Todos is stubbed to a simple local collection mock,
// so for instance on the client we can do:
Todos.insert({ a: 'document' });

// Restore the `Todos` collection
StubCollections.restore();

In a Mocha test, it makes sense to use stub-collections in a beforeEach/afterEach block.

Unit testing

Unit testing is the process of isolating a section of code and then testing that the internals of that section work as you expect. As we’ve split our code base up into ES2015 modules it’s natural to test those modules one at a time.

By isolating a module and simply testing its internal functionality, we can write tests that are fast and accurate—they can quickly tell you where a problem in your application lies. Note however that incomplete unit tests can often hide bugs because of the way they stub out dependencies. For that reason it’s useful to combine unit tests with slower (and perhaps less commonly run) integration and acceptance tests.

A simple unit test

In the Todos example application, thanks to the fact that we’ve split our User Interface into smart and reusable components, it’s natural to want to unit test some of our reusable components (we’ll see below how to integration test our smart components).

To do so, we’ll use a very simple test helper that renders a Blaze component off-screen with a given data context (note that the React test utils can do a similar thing for React). As we place it in imports/ui/test-helpers.js it won’t load in our app by in normal mode (as it’s not required anywhere):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import { _ } from 'meteor/underscore';
import { Template } from 'meteor/templating';
import { Blaze } from 'meteor/blaze';
import { Tracker } from 'meteor/tracker';

const withDiv = function withDiv(callback) {
const el = document.createElement('div');
document.body.appendChild(el);
try {
callback(el);
} finally {
document.body.removeChild(el);
}
};

export const function withRenderedTemplate(template, data, callback) {
withDiv((el) => {
const ourTemplate = _.isString(template) ? Template[template] : template;
Blaze.renderWithData(ourTemplate, data, el);
Tracker.flush();
callback(el);
});
};

A simple example of a reusable component to test is the Todos_item template. Here’s what a unit test looks like (you can see some others in the app repository):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
/* eslint-env mocha */
/* eslint-disable func-names, prefer-arrow-callback */

import { Factory } from 'meteor/factory';
import { chai } from 'meteor/practicalmeteor:chai';
import { Template } from 'meteor/templating';
import { $ } from 'meteor/jquery';


import { withRenderedTemplate } from '../../test-helpers.js';
import '../todos-item.js';

describe('Todos_item', function () {
beforeEach(function () {
Template.registerHelper('_', key => key);
});

afterEach(function () {
Template.deregisterHelper('_');
});

it('renders correctly with simple data', function () {
const todo = Factory.build('todo', { checked: false });
const data = {
todo,
onEditingChange: () => 0,
};

withRenderedTemplate('Todos_item', data, el => {
chai.assert.equal($(el).find('input[type=text]').val(), todo.text);
chai.assert.equal($(el).find('.list-item.checked').length, 0);
chai.assert.equal($(el).find('.list-item.editing').length, 0);
});
});
});

Of particular interest in this test is the following:

Importing

When we run our app in test mode, only our test files will be eagerly loaded. In particular, this means that in order to use our templates, we need to import them! In this test, we import todos-item.js, which itself imports todos.html (yes, you do need to import the HTML files of your Blaze templates!)

Stubbing

To be a unit test, we must stub out the dependencies of the module. In this case, thanks to the way we’ve isolated our code into a reusable component, there’s not much to do; principally we need to stub out the {{_}} helper that’s created by the tap:i18n system. Note that we stub it out in a beforeEach and restore it the afterEach.

Creating data

We can use the Factory package’s .build() API to create a test document without inserting it into any collection. As we’ve been careful not to call out to any collections directly in the reusable component, we can pass the built todo document directly into the template.

Running unit tests

To run the tests that our app defines, we run our app in test mode:

1
meteor test --driver-package practicalmeteor:mocha

As we’ve defined a test file (imports/todos/todos.tests.js), what this means is that the file above will be eagerly loaded, adding the 'builds correctly from factory' test to the Mocha registry.

To run the tests, visit http://localhost:3000 in your browser. This kicks off practicalmeteor:mocha, which runs your tests both in the browser and on the server. It displays the test results in the browser in a Mocha test reporter:

Usually, while developing an application, it make sense to run meteor test on a second port (say 3100), while also running your main application in a separate process:

1
2
3
4
5
# in one terminal window
meteor

# in another
meteor test --driver-package practicalmeteor:mocha --port 3100

Then you can open two browser windows to see the app in action while also ensuring that you don’t break any tests as you make changes.

Isolation techniques

In the unit test above we saw a very limited example of how to isolate a module from the larger app. This is critical for proper unit testing. Some other utilities and techniques include:

There’s a lot of scope for better isolation and testing utilities (the two packages from the example app above could be improved greatly!). We encourage the community to take the lead on these.

Integration testing

An integration test is a test that crosses module boundaries. In the simplest case, this simply means something very similar to a unit test, where you perform your isolation around multiple modules, creating a non-singular “system under test”.

Although conceptually different to unit tests, such tests typically do not need to be run any differently to unit tests and can use the same meteor test mode and isolation techniques as we use for unit tests.

However, an integration test that crosses the client-server boundary of a Meteor application (where the modules under test cross that boundary) requires a different testing infrastructure, namely Meteor’s “full app” testing mode.

Let’s take a look at example of both kinds of tests.

Simple integration test

Our reusable components were a natural fit for a unit test; similarly our smart components tend to require an integration test to really be exercised properly, as the job of a smart component is to bring data together and supply it to a reusable component.

In the Todos example app, we have an integration test for the Lists_show_page smart component. This test simply ensures that when the correct data is present in the database, the template renders correctly – that it is gathering the correct data as we expect. It isolates the rendering tree from the more complex data subscription part of the Meteor stack. If we wanted to test that the subscription side of things was working in concert with the smart component, we’d need to write a full app integration test.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
/* eslint-env mocha */
/* eslint-disable func-names, prefer-arrow-callback */

import { Meteor } from 'meteor/meteor';
import { Factory } from 'meteor/factory';
import { Random } from 'meteor/random';
import { chai } from 'meteor/practicalmeteor:chai';
import { StubCollections } from 'meteor/stub-collections';
import { Template } from 'meteor/templating';
import { _ } from 'meteor/underscore';
import { $ } from 'meteor/jquery';
import { FlowRouter } from 'meteor/kadira:flow-router';
import { sinon } from 'meteor/practicalmeteor:sinon';


import { withRenderedTemplate } from '../../test-helpers.js';
import '../lists-show-page.js';

import { Todos } from '../../../api/todos/todos.js';
import { Lists } from '../../../api/lists/lists.js';

describe('Lists_show_page', function () {
const listId = Random.id();

beforeEach(function () {
StubCollections.stub([Todos, Lists]);
Template.registerHelper('_', key => key);
sinon.stub(FlowRouter, 'getParam', () => listId);
sinon.stub(Meteor, 'subscribe', () => ({
subscriptionId: 0,
ready: () => true,
}));
});

afterEach(function () {
StubCollections.restore();
Template.deregisterHelper('_');
FlowRouter.getParam.restore();
Meteor.subscribe.restore();
});

it('renders correctly with simple data', function () {
Factory.create('list', { _id: listId });
const timestamp = new Date();
const todos = _.times(3, i => Factory.create('todo', {
listId,
createdAt: new Date(timestamp - (3 - i)),
}));

withRenderedTemplate('Lists_show_page', {}, el => {
const todosText = todos.map(t => t.text).reverse();
const renderedText = $(el).find('.list-items input[type=text]')
.map((i, e) => $(e).val())
.toArray();
chai.assert.deepEqual(renderedText, todosText);
});
});
});

Of particular interest in this test is the following:

Importing

As we’ll run this test in the same way that we did our unit test, we need to import the relevant modules under test in the same way that we did in the unit test.

Stubbing

As the system under test in our integration test has a larger surface area, we need to stub out a few more points of integration with the rest of the stack. Of particular interest here is our use of the stub-collections and of Sinon to stub out Flow Router and our Subscription.

Creating data

In this test, we used Factory package’s .create() API, which inserts data into the real collection. However, as we’ve proxied all of the Todos and Lists collection methods onto a local collection (this is what stub-collections is doing), we won’t run into any problems with trying to perform inserts from the client.

This integration test can be run the exact same way as we ran unit tests above.

Full-app integration test

In the Todos example application, we have a integration test which ensures that we see the full contents of a list when we route to it, which demonstrates a few techniques of integration tests:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
/* eslint-env mocha */
/* eslint-disable func-names, prefer-arrow-callback */

import { Meteor } from 'meteor/meteor';
import { Tracker } from 'meteor/tracker';
import { DDP } from 'meteor/ddp-client';
import { FlowRouter } from 'meteor/kadira:flow-router';
import { assert } from 'meteor/practicalmeteor:chai';
import { Promise } from 'meteor/promise';
import { $ } from 'meteor/jquery';

import { generateData } from './../../api/generate-data.app-tests.js';
import { Lists } from '../../api/lists/lists.js';
import { Todos } from '../../api/todos/todos.js';


// Utility -- returns a promise which resolves when all subscriptions are done
const waitForSubscriptions = () => new Promise(resolve => {
const poll = Meteor.setInterval(() => {
if (DDP._allSubscriptionsReady()) {
clearInterval(poll);
resolve();
}
}, 200);
});

// Tracker.afterFlush runs code when all consequent of a tracker based change
// (such as a route change) have occured. This makes it a promise.
const afterFlushPromise = Promise.denodeify(Tracker.afterFlush);

if (Meteor.isClient) {
describe('data available when routed', () => {
// First, ensure the data that we expect is loaded on the server
// Then, route the app to the homepage
beforeEach(() => generateData().then(() => FlowRouter.go('/')));

describe('when logged out', () => {
it('has all public lists at homepage', () => {
assert.equal(Lists.find().count(), 3);
});

it('renders the correct list when routed to', () => {
const list = Lists.findOne();
FlowRouter.go('Lists.show', { _id: list._id });

return afterFlushPromise()
.then(() => {
assert.equal($('.title-wrapper').html(), list.name);
})
.then(() => waitForSubscriptions())
.then(() => {
assert.equal(Todos.find({ listId: list._id }).count(), 3);
});
});
});
});
}

Of note here:

Running full-app tests

To run the full-app tests in our application, we run:

1
meteor test --full-app --driver-package practicalmeteor:mocha

When we connect to the test instance in a browser, we want to render a testing UI rather than our app UI, so the mocha-web-reporter package will hide any UI of our application and overlay it with its own. However the app continues to behave as normal, so we are able to route around and check the correct data is loaded.

Creating data

To create test data in full-app test mode, it usually makes sense to create some special test methods which we can call from the client side. Usually when testing a full app, we want to make sure the publications are sending through the correct data (as we do in this test), and so it’s not sufficient to stub out the collections and place synthetic data in them. Instead we’ll want to actually create data on the server and let it be published.

Similar to the way we cleared the database using a method in the beforeEach in the test data section above, we can call a method to do that before running our tests. In the case of our routing tests, we’ve used a file called imports/api/generate-data.app-tests.js which defines this method (and will only be loaded in full app test mode, so is not available in general!):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
// This file will be auto-imported in the app-test context,
// ensuring the method is always available

import { Meteor } from 'meteor/meteor';
import { Factory } from 'meteor/factory';
import { resetDatabase } from 'meteor/xolvio:cleaner';
import { Random } from 'meteor/random';
import { _ } from 'meteor/underscore';

const createList = (userId) => {
const list = Factory.create('list', { userId });
_.times(3, () => Factory.create('todo', { listId: list._id }));
return list;
};

// Remember to double check this is a test-only file before
// adding a method like this!
Meteor.methods({
generateFixtures: function generateFixturesMethod() {
resetDatabase();

// create 3 public lists
_.times(3, () => createList());

// create 3 private lists
_.times(3, () => createList(Random.id()));
},
});

let generateData;
if (Meteor.isClient) {
// Create a second connection to the server to use to call
// test data methods. We do this so there's no contention
// with the currently tested user's connection.
const testConnection = Meteor.connect(Meteor.absoluteUrl());

generateData = Promise.denodeify((cb) => {
testConnection.call('generateFixtures', cb);
});
}

export { generateData };

Note that we’ve exported a client-side symbol generateData which is a promisified version of the method call, which makes it simpler to use this sequentially in tests.

Also of note is the way we use a second DDP connection to the server in order to send these test “control” method calls.

Acceptance testing

Acceptance testing is the process of taking an unmodified version of our application and testing it from the “outside” to make sure it behaves in a way we expect. Typically if an app passes acceptance tests, we have done our job properly from a product perspective.

As acceptance tests test the behavior of the application in a full browser context in a generic way, there are a range of tools that you can to specify and run such tests. In this guide we’ll demonstrate using Chimp, an acceptance testing tool with a few neat Meteor-specific features that makes it easy to use.

We can install the Chimp tool globally using:

1
meteor npm install --global chimp

Note that you can also install Chimp as a devDependency in your package.json but you may run into problems deploying your application as it includes binary dependencies. You can avoid such problems by running meteor npm prune to remove non-production dependencies before deploying.

Chimp has a variety of options for setting it up, but we can add some npm scripts which will run the currently tests we define in Chimp’s two main modes. We can add them to our package.json:

1
2
3
4
5
6
{
"scripts": {
"chimp-watch": "chimp --ddp=http://localhost:3000 --watch --mocha --path=tests",
"chimp-test": "chimp --mocha --path=tests"
}

}

Chimp will now look in the tests/ directory (otherwise ignored by the Meteor tool) for files in which you define acceptance tests. In the Todos example app, we define a simple test that ensures we can click the “create list” button:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
/* eslint-env mocha */
/* eslint-disable func-names, prefer-arrow-callback */

// These are Chimp globals
/* globals browser assert server */

function countLists() {
browser.waitForExist('.list-todo');
const elements = browser.elements('.list-todo');
return elements.value.length;
};

describe('list ui', function () {
beforeEach(function () {
browser.url('http://localhost:3000');
server.call('generateFixtures');
});

it('can create a list @watch', function () {
const initialCount = countLists();

browser.click('.js-new-list');

assert.equal(countLists(), initialCount + 1);
});
});

Running acceptance tests

To run acceptance tests, we simply need to start our Meteor app as usual, and point Chimp at it.

In one terminal, we can do:

1
meteor

In another:

1
meteor npm run chimp-watch

The chimp-watch command will then run the test in a browser, and continue to re-run it as we change the test or the application. (Note that the test assumes we are running the app on port 3000).

Thus it’s a good way to develop the test—this is why chimp has a feature where we mark tests with a @watch in the name to call out the tests we want to work on (running our entire acceptance test suite can be time consuming in a large application).

The chimp-test command will run all of the tests once only and is good for testing that our suite passes, either as a manual step, or as part of a continuous integration process.

Creating data

Although we can run the acceptance test against our “pure” Meteor app, as we’ve done above, it often makes sense to start our meteor server with a special test driver, tmeasday:acceptance-test-driver. (You’ll need to meteor add it to your app):

1
meteor test --full-app --driver-package tmeasday:acceptance-test-driver

The advantage of running our acceptance test suite pointed at an app that runs in full app test mode is that all of the data generating methods that we’ve created remain available. Otherwise the acceptance-test-driver does nothing.

In Chimp tests, you have a DDP connection to the server available on the server variable. You can thus use server.call() (which is wrapped to be synchronous in Chimp tests) to call these methods. This is a convenient way to share data preparation code between acceptance and integration tests.

Continuous Integration

Continuous integration testing is the process of running tests on every commit of your project.

There are two principal ways to do it: on the developer’s machine before allowing them to push code to the central repository, and on a dedicated CI server after each push. Both techniques are useful, and both require running tests in a commandline-only fashion.

Command line

We’ve seen one example of running tests on the command line, using our meteor npm run chimp-test mode.

We can also use a command-line driver for Mocha dispatch:mocha-phantomjs to run our standard tests on the command line.

Adding and using the package is straightforward:

1
2
meteor add dispatch:mocha-phantomjs
meteor test --once --driver-package dispatch:mocha-phantomjs

(The --once argument ensures the Meteor process stops once the test is done).

We can also add that command to our package.json as a test script:

1
2
3
4
5
{
"scripts": {
"test": "meteor test --once --driver-package dispatch:mocha-phantomjs"
}

}

Now we can run the tests with meteor npm test.

CircleCI

CircleCI is a great continuous integration service that allows us to run (possibly time consuming) tests on every push to a repository like GitHub. To use it with the the commandline test we’ve defined above, we can follow their standard getting started tutorial and use a circle.yml file similar to this:

1
2
3
4
5
6
7
8
9
10
machine:
node:
version: 0.10.43
dependencies:
override:
- curl https://install.meteor.com | /bin/sh
- npm install
checkout:
post:
- git submodule update --init