Code coverage

January 09, 2015

It's a rare project that maintains 100% code coverage, but in general we want to keep coverage as high as we can. This helps maintain a semi-formal specification of our software by enumerating the various use cases, and tracks which areas of the code are used to service the use cases.

This also reveals areas of the code that are not used to service the specified use cases, which we want to know about so that we can either purge the dead code, or formalize and validate the use cases implied by the uncovered code.

Improving code coverage

Let's find some uncovered code and add a test for it.

npm run-script coverage

Browsing the coverage report at <project>/coverage/lcov-report/index.html, we find some uncovered code:

Non-covered code
Non-covered code

It looks like we forgot to test putLatestFormat() in the pagecontent bucket handler. Let's write a test for it:

test/pagecontent/putLatestFormat.js:

describe('pagecontent bucket handler', function() {
    it('should allow the latest format to be submitted', function() {
        this.timeout(20000);
        return preq.put({
            uri: config.bucketURL + '/Main_Page/html',
            headers: { 'content-type': 'text/html' },
            body: 'this is the latest'
        })
        .then(function(res) {
            assert.deepEqual(res.status, 201);
            return preq.get({
              uri: config.bucketURL + '/Main_Page/html/' + res.headers.etag,
            });
        })
        .then(function(res) {
            assert.deepEqual(res.status, 200);
            assert.deepEqual(res.body, 'this is the latest');
        });
    });
});

That should do it. Let's make sure it passes:

npm test
Test results
Test results

The test passes, so let's generate a new coverage report:

npm run-script coverage
Coverage report
Coverage report

Now we can verify that putLatestFormat() is now covered:

Covered code
Covered code

Not only have we increased our code coverage, but we have reverse-engineered a specification for how a user can submit (and later retrieve) the latest format for some page content.

We have also drawn attention to an interesting implementation detail: that in this case a PUT request results in a 201 status, which might hint that this ought to instead use a POST request. In any case, we have a point of reference for future design discussions and refactoring efforts.

Publishing code coverage

In RESTBase, we build on Travis CI and report coverage to Coveralls.

Not only does this keep us informed of the build status of each pull request, it lets us know via pull request comments how our changes affect the code coverage.

PR coverage status
PR coverage status

To enable code coverage reporting, make the following changes:

package.json

Add a couple new npm scripts:

"scripts": {
  "coverage": "istanbul cover _mocha --report lcovonly -- -R spec",
  "coveralls": "cat ./coverage/lcov.info | coveralls && rm -rf ./coverage"
}

Add istanbul, mocha-lcov-reporter, and coveralls as dev dependencies:

"devDependencies": {
  "istanbul": "0.3.5",
  "mocha-lcov-reporter": "0.0.1",
  "coveralls": "2.11.2"
}

.travis.yml

Tell Travis how to generate coverage info and send it off to Coveralls:

script: npm run-script coverage && npm run-script coveralls

Now when you submit a pull request, Travis builds your code and reports coverage info to Coveralls. Coveralls comments on your pull request, and creates some nice reports on its site:

Jobs coverage report
Jobs coverage report
Files coverage report
Files coverage report