Code coverage for integration tests

Problem to solve

Currently, we are not gathering code coverage for our automated test (#199 (closed)). We don't know how much of our code is untested.

Proposal

Use nyc in our test suite to gather coverage. This might require us to redesign the activation of our extension or discuss some possible fixes with the vscode team. The redesign might bring a benefit of VS Code starting faster for our users.

Further details

This is a complex task.

There is a rough guideline for adding nyc to an extension project - GitHub issue in vscode

And there is a good example in the microsoft/vscode-js-debug: The VS Code JavaScript debugger project.

src/test/testRunner.ts

function setupCoverage() {
  const NYC = require('nyc');
  const nyc = new NYC({
    cwd: join(__dirname, '..', '..', '..'),
    exclude: ['**/test/**', '.vscode-test/**'],
    reporter: ['text', 'html'],
    all: true,
    instrument: true,
    hookRequire: true,
    hookRunInContext: true,
    hookRunInThisContext: true,
  });

  nyc.reset();
  nyc.wrap();

  return nyc;
}

// ...

  finally {
    if (nyc) {
      nyc.writeCoverageFile();
      nyc.report();
    }
  }

The issue with this solution is that it only gathers coverage for files that have been required AFTER this setupCoverage() has been called.

This works for the vscode-js-debug extension, because they don't immediately activate the extension. (Documentation on activation events).

• vscode-js-debug - package.json

  "activationEvents": [
    "onLanguage:json",
    "onDebugDynamicConfigurations",
    "onDebugInitialConfigurations",
    "onDebugResolve:node",
    "onDebugResolve:extensionHost",
    "onDebugResolve:chrome"
  ],

• gitlab-workflow - package.json

  "activationEvents": [
    "*"
  ],

By the time we initialize nyc the whole extension with all dependencies has been already required.

I've spent 6 hours finding the problem and then investigating a solution. The most promising solution might be

Change the activation events

I've tried the following events:

    "onCommand:gl.showIssuesAssignedToMe",
    "onCommand:gl.showMergeRequestsAssignedToMe",
    "onCommand:gl.setToken",
    "onCommand:gl.removeToken",
    "onCommand:gl.openActiveFile",
    "onCommand:gl.copyLinkToActiveFile",
    "onCommand:gl.openCurrentMergeRequest",
    "onCommand:gl.openCreateNewIssue",
    "onCommand:gl.openCreateNewMR",
    "onCommand:gl.openProjectPage",
    "onCommand:gl.openCurrentPipeline",
    "onCommand:gl.pipelineActions",
    "onCommand:gl.issueSearch",
    "onCommand:gl.mergeRequestSearch",
    "onCommand:gl.compareCurrentBranch",
    "onCommand:gl.createSnippet",
    "onCommand:gl.validateCIConfig",
    "onCommand:gl.refreshSidebar",
    "onCommand:gl.showRichContent",
    "onView:issuesAndMrs",
    "workspaceContains:.git"

Note: the .git activation event is necessary because we want to show the status bar almost immediately

This solution produced results but with very fragile test infrastructure code.

Manually generated coverage report

Links / references