[Solved] Green tests are nice, but what about coverage?

Showcase
1

edit: I’ve updated the title to indicate post contains a solution despite the question mark

Now that there is a test explorer for VSCode you probably wonder is there a way to somehow show coverage results in VSCode?
Well, I’m glad you asked. Let me get you covered - at least some of you.

Formatter, tokentree and checkstyle all use munit and mcover to run tests and collect coverage results. Each project has a test.hxml that references both libraries and enables coverage macros. We run tests from those test.hxml files rather than using munit’s command line tool. It was almost three years ago that checkstyle started recording coverage on codecov.io using a similar setup to what I’m going to showcase here.

I have no knowledge on how well mcover plays with test frameworks other than munit, so everything in here might apply only to people using munit for their tests.

Usually mcover just prints out coverage results to your console, either in a summary or a more detailed report including a list of each branch that wasn’t covered. While a list in a console window gives you at least some hints on how to improve your coverage, it’s not really that nice and easy to work with.
Luckily there are two additional reporters (or print clients) that you can use to get coverage results into a more manageable form:

  1. Use CodecovJsonPrintClient to report coverage data to codecov.io e.g. formatter coverage
    Just add CodecovJsonPrintClient to your main test class, just before calling run on your TestRunner instance, e.g.:
var suites = [TestSuite];
var client = new MCoverPrintClient();

MCoverage.getLogger().addClient(new CodecovJsonPrintClient());

var runner = new TestRunner(client);
runner.run(suites);

You will end up having a coverage.json file in your workspace root - or wherever you run your tests. You can submit that file to your codecov account by e.g. using their bash uploader which will automatically pick up coverage.json.
You might want to run it as part of your CI build job to give you nice coverage results. You can also look at changes in coverage for new commits / pull requests or even between branches. When integrated with your github repository you can see how a new pull request affects your coverage before merge, allowing you to add more testcases so overall coverage doesn’t drop.

codecov
codecov870×259 21.1 KB

Yeah I know, I promised VSCode not somewhere in a cloud. Please hang on.

  1. Use LcovPrintClient to visualise coverage data in VSCode
    Like with codecov you add a print client to make mcover generate a lcov.info file, which you can then use to get coverage results into VSCode:
var suites = [TestSuite];
var client = new MCoverPrintClient();

MCoverage.getLogger().addClient(new LcovPrintClient("Unittests"));

var runner = new TestRunner(client);
runner.run(suites);

After running your tests a file named lcov.info will appear in your workspace root (or wherever you run your tests). It’s a text file that you can feed into a number of tools that support lcov format. Two of which come in the form of VSCode extensions:

a) coverage-gutters shows coverage results for your current file either in gutter (left, next to line numbers), ruler (right) or per line (a lot of colour right in your code) - or any combination of them (via settings.json). You can see a full set here:

coverage-gutters
coverage-gutters870×250 35.9 KB

You can toggle coverage-gutters on and off by clicking Watch / Remove Watch in VSCode’s status bar.

b) code-coverage is a bit less colourful and uses diagnostics to show lines not covered by tests. As a result your problems view might get flodded with entries because each line not covered appears as a separate issue. You can click each entry to jump to it’s corresponding position. Diagnostics show up as squiggly lines in your code.

code-coverage
code-coverage870×391 54.2 KB

coverage-gutters and code-coverage both show lines that were not covered during your test run. Additionally coverage-gutters also marks lines that have partial coverage where e.g. an if-branch was covered but tests never hit your else-branch. So it gives you a bit more information especially when you don’t have an else-branch. I suggest you play with both to see which one works best for you.

In previous versions coverage-gutters’ lcov parser was rather slow, taking a long time to read your lcov.info file especially for large projects. Recent versions seem to have addressed that issue.

Please note: For LcovPrintClient to work with both extensions you need a mcover version above 2.2.1, which currently means a git / dev version. covereage-gutters should work with 2.2.1, but code-coverage needs a more recent version, that generates absolute file paths before showing coverage.
Current git version also handles coverage data of multiple classes in the same file.

8 Likes
2

This looks really cool! Thanks for the effort to explain this!
I am having a bit of a hard time getting this to work though…

If I use mcover 2.2.1 I need to put the creation of the LcovPrinterClient between #if sys #end.
In that case I also get a “No generated coverage” exception.

Using the git version is also not working for me. I am using this:
haxelib git mcover https://github.com/massiveinteractive/mcover.git

If I do this, the compiler is not able to find the mcover namespace…

Any suggestions?

3

You can’t install mcover with that haxelib git command because the actual library / haxelib.json is in a subdirectory. You need to specify the subdirectory like this:

haxelib git mcover https://github.com/massiveinteractive/mcover master src
4

To enable coverage collection you will also need to put something like --macro mcover.MCover.coverage(['formatter'], ['src'], ['formatter.Cli']) into your test.hxml or CLI call.
Parameters are

  1. list of packages
  2. list of classpaths to search
  3. list of exclusions
5

The new mcover version 2.2.2 on haxelib now fully supports and works with both extension mentioned in my initial post.

There is also a new contender named code-coverage-highlighter. It shows covered lines with a green background and lines not covered or partially covered in red. It doesn’t really distinguish between partial and non covered lines, so it doesn’t give as many details as coverage-gutters.
However along with highlighting your code it adds a percentage indicator to your status bar, showing a coverage total of your current file. And it’s clickable, so you can quickly toggle coverage highlighting on or off.

code-coverage-highlighter

2 Likes
6

MCover also works fine with UTest library:

import mcover.coverage.MCoverage;
import mcover.coverage.client.LcovPrintClient;
import utest.Runner;
import utest.ui.Report;

/** Runs the test suite. **/
class TestAll {

	/** The test cases. **/
	static final tests = [
		new Test1(),
		new Test2()
	];

	/** Application entry point. **/
	static function main(): Void {
		final runner = new Runner();
		runner.onComplete.add(_ -> {
			final logger = MCoverage.getLogger();
			logger.addClient(new LcovPrintClient("coverage", "var/lcov.info"));
			logger.report();
		});

		Report.create(runner);
		for (test in tests) runner.addCase(test);
		runner.run();
	}
}

Beware that coverage can be incomplete if you use the “no-brace function syntax” (whatever test library you’re using):

2 Likes