BB-1030 Collect code coverage from every test
Code coverage is not being calculated for integration tests nor for front end tests.
The solution is to run codecov
with --parallel-mode
that way every run will have a unique number attached at the end of the output file. Then, after every test has been run we run codecov combine
which combines every code coverage file. Finally, we run codecov report
to print the report and check for a coverage threshold.
There's multiple updates to the way CircleCI runs the workflow as well as how we run tests:
- Taking advantage of some new things in CirlceCI the workflow has been re-written to run in this way:
- First the quality test is run together with the new migrations checks. If either test fails the rest of the test will not run.
- Then, the workflow runs every unit test (JS jasmine tests, django unit tests and "browser" tests). Although "browsers" tests are not exactly unit test they fall under this category because they are more lightweight than integration tests and don't call any external services. If either of these tests fail then the integration tests will not run.
- There was a change on how we run jasmine tests. We used to run them using python jasmine which is really simple to setup for python projects but it's really hard to get code coverage since most stable tools for JS code coverage are expecting to be run with JS test runners. The JS tests are now run with Karma + Jasmine which allow us to use istanbul to calculate code coverage.
- By using Karma + Jasmine we can now use karma's browser launchers, which are easier to use. The downside here is that we now have to deal with
npm
. - After the unit tests are run the workflow will run the integration tests separating them into three parallel jobs. The custom test runner is now deprecated since the environment variable used to see in which "cirlce ci node" the test were running is not usable with the new workflow. Instead, a
test_group
variable is specified on each parallel job. Thetest_group
variable is then passed into the job as an environment variable andunittest
'sskipIf
is used to check which tests should run on each parallel job. This makes the entire integration tests running easier to understand. - After quality, unit and integration tests are run successfully a job to combine every code coverage is run. Every previously ran job persist its coverage file to a workspace which the
coverage
job then mounts and combine. - Finally, the cleanup job is now scheduled to run every day at 1 am instead of being run with every CirlceCI run. The cleanup job will also run only for the
master
branch.s