Skip to content

BB-1030 Collect code coverage from every test

Josue Balandrano Coronel requested to merge josuebc/BB-1030 into master

Code coverage is not being calculated for integration tests nor for front end tests. The solution is to run codecov with --parallel-mode that way every run will have a unique number attached at the end of the output file. Then, after every test has been run we run codecov combine which combines every code coverage file. Finally, we run codecov report to print the report and check for a coverage threshold. There's multiple updates to the way CircleCI runs the workflow as well as how we run tests:

Screen Shot 2019-04-12 at 9 54 11 AM The main gains from this type of workflow is to make sure everything is correct before running integration tests (since those are quite heavy) and to be able to re-run any step without having to re-run the entire workflow.
  • First the quality test is run together with the new migrations checks. If either test fails the rest of the test will not run.
  • Then, the workflow runs every unit test (JS jasmine tests, django unit tests and "browser" tests). Although "browsers" tests are not exactly unit test they fall under this category because they are more lightweight than integration tests and don't call any external services. If either of these tests fail then the integration tests will not run.
  • There was a change on how we run jasmine tests. We used to run them using python jasmine which is really simple to setup for python projects but it's really hard to get code coverage since most stable tools for JS code coverage are expecting to be run with JS test runners. The JS tests are now run with Karma + Jasmine which allow us to use istanbul to calculate code coverage.
  • By using Karma + Jasmine we can now use karma's browser launchers, which are easier to use. The downside here is that we now have to deal with npm.
  • After the unit tests are run the workflow will run the integration tests separating them into three parallel jobs. The custom test runner is now deprecated since the environment variable used to see in which "cirlce ci node" the test were running is not usable with the new workflow. Instead, a test_group variable is specified on each parallel job. The test_group variable is then passed into the job as an environment variable and unittest's skipIf is used to check which tests should run on each parallel job. This makes the entire integration tests running easier to understand.
  • After quality, unit and integration tests are run successfully a job to combine every code coverage is run. Every previously ran job persist its coverage file to a workspace which the coverage job then mounts and combine.
  • Finally, the cleanup job is now scheduled to run every day at 1 am instead of being run with every CirlceCI run. The cleanup job will also run only for the master branch.s

Merge request reports