JavascriptTesting.md 6.55 KB
Newer Older
1
2
3
4
5
6
7
8
9
10
11
<!--
This file is part of CPAchecker,
a tool for configurable software verification:
https://cpachecker.sosy-lab.org

SPDX-FileCopyrightText: 2018 Lokesh Nandanwar
SPDX-FileCopyrightText: 2018-2020 Dirk Beyer <https://www.sosy-lab.org>

SPDX-License-Identifier: Apache-2.0
-->

12
13
# JavaScript Testing

nandanwar's avatar
nandanwar committed
14
This documentation doesn't discuss much about testing. You can check [Protractor](https://www.protractortest.org/) and [Jasmine](https://jasmine.github.io/) documentation for more details.
15
16
17
18
19
20
21
22
23
24
25
26
27
28
This documentation adds both karma/jasmine unit test and protractor end-to-end testing support for AngularJS-JavaScript report.

These tools are configured for specific conventions described below.

*It is unwise and rarely possible to run the the unit tests, and the e2e tests at the same time.
We recommend that you shut down one before starting another.*

## Writing Test Cases
To create tests, add it() (or test()) blocks with the name of the test and its code. You may optionally wrap them in describe() blocks for logical grouping but this is not required.

Jasmine provides a built-in expect() global function for making assertions. A basic test could look like this:

```js
describe("true", function () {
29
    let a;
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
    it("Should be true", function () {
        a = true;
        expect(a).toBe(true);
    });
});
```

All expect() matchers supported by Jasmine/Karma for unit testing are [extensively documented here](https://jasmine.github.io/2.0/introduction.html#section-Included_Matchers).<br>
End-to-end testing also uses jasmine framework, so these matchers can also be used in end-to-end testing.

## Unit Testing

JavaScript unit-tests are located in the `src/org/sosy_lab/cpachecker/core/counterexample/test` folder. Their filenames must end in `.js`.

The above directory is a root for testing JavaScript for AngularJS report application.

Look for the example `src/org/sosy_lab/cpachecker/core/counterexample/test/test.js`.
Add more `.js` files as you wish in the above folder; we have configured karma to find them.

### Installation

**1. karma.conf.js:**

This is the configuration file for your `Karma/Jasmine` and it houses information such as the file format that `Jasmine` should look for when running test cases.

**2. 'test' directory:**

mlachner's avatar
mlachner committed
57
There is a directory titled `test` in the testing directory.
58
59
60
61
62
63
64

It houses all `test case` files for Unit testing.

### Configuration

At first the javascript loads all the scripts in the the `report.html` to the `testreport.html` and then we start testing our functions using Jasmine/Karma.

mlachner's avatar
mlachner committed
65
We will use all available browsers out of `Google Chrome`, `Chromium` and `Firefox` that are installed locally in headless mode for Unit testing application. Make sure that at least one of these is installed on your machine.
66
67
68
69
70
71
72
73
74
75

For more information regarding configuration you can browse `karma.conf.js`.


### Running tests

Run the below commands in your terminal for Unit testing:

1. `npm install`
At first you have to install all dependencies and third party libraries to run the test cases.
mlachner's avatar
mlachner committed
76
This command is not needed every time you run the tests, it is required for the first time.
77

mlachner's avatar
mlachner committed
78
79
80
81
82
83
84
2. `npm run build`
Next, you will need to bundle all necessary `.js` and `.css` of our source code and all third party libraries into single files.
The tests assume that these bundles files are available and will not work with only the raw files.
This command is not needed for every test run either, but should be used every time any changes to the JS-related source files are made.

3.  `npm test`
This command first compiles the application, then simultaneously re-compiles and runs the karma test-runner.
85
86
87
88
89
90
91
92
93
94
95
Test-runner output appears in the terminal window.
We can update our app and our tests in real-time, keeping a weather eye on the console for broken tests during development by setting up `karma.conf.js` file ( Set variable `autoWatch : true` and `singleRun : false` ).

### Unit tests Report

The test reports for Unit testing is generated by `karma-html-reporter` plugin.
You can get test reports in the folder `src/org/sosy_lab/cpachecker/core/counterexample/unit_testing_report` after running tests.


## End-to-End testing

96
### Setup
97

98
99
First we generate our AngularJS report application for an example program and then we use this report for end-to-end testing.
We are using Chromium browser and Selenium web driver for end-to-end testing application.
100

101
102
103
The tests are written for Protractor and are located in the `src/org/sosy_lab/cpachecker/core/counterexample/e2e-tests` folder.
Their filenames must end in `_spec.js`.
An example test is in `src/org/sosy_lab/cpachecker/core/counterexample/e2e-tests/test_spec.js`.
nandanwar's avatar
nandanwar committed
104
Add more `_spec.js` files as you wish in the above folder; we have configured Protractor to find them.
105

106
The configuration for Protractor is present in `e2e-conf.js`.
107
108
109
110
111

### Running tests

Run the below commands in your terminal for End-to-end testing:

112
113
114
115
116
1. Check if you have Java >= 8, Node >= 8, NPM, and Chrome or Chromium Browser installed.
   If not install it for running tests.
   On Ubuntu 18.04, these can be installed from the package repository.
   Alternatively, use our docker image:
   `docker run -it -v $(pwd):$(pwd) -w $(pwd) registry.gitlab.com/sosy-lab/software/cpachecker/test:java-node /bin/bash`
117

118
119
120
2. Generate Counterexample report for a specific program (`test/programs/simple/SSAMap-bug.c`) in the root of this application using
  `scripts/cpa.sh -predicateAnalysis test/programs/simple/SSAMap-bug.c` (for Unix based OS) and
  `scripts\cpa.bat -predicateAnalysis-linear -setprop analysis.checkCounterexamples=false -setprop solver.solver=smtinterpol test\programs\simple\SSAMap-bug.c` (for Windows based OS).
121
122

3. Navigate to `src/org/sosy_lab/cpachecker/core/counterexample` and run
123
124
125
  `npm install` and `./node_modules/protractor/node_modules/webdriver-manager/bin/webdriver-manager update`
  to install all dependencies and third party libraries.
  This is not needed only once.
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149

4.  `npm run e2e-test`<br>
This command first compiles the application, then simultaneously re-compiles and runs the Protractor test-runner. 
Test-runner output appears in the terminal window.


### End-to-end Test Report

The test report for End-to-end testing is generated by `protractor-jasmine2-screenshot-reporter` plugin. It also captures the screenshots after each test case.
You can get test report and screenshots in the folder `src/org/sosy_lab/cpachecker/core/counterexample/e2e-tests_report/screenshots/` after running tests.


## Continous Integration with Gitlab

All End-to-end tests and unit tests are configured to run in GitLab CI. The configuration file is `.gitlab-ci.yml` in the root folder of this repository.