README.md 13.9 KB
Newer Older
Gilbert Roulot's avatar
Gilbert Roulot committed
1 2
# GitLab License Management

3 4
[![pipeline status](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/badges/main/pipeline.svg)](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/commits/main)
[![coverage report](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/badges/main/coverage.svg)](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/commits/main)
Gilbert Roulot's avatar
Gilbert Roulot committed
5 6

GitLab tool for detecting licenses of the dependencies used by the provided source.
7
It is currently based on [License Finder][license_finder]
8
only, but this may change in the future.
Gilbert Roulot's avatar
Gilbert Roulot committed
9 10 11 12 13 14 15 16 17

## How to use

1. `cd` into the directory of the source code you want to scan
1. Run the Docker image:

    ```sh
    docker run \
      --volume "$PWD":/code \
mo's avatar
mo committed
18 19 20
      --env=LM_REPORT_VERSION="2.1" \
      --env=CI_PROJECT_DIR=/code \
      registry.gitlab.com/gitlab-org/security-products/analyzers/license-finder:latest
Gilbert Roulot's avatar
Gilbert Roulot committed
21 22
    ```

mo's avatar
mo committed
23
1. The results will be stored in the `gl-license-scanning-report.json` file in the application directory.
Gilbert Roulot's avatar
Gilbert Roulot committed
24

Gilbert Roulot's avatar
Gilbert Roulot committed
25 26
## Development

mo's avatar
mo committed
27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
### Getting Started

Install [Docker](https://docs.docker.com/engine/installation/).

```sh
$ git clone git@gitlab.com:gitlab-org/security-products/analyzers/license-finder.git
$ cd license-finder
```

Download the latest version of the Docker image.

```sh
$ ./bin/docker-pull
```

Launch a shell in the Docker container.

```sh
$ ./bin/docker-shell
```

Gilbert Roulot's avatar
Gilbert Roulot committed
48 49
### Running the application

50
License Management is a Docker image. You can build it like this from the project root:
Gilbert Roulot's avatar
Gilbert Roulot committed
51 52

```sh
53
$ ./bin/docker-build
Gilbert Roulot's avatar
Gilbert Roulot committed
54 55 56 57 58
```

You can then run License Management on some target directory:

```sh
mo's avatar
mo committed
59
$ docker run --rm --volume "/path/to/my/project":/code --env CI_PROJECT_DIR=/code
Gilbert Roulot's avatar
Gilbert Roulot committed
60 61
```

mo's avatar
mo committed
62 63
### Running the tests

64
You can run the tests from inside a docker container:
Gilbert Roulot's avatar
Gilbert Roulot committed
65 66

```sh
67
$ ./bin/docker-shell
68
$ ./bin/setup
Tetiana Chupryna's avatar
Tetiana Chupryna committed
69
$ ./bin/test [path to file]
70 71 72 73 74 75
```

If you need to debug any specific issues you can do this from within the docker container by
following these steps:

```sh
76 77 78
$ ./bin/docker-shell
$ enable_dev_mode
$ bundle open license_finder
79 80
```

mo's avatar
mo committed
81
The `docker-shell` script will mount the current project as a volume into `/builds/gitlab-org/security-products/analyzers/license-finder`.
82 83
This allows you to edit code from your host machine using your preferred editor and
see the affect of those changes from within the running docker container.
mo's avatar
mo committed
84

mo's avatar
mo committed
85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103
### Building Debian packages

Debian packages for each tool is built using [omnibus](https://github.com/chef/omnibus).
Read the [documentation](https://github.com/chef/omnibus#-omnibus) for more information.
The list of projects that can be built can be found in `config/projects/`.

E.g.

To build the `golang` package follow these steps:

```sh
$ ./bin/docker-omnibus
$ GOLANG_VERSION=1.15.3 ./bin/omnibus build golang
```

When a new version of a tool needs to be built, add
the new version to the tool file located in `./config/software/asdf_<tool>.rb`
and include the appropriate checksum.

mo's avatar
mo committed
104 105 106 107 108 109
### Updating the SPDX index

We will need to periodically update the SPDX index. This can be achieved with
the following command.

```bash
110
$ ./bin/update-spdx
mo's avatar
mo committed
111 112
```

Igor Frenkel's avatar
Igor Frenkel committed
113 114 115 116 117 118
#### Testing policies from the app-side

License Compliance requires the `spdx` database to be loaded in order for the licenses detected by `license-finder` to be matched to policies in the GitLab application.

If you are working locally, you need to import these licenses into the app. You can do so from console via: `ImportSoftwareLicensesWorker.new.perform`

Gilbert Roulot's avatar
Gilbert Roulot committed
119 120 121 122 123 124
## Supported languages and package managers

The following table shows which languages and package managers are supported.

| Language   | Package managers                                                  |
|------------|-------------------------------------------------------------------|
mo's avatar
mo committed
125
| .NET       | [.NET Core CLI][dotnet_core], [Nuget][nuget]                      |
mo's avatar
mo committed
126
| C/C++      | [Conan][conan]                                                    |
mo's avatar
mo committed
127
| Go         | [Go modules][gomod], [Godep][godep], go get                       |
mo's avatar
mo committed
128
| Java       | [Gradle][gradle], [Maven v3.2.5+][maven]                          |
mo's avatar
mo committed
129 130 131 132
| JavaScript | [npm][npm], [yarn][yarn], [Bower][bower]                          |
| PHP        | [composer][composer]                                              |
| Python     | [pip][pip], [pipenv][pipenv]                                      |
| Ruby       | [Bundler][bundler]                                                |
Gilbert Roulot's avatar
Gilbert Roulot committed
133

134 135 136
Inject `SETUP_CMD` to the docker command to override the given package managers
and run your custom command to setup your environment with a custom package manager.

137 138 139 140 141
```sh
docker run \
  --volume "$PWD":/code \
  --env "SETUP_CMD=./my-custom-install-script.sh" \
  --rm \
mo's avatar
mo committed
142
  registry.gitlab.com/gitlab-org/security-products/analyzers/license-finder:latest analyze /code
143
```
144

145 146
## Settings

Can Eldem's avatar
Can Eldem committed
147
The License Management tool can be customized with environments variables for some project types.
148 149 150

| Environment variable | Project type | Function |
|----------------------|--------------|----------|
mo's avatar
mo committed
151
| ADDITIONAL_CA_CERT_BUNDLE | * | Additional certificate chain to install in the trusted store. |
152
| MAVEN_CLI_OPTS       | Java (Maven) | Additional arguments for the mvn executable. If not supplied, defaults to `-DskipTests`. |
153
| LICENSE_FINDER_CLI_OPTS | * | Additional arguments for the `license_finder` executable. |
154 155
| LM_JAVA_VERSION      | Java (Maven) | Version of Java. If set to `11`, Maven and Gradle use Java 11 instead of Java 8. |
| LM_PYTHON_VERSION    | Python       | Version of Python. If set to `3`, dependencies are installed using Python 3 instead of Python 2.7. |
mo's avatar
mo committed
156
| LOG_LEVEL    | * | Control the verbosity of the logs. (`debug`, `info`, `warn` (default), `error`, `fatal`)  |
157
| LM_REPORT_FILE    | * | Name of the generated report. If not supplied, defaults to `gl-license-scanning-report.json`  |
158 159 160 161 162


Inject the required environment variables to the docker command using the [`--env` option flag](https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file)
or its shorthand form (`--env MY_SETTING_VAR`) if the configuration comes from an external environment.

Gilbert Roulot's avatar
Gilbert Roulot committed
163 164
## Versioning and release process

165
Please check the [Release Process documentation](https://gitlab.com/gitlab-org/security-products/release/blob/main/docs/release_process.md).
Gilbert Roulot's avatar
Gilbert Roulot committed
166

167
## How to update the upstream Scanner
mo's avatar
mo committed
168

169 170
1. Check for the latest version of `LicenseFinder` at [https://rubygems.org/gems/license_finder][license_finder]
1. Check the version of the `license_finder` gem that is currently being used in the [`Gemfile.lock`][gemfile_lock]
mo's avatar
mo committed
171
1. If an update is available, create a new branch
172 173
1. Bump the license management version in [CHANGELOG.md][changelog] and in [version.rb][version_rb]
1. Update the `license_finder` version constraint in the [gemspec][gemspec]
mo's avatar
mo committed
174
1. Run `bundle update license_finder`
mo's avatar
mo committed
175
1. Test the changes by following the instructions for [running the tests](#running-the-tests)
mo's avatar
mo committed
176 177
1. Submit a merge request.

mo's avatar
mo committed
178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319
## Step by Step: Detection

1. Run the Docker image:

    ```sh
    docker run \
      --volume "$PWD":/code \
      --env=LM_REPORT_VERSION="2.1" \
      --env=CI_PROJECT_DIR=/code \
      registry.gitlab.com/gitlab-org/security-products/analyzers/license-finder:latest
    ```

1. The `ENTRYPOINT` for the container will execute [run.sh](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/-/blob/191185c4303768c6d9a1431c35143501c06ee4d7/run.sh):

    ```Dockerfile
    ENTRYPOINT ["/run.sh"]
    ```

1. This shell script sets up the runtime environment then invokes the `license_management` executable:

    ```sh
    #!/bin/bash -l
    export LM_JAVA_VERSION=${LM_JAVA_VERSION:-"8"}
    export LM_PYTHON_VERSION=${LM_PYTHON_VERSION:-"3"}
    export LM_REPORT_FILE=${LM_REPORT_FILE:-'gl-license-scanning-report.json'}
    ...
    license_management report $@
    ```

1. The `license_management` executable loads monkey patches for [license_finder][license_finder] then invokes the CLI:

    ```ruby
    require 'license/management'

    LicenseFinder::CLI::Main.start(ARGV)
    ```

1. [license_finder][license_finder] searches for lockfiles in the project.

    ```ruby
    def active?
      project_path.join('pom.xml').exist?
    end
    ```

1. When a [license_finder][license_finder] determines that a package manager is active, it then invokes the `prepare` step for that package manager.

    ```ruby
    def prepare
      within_project_path do
        tool_box.install(tool: :java, version: java_version, env: default_env)
      end
    end
    ```

1. The `tool_box` determines the required version of tools (i.e Java, Ruby, Python etc) for the package manager and then installs it by looking in `/opt/toolcache/` for a matching `*.deb` file or falls back to `asdf` to install the tool from source.

    ```ruby
    def install(tool:, version: , env: {})
      Dir.chdir project_path do
        deb = deb_for(tool, version)
        shell.execute([:dpkg, '-i', deb]) if deb&.exist?
        shell.execute([:asdf, :install, tool.to_s, version], env: env)
      end
    end

    def deb_for(tool, version)
      Pathname.glob("/opt/toolcache/#{tool}-#{version}*.deb")[0]
    end
    ```

1. After the tool(s) are installed the package manager class builds a list of dependencies identified in the project. If an `install_path` is provided then the files in this directory are scanned for software licenses.

    ```ruby
    def current_packages
      within_project_path do
        return [] unless shell.execute(detect_licenses_command, env: default_env)[-1].success?

        resource_files.flat_map { |file| map_from(file.read) }.uniq
      end
    end
    ```

1. Once all the dependencies and their licenses are identified a JSON report is generated for the desired version of the report. The `Report` class is backwards compatible and able to generate any previous version of the report.

    ```ruby
    def to_s
      JSON.pretty_generate(version_for(report_version).to_h)
    end

    def version_for(version)
      VERSIONS.fetch(version.to_s).new(dependencies)
    end
    ```

1. The final JSON report is written to [gl-license-scanning-report.json](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/-/blob/191185c4303768c6d9a1431c35143501c06ee4d7/spec/fixtures/schema/v2.1.json) in the root of the project.

    ```json
    {
      "version": "2.1",
      "licenses": [
        {
          "id": "MPL-2.0",
          "name": "Mozilla Public License 2.0",
          "url": "https://opensource.org/licenses/MPL-2.0"
        }
      ],
      "dependencies": [
        {
          "name": "rhino",
          "version": "1.7.10",
          "package_manager": "maven",
          "path": "pom.xml",
          "licenses": [
            "MPL-2.0"
          ]
        }
      ]
    }
    ```

For additional information watch:

* License Compliance Past, Present and Future: https://youtu.be/j2TguACMvho
* Overview of the license approvals: https://youtu.be/e0qfNbnnI4c
* Overview of how license scanning works in GitLab CI: https://youtu.be/biC1t-7bMhg
* How to use the OSS Review Toolkit: https://youtu.be/dNmH_kYJ34g

## Opportunities for improvement

* [ ] Cache the [`.gitlab/cache`][cache-dir] directory in [License-Scanning.gitlab-ci.yml][ci-template] to speed up `prepare` step
* [ ] Make `--recursive` scan the default
* [ ] Override the `nodejs` plugin to install the [Linux Binaries][nodejs-binaries] instead of compiling from source code.
* [ ] Replace license detection engine with [license classifier][google-classifier] or [licensee][licensee]
* [ ] Run `prepare` step for each active package manager in parallel
* [ ] Store Debian packages for each tool in hosted Debian repository instead of storing in `/opt/toolcache`.
* Alternatives to consider:
  * [Google License Classifier][google-classifier]
  * [Licensed][licensed]
  * [ORT][ort]
  * [Scout][scout]

Gilbert Roulot's avatar
Gilbert Roulot committed
320 321 322
# Contributing

If you want to help, read the [contribution guidelines](CONTRIBUTING.md).
323 324

If an unknown license is detected, please consider updating the mapping defined
325
in [normalized-licenses.yml](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/blob/main/normalized-licenses.yml). A mapping can be for a detected name or url and must correspond to an SPDX identifier found in [spdx-licenses.json](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/blob/main/spdx-licenses.json).
326

mo's avatar
mo committed
327
[bower]: https://bower.io/
mo's avatar
mo committed
328
[bundler]: https://bundler.io/
mo's avatar
mo committed
329
[cache-dir]: https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/-/blob/191185c4303768c6d9a1431c35143501c06ee4d7/lib/license/finder/ext/package_manager.rb#L38
330
[changelog]: https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/-/blob/main/CHANGELOG.md
mo's avatar
mo committed
331
[ci-template]: https://gitlab.com/gitlab-org/gitlab/-/blob/7385c6a598bae6d28973dcd1a8c436511011ef16/lib/gitlab/ci/templates/Security/License-Scanning.gitlab-ci.yml#L15-35
mo's avatar
mo committed
332 333 334
[composer]: https://getcomposer.org
[conan]: https://conan.io/
[dotnet_core]: https://docs.microsoft.com/en-us/dotnet/core/tools/
335 336
[gemfile_lock]: https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/-/blob/main/Gemfile.lock
[gemspec]: https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/-/blob/main/license-management.gemspec
mo's avatar
mo committed
337
[godep]: https://github.com/tools/godep
mo's avatar
mo committed
338
[gomod]: https://github.com/golang/go/wiki/Modules
mo's avatar
mo committed
339
[google-classifier]: https://github.com/google/licenseclassifier
mo's avatar
mo committed
340
[gradle]: https://gradle.org/
mo's avatar
mo committed
341
[license_finder]: https://rubygems.org/gems/license_finder
mo's avatar
mo committed
342 343
[licensed]: https://github.com/github/licensed
[licensee]: https://github.com/licensee/licensee
mo's avatar
mo committed
344
[maven]: https://maven.apache.org/
mo's avatar
mo committed
345
[nodejs-binaries]: https://nodejs.org/en/download/
mo's avatar
mo committed
346
[npm]: https://www.npmjs.com/
mo's avatar
mo committed
347
[nuget]: https://www.nuget.org/
mo's avatar
mo committed
348
[ort]: https://github.com/oss-review-toolkit/ort
mo's avatar
mo committed
349 350
[pip]: https://pip.pypa.io/en/stable/
[pipenv]: https://github.com/pypa/pipenv
mo's avatar
mo committed
351
[scout]: https://github.com/chef/license_scout
352
[version_rb]: https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/-/blob/main/lib/license/management/version.rb
mo's avatar
mo committed
353
[yarn]: https://yarnpkg.com/