# GitLab License Management

[![pipeline status](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/badges/main/pipeline.svg)](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/commits/main)
[![coverage report](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/badges/main/coverage.svg)](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/commits/main)

GitLab tool for detecting licenses of the dependencies used by the provided source.
It is currently based on [License Finder][license_finder]
only, but this may change in the future.

## How to use

1. `cd` into the directory of the source code you want to scan
1. Run the Docker image:

    ```sh
    docker run \
      --volume "$PWD":/code \
      --env=LM_REPORT_VERSION="2.1" \
      --env=CI_PROJECT_DIR=/code \
      registry.gitlab.com/gitlab-org/security-products/analyzers/license-finder:latest
    ```

1. The results will be stored in the `gl-license-scanning-report.json` file in the application directory.

## Development

### Getting Started

Install [Docker](https://docs.docker.com/engine/installation/).

```sh
$ git clone git@gitlab.com:gitlab-org/security-products/analyzers/license-finder.git
$ cd license-finder
```

Download the latest version of the Docker image.

```sh
$ ./bin/docker-pull
```

Launch a shell in the Docker container.

```sh
$ ./bin/docker-shell
```

### Running the application

License Management is a Docker image. You can build it like this from the project root:

```sh
$ ./bin/docker-build
```

You can then run License Management on some target directory:

```sh
$ docker run --rm --volume "/path/to/my/project":/code --env CI_PROJECT_DIR=/code
```

### Running the tests

You can run the tests from inside a docker container:

```sh
$ ./bin/docker-shell
$ ./bin/setup
$ ./bin/test [path to file]
```

If you need to debug any specific issues you can do this from within the docker container by
following these steps:

```sh
$ ./bin/docker-shell
$ enable_dev_mode
$ bundle open license_finder
```

The `docker-shell` script will mount the current project as a volume into `/builds/gitlab-org/security-products/analyzers/license-finder`.
This allows you to edit code from your host machine using your preferred editor and
see the affect of those changes from within the running docker container.

### Building Debian packages

Debian packages for each tool is built using [omnibus](https://github.com/chef/omnibus).
Read the [documentation](https://github.com/chef/omnibus#-omnibus) for more information.
The list of projects that can be built can be found in `config/projects/`.

E.g.

To build the `golang` package follow these steps:

```sh
$ ./bin/docker-omnibus
$ GOLANG_VERSION=1.15.3 ./bin/omnibus build golang
```

When a new version of a tool needs to be built, add
the new version to the tool file located in `./config/software/asdf_<tool>.rb`
and include the appropriate checksum.

### Updating the SPDX index

We will need to periodically update the SPDX index. This can be achieved with
the following command.

```bash
$ ./bin/update-spdx
```

#### Testing policies from the app-side

License Compliance requires the `spdx` database to be loaded in order for the licenses detected by `license-finder` to be matched to policies in the GitLab application.

If you are working locally, you need to import these licenses into the app. You can do so from console via: `ImportSoftwareLicensesWorker.new.perform`

## Supported languages and package managers

The following table shows which languages and package managers are supported.

| Language   | Package managers                                                  |
|------------|-------------------------------------------------------------------|
| .NET       | [.NET Core CLI][dotnet_core], [Nuget][nuget]                      |
| C/C++      | [Conan][conan]                                                    |
| Go         | [Go modules][gomod], [Godep][godep], go get                       |
| Java       | [Gradle][gradle], [Maven v3.2.5+][maven]                          |
| JavaScript | [npm][npm], [yarn][yarn], [Bower][bower]                          |
| PHP        | [composer][composer]                                              |
| Python     | [pip][pip], [pipenv][pipenv]                                      |
| Ruby       | [Bundler][bundler]                                                |

Inject `SETUP_CMD` to the docker command to override the given package managers
and run your custom command to setup your environment with a custom package manager.

```sh
docker run \
  --volume "$PWD":/code \
  --env "SETUP_CMD=./my-custom-install-script.sh" \
  --rm \
  registry.gitlab.com/gitlab-org/security-products/analyzers/license-finder:latest analyze /code
```

## Settings

The License Management tool can be customized with environments variables for some project types.

| Environment variable | Project type | Function |
|----------------------|--------------|----------|
| ADDITIONAL_CA_CERT_BUNDLE | * | Additional certificate chain to install in the trusted store. |
| MAVEN_CLI_OPTS       | Java (Maven) | Additional arguments for the mvn executable. If not supplied, defaults to `-DskipTests`. |
| LICENSE_FINDER_CLI_OPTS | * | Additional arguments for the `license_finder` executable. |
| LM_JAVA_VERSION      | Java (Maven) | Version of Java. If set to `11`, Maven and Gradle use Java 11 instead of Java 8. |
| LM_PYTHON_VERSION    | Python       | Version of Python. If set to `3`, dependencies are installed using Python 3 instead of Python 2.7. |
| LOG_LEVEL    | * | Control the verbosity of the logs. (`debug`, `info`, `warn` (default), `error`, `fatal`)  |
| LM_REPORT_FILE    | * | Name of the generated report. If not supplied, defaults to `gl-license-scanning-report.json`  |


Inject the required environment variables to the docker command using the [`--env` option flag](https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file)
or its shorthand form (`--env MY_SETTING_VAR`) if the configuration comes from an external environment.

## Versioning and release process

Please check the [Release Process documentation](https://gitlab.com/gitlab-org/security-products/release/blob/main/docs/release_process.md).

## How to update the upstream Scanner

1. Check for the latest version of `LicenseFinder` at [https://rubygems.org/gems/license_finder][license_finder]
1. Check the version of the `license_finder` gem that is currently being used in the [`Gemfile.lock`][gemfile_lock]
1. If an update is available, create a new branch
1. Bump the license management version in [CHANGELOG.md][changelog] and in [version.rb][version_rb]
1. Update the `license_finder` version constraint in the [gemspec][gemspec]
1. Run `bundle update license_finder`
1. Test the changes by following the instructions for [running the tests](#running-the-tests)
1. Submit a merge request.

## Step by Step: Detection

1. Run the Docker image:

    ```sh
    docker run \
      --volume "$PWD":/code \
      --env=LM_REPORT_VERSION="2.1" \
      --env=CI_PROJECT_DIR=/code \
      registry.gitlab.com/gitlab-org/security-products/analyzers/license-finder:latest
    ```

1. The `ENTRYPOINT` for the container will execute [run.sh](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/-/blob/191185c4303768c6d9a1431c35143501c06ee4d7/run.sh):

    ```Dockerfile
    ENTRYPOINT ["/run.sh"]
    ```

1. This shell script sets up the runtime environment then invokes the `license_management` executable:

    ```sh
    #!/bin/bash -l
    export LM_JAVA_VERSION=${LM_JAVA_VERSION:-"8"}
    export LM_PYTHON_VERSION=${LM_PYTHON_VERSION:-"3"}
    export LM_REPORT_FILE=${LM_REPORT_FILE:-'gl-license-scanning-report.json'}
    ...
    license_management report $@
    ```

1. The `license_management` executable loads monkey patches for [license_finder][license_finder] then invokes the CLI:

    ```ruby
    require 'license/management'

    LicenseFinder::CLI::Main.start(ARGV)
    ```

1. [license_finder][license_finder] searches for lockfiles in the project.

    ```ruby
    def active?
      project_path.join('pom.xml').exist?
    end
    ```

1. When a [license_finder][license_finder] determines that a package manager is active, it then invokes the `prepare` step for that package manager.

    ```ruby
    def prepare
      within_project_path do
        tool_box.install(tool: :java, version: java_version, env: default_env)
      end
    end
    ```

1. The `tool_box` determines the required version of tools (i.e Java, Ruby, Python etc) for the package manager and then installs it by looking in `/opt/toolcache/` for a matching `*.deb` file or falls back to `asdf` to install the tool from source.

    ```ruby
    def install(tool:, version: , env: {})
      Dir.chdir project_path do
        deb = deb_for(tool, version)
        shell.execute([:dpkg, '-i', deb]) if deb&.exist?
        shell.execute([:asdf, :install, tool.to_s, version], env: env)
      end
    end

    def deb_for(tool, version)
      Pathname.glob("/opt/toolcache/#{tool}-#{version}*.deb")[0]
    end
    ```

1. After the tool(s) are installed the package manager class builds a list of dependencies identified in the project. If an `install_path` is provided then the files in this directory are scanned for software licenses.

    ```ruby
    def current_packages
      within_project_path do
        return [] unless shell.execute(detect_licenses_command, env: default_env)[-1].success?

        resource_files.flat_map { |file| map_from(file.read) }.uniq
      end
    end
    ```

1. Once all the dependencies and their licenses are identified a JSON report is generated for the desired version of the report. The `Report` class is backwards compatible and able to generate any previous version of the report.

    ```ruby
    def to_s
      JSON.pretty_generate(version_for(report_version).to_h)
    end

    def version_for(version)
      VERSIONS.fetch(version.to_s).new(dependencies)
    end
    ```

1. The final JSON report is written to [gl-license-scanning-report.json](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/-/blob/191185c4303768c6d9a1431c35143501c06ee4d7/spec/fixtures/schema/v2.1.json) in the root of the project.

    ```json
    {
      "version": "2.1",
      "licenses": [
        {
          "id": "MPL-2.0",
          "name": "Mozilla Public License 2.0",
          "url": "https://opensource.org/licenses/MPL-2.0"
        }
      ],
      "dependencies": [
        {
          "name": "rhino",
          "version": "1.7.10",
          "package_manager": "maven",
          "path": "pom.xml",
          "licenses": [
            "MPL-2.0"
          ]
        }
      ]
    }
    ```

For additional information watch:

* License Compliance Past, Present and Future: https://youtu.be/j2TguACMvho
* Overview of the license approvals: https://youtu.be/e0qfNbnnI4c
* Overview of how license scanning works in GitLab CI: https://youtu.be/biC1t-7bMhg
* How to use the OSS Review Toolkit: https://youtu.be/dNmH_kYJ34g

## Opportunities for improvement

* [ ] Cache the [`.gitlab/cache`][cache-dir] directory in [License-Scanning.gitlab-ci.yml][ci-template] to speed up `prepare` step
* [ ] Make `--recursive` scan the default
* [ ] Override the `nodejs` plugin to install the [Linux Binaries][nodejs-binaries] instead of compiling from source code.
* [ ] Replace license detection engine with [license classifier][google-classifier] or [licensee][licensee]
* [ ] Run `prepare` step for each active package manager in parallel
* [ ] Store Debian packages for each tool in hosted Debian repository instead of storing in `/opt/toolcache`.
* Alternatives to consider:
  * [Google License Classifier][google-classifier]
  * [Licensed][licensed]
  * [ORT][ort]
  * [Scout][scout]

# Contributing

If you want to help, read the [contribution guidelines](CONTRIBUTING.md).

If an unknown license is detected, please consider updating the mapping defined
in [normalized-licenses.yml](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/blob/main/normalized-licenses.yml). A mapping can be for a detected name or url and must correspond to an SPDX identifier found in [spdx-licenses.json](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/blob/main/spdx-licenses.json).

[bower]: https://bower.io/
[bundler]: https://bundler.io/
[cache-dir]: https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/-/blob/191185c4303768c6d9a1431c35143501c06ee4d7/lib/license/finder/ext/package_manager.rb#L38
[changelog]: https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/-/blob/main/CHANGELOG.md
[ci-template]: https://gitlab.com/gitlab-org/gitlab/-/blob/7385c6a598bae6d28973dcd1a8c436511011ef16/lib/gitlab/ci/templates/Security/License-Scanning.gitlab-ci.yml#L15-35
[composer]: https://getcomposer.org
[conan]: https://conan.io/
[dotnet_core]: https://docs.microsoft.com/en-us/dotnet/core/tools/
[gemfile_lock]: https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/-/blob/main/Gemfile.lock
[gemspec]: https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/-/blob/main/license-management.gemspec
[godep]: https://github.com/tools/godep
[gomod]: https://github.com/golang/go/wiki/Modules
[google-classifier]: https://github.com/google/licenseclassifier
[gradle]: https://gradle.org/
[license_finder]: https://rubygems.org/gems/license_finder
[licensed]: https://github.com/github/licensed
[licensee]: https://github.com/licensee/licensee
[maven]: https://maven.apache.org/
[nodejs-binaries]: https://nodejs.org/en/download/
[npm]: https://www.npmjs.com/
[nuget]: https://www.nuget.org/
[ort]: https://github.com/oss-review-toolkit/ort
[pip]: https://pip.pypa.io/en/stable/
[pipenv]: https://github.com/pypa/pipenv
[scout]: https://github.com/chef/license_scout
[version_rb]: https://gitlab.com/gitlab-org/security-products/analyzers/license-finder/-/blob/main/lib/license/management/version.rb
[yarn]: https://yarnpkg.com/