Commit d6f393fb authored by mo's avatar mo

Update Documentation

parent 191185c4
......@@ -15,14 +15,36 @@ only, but this may change in the future.
docker run \
--volume "$PWD":/code \
--rm \ analyze /code
--env=LM_REPORT_VERSION="2.1" \
--env=CI_PROJECT_DIR=/code \
1. The results will be stored in the `gl-license-scanning-report.json` file in the application directory.
## Development
### Getting Started
Install [Docker](
$ git clone
$ cd license-finder
Download the latest version of the Docker image.
$ ./bin/docker-pull
Launch a shell in the Docker container.
$ ./bin/docker-shell
### Running the application
License Management is a Docker image. You can build it like this from the project root:
......@@ -42,7 +64,6 @@ $ docker run --rm --volume "/path/to/my/project":/code --env CI_PROJECT_DIR=/cod
You can run the tests from inside a docker container:
$ ./bin/docker-build
$ ./bin/docker-shell
$ ./bin/setup
$ ./bin/test [path to file]
......@@ -52,7 +73,6 @@ If you need to debug any specific issues you can do this from within the docker
following these steps:
$ ./bin/docker-build
$ ./bin/docker-shell
$ enable_dev_mode
$ bundle open license_finder
......@@ -62,6 +82,25 @@ The `docker-shell` script will mount the current project as a volume into `/buil
This allows you to edit code from your host machine using your preferred editor and
see the affect of those changes from within the running docker container.
### Building Debian packages
Debian packages for each tool is built using [omnibus](
Read the [documentation]( for more information.
The list of projects that can be built can be found in `config/projects/`.
To build the `golang` package follow these steps:
$ ./bin/docker-omnibus
$ GOLANG_VERSION=1.15.3 ./bin/omnibus build golang
When a new version of a tool needs to be built, add
the new version to the tool file located in `./config/software/asdf_<tool>.rb`
and include the appropriate checksum.
### Updating the SPDX index
We will need to periodically update the SPDX index. This can be achieved with
......@@ -80,7 +119,7 @@ The following table shows which languages and package managers are supported.
| .NET | [.NET Core CLI][dotnet_core], [Nuget][nuget] |
| C/C++ | [Conan][conan] |
| Go | [Go modules][gomod], [Godep][godep], go get |
| Java | [Gradle][gradle], [Maven v3.2.5+)][maven] |
| Java | [Gradle][gradle], [Maven v3.2.5+][maven] |
| JavaScript | [npm][npm], [yarn][yarn], [Bower][bower] |
| PHP | [composer][composer] |
| Python | [pip][pip], [pipenv][pipenv] |
......@@ -130,6 +169,148 @@ Please check the [Release Process documentation](
1. Test the changes by following the instructions for [running the tests](#running-the-tests)
1. Submit a merge request.
## Step by Step: Detection
1. Run the Docker image:
docker run \
--volume "$PWD":/code \
--env=LM_REPORT_VERSION="2.1" \
--env=CI_PROJECT_DIR=/code \
1. The `ENTRYPOINT` for the container will execute [](
1. This shell script sets up the runtime environment then invokes the `license_management` executable:
#!/bin/bash -l
export LM_REPORT_FILE=${LM_REPORT_FILE:-'gl-license-scanning-report.json'}
license_management report $@
1. The `license_management` executable loads monkey patches for [license_finder][license_finder] then invokes the CLI:
require 'license/management'
1. [license_finder][license_finder] searches for lockfiles in the project.
def active?
1. When a [license_finder][license_finder] determines that a package manager is active, it then invokes the `prepare` step for that package manager.
def prepare
within_project_path do
tool_box.install(tool: :java, version: java_version, env: default_env)
1. The `tool_box` determines the required version of tools (i.e Java, Ruby, Python etc) for the package manager and then installs it by looking in `/opt/toolcache/` for a matching `*.deb` file or falls back to `asdf` to install the tool from source.
def install(tool:, version: , env: {})
Dir.chdir project_path do
deb = deb_for(tool, version)
shell.execute([:dpkg, '-i', deb]) if deb&.exist?
shell.execute([:asdf, :install, tool.to_s, version], env: env)
def deb_for(tool, version)
1. After the tool(s) are installed the package manager class builds a list of dependencies identified in the project. If an `install_path` is provided then the files in this directory are scanned for software licenses.
def current_packages
within_project_path do
return [] unless shell.execute(detect_licenses_command, env: default_env)[-1].success?
resource_files.flat_map { |file| map_from( }.uniq
1. Once all the dependencies and their licenses are identified a JSON report is generated for the desired version of the report. The `Report` class is backwards compatible and able to generate any previous version of the report.
def to_s
def version_for(version)
1. The final JSON report is written to [gl-license-scanning-report.json]( in the root of the project.
"version": "2.1",
"licenses": [
"id": "MPL-2.0",
"name": "Mozilla Public License 2.0",
"url": ""
"dependencies": [
"name": "rhino",
"version": "1.7.10",
"package_manager": "maven",
"path": "pom.xml",
"licenses": [
For additional information watch:
* License Compliance Past, Present and Future:
* Overview of the license approvals:
* Overview of how license scanning works in GitLab CI:
* How to use the OSS Review Toolkit:
## Opportunities for improvement
* [ ] Cache the [`.gitlab/cache`][cache-dir] directory in [License-Scanning.gitlab-ci.yml][ci-template] to speed up `prepare` step
* [ ] Make `--recursive` scan the default
* [ ] Override the `nodejs` plugin to install the [Linux Binaries][nodejs-binaries] instead of compiling from source code.
* [ ] Replace license detection engine with [license classifier][google-classifier] or [licensee][licensee]
* [ ] Run `prepare` step for each active package manager in parallel
* [ ] Store Debian packages for each tool in hosted Debian repository instead of storing in `/opt/toolcache`.
* Alternatives to consider:
* [Google License Classifier][google-classifier]
* [Licensed][licensed]
* [ORT][ort]
* [Scout][scout]
# Contributing
If you want to help, read the [contribution guidelines](
......@@ -139,7 +320,9 @@ in [normalized-licenses.yml](
......@@ -147,12 +330,18 @@ in [normalized-licenses.yml](
......@@ -106,7 +106,8 @@ RSpec.describe "bundler" do
specify do
# This test is flaky. The reverse proxy to periodically fails.
xspecify do
expect(subject).to match_schema
expect(subject.dependency_names).to include("net-hippie")
expect(subject.licenses_for('net-hippie')).to match_array('MIT')
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment