## Executive Summary SPDX license expressions present a flexible method of declaring a license. An expression can consist of: * Compound expressions — denoted by `AND`, `OR`, and parenthesised groupings (e.g. `MIT OR (Apache-2.0 AND GPL-2.0-only)`) * Exception qualifiers — denoted by `WITH` followed by a valid SPDX exception identifier (e.g. `GPL-2.0-only WITH Classpath-exception-2.0`) * Or-later — denoted by the `+` suffix (e.g. `GPL-2.0+`) * License references — `LicenseRef-`, `AdditionRef-`, and `DocumentRef-` prefixes Gitlab should be able to show SPDX license expressions in the dependency list and also have policies created around their usage. Currently license expressions are shown as `unknown`, which prevents users from fully leveraging Dependency / license scanning and tangential functionality. ### Problem to solve License scanning and license approval policies only recognize singular licenses. This creates a challenge for customers that utilize software that uses more than one license, allows dependents to choose from an option of licenses, or if exceptions are given to a license requirements. Currently we return composite licenses by simply identifying them as `UNKNOWN`, which is misinformation and leads to customers needing to manually examine licenses attributed to their dependencies. Beyond the mislabeling of these composite licenses, users are unable to define policies that incorporate this type of licensing. ### Proposal * Composite licenses should be identified as a part of a Dependency scan (license scanning) * Composite licenses should not be returned as `UNKNOWN` unless the license string contains a license that is not identifiable * Users should be able to define a policy to handle composite licenses - this will focus on augmenting license approval policies to permit users to allow/deny composite licenses as they currently do for singularly licensed dependencies. * ~Stretch Supporting `LicenseRef-*` as mentioned on [this thread](https://gitlab.com/gitlab-org/gitlab/-/work_items/594303#note_3263913138). ### User experience goals #### Dependency list * Users are able to view composite licenses * Users are able to filter on licenses * Composite licenses will appear in the Licenses column and include the operator - `AND`, `OR`, `WITH` #### License approval policies * Users can select to `Choose a license` that is a composite license within the `Edit allowlist` user interface * When defining `License Type` criteria users can select a composite license within the `Select license types` dropdown ### Metrics 1. Licenses identified as Composite 2. Policies created that incorporate composite licenses ### Engineering Assessment Several discrete components are involved and the research and implementation decisions in some system components affect those that follow. Therefore, the proposed solution must have several steps. 1. `pmdb`. Add support for spdx expressions in `license-processor` and update database to allow expression storage. Additionally, `license-exporter` will need to be updated (minimally). The exported data format should stay the same. 2. `monolith package_metadata sync`. Update `package_metadata` ingestion and the `PackageMetadata::Package` model to support spdx_expressions. 3. `monolith license policies`. Add license expression evaluation capability to `SCA::LicenseCompliance` and related models so that allow/deny lists can evaluate dependency licenses. There is potentially a stage 4 to backfill data for packages which had spdx expressions in their licenses and were classified as unknown. However, because the package registry specific feeders have different ways in which they run from start, the backfill is likely out of scope for this epic. ### Stage 1: pmdb `pmdb` feeders/interfacers pass raw license data for a package and its version to the `license-processor`. The processor is responsible for classifying the license and matching it to a known spdx identifier. The identifier is converted to an internal database id and saved alongside the `package` and `version`. Proposed solution: - `db`: add spdx expression string column - `license-processor`: add logic for identifying and validating an spdx expression - `license-processor`: insert spdx expression into db - `license-exporter`: conditionally fetch the spdx license name or the expression based on which column is populated ### Stage 2: package_metadata sync TBC ### Stage 3: license policy classification TBC ## References - [design document](https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/complex_spdx_expression_evaluation/) - Needs to be updated to incorporate the evaluation engine that does **not** use bitmasks. - [proof of concept](https://gitlab.com/hacks4oats/spdx-license-evaluator) - This is a small project I wrote to parse the expressions. There's some stuff that needs to be done like parsing the entire list of known SPDX licenses, and the evaluation piece. These portions are very complex in my opinion. The bigger portion is compiling for Ruby interfacing with Magnus, and possibly for WASM if there's a playground aspect that we want to play out with this. E.g. fetch all the licenses of a project, and test a new policy against them in the playground to see what the result is like. ### Dependencies * Team dependencies: * \~"group::security insights" EM: @nmccorrison PM: @mclausen35 * \~"group::security policies" EM: @alan PM: @g.hickman * Epic/issue dependencies: * \~"group::security insights" https://gitlab.com/gitlab-org/gitlab/-/issues/551131 * \~"group::security policies" https://gitlab.com/gitlab-org/gitlab/-/issues/424827 * External dependencies: none ```glql display: table query: project = "gitlab-org/gitlab" AND id IN (551131, 424827) fields: title, state, milestone, assignee, health title: Epic Dependency Tracking ``` ### DRIs * Engineer: @ifrenkel * EM: @nilieskou * PM: @johncrowley * Engineering Owner: @twoodham #### Initiative Driver - Product or Engineering? * [x] **Product-driven initiatives (P1/P2/P3)** - Customer-facing features or improvements driven by Product teams that require engineering resources and commitment * [ ] **Engineering-driven initiatives (E1/E2/E3)** - Internal technical improvements that may not have customer-facing components #### Sizing and Funding (Optional) * **Size**: * **Funding Status**: TBD, based on interlock feedback ### Hygiene Guidelines :bulb: \_See additional details about this process at https://handbook.gitlab.com/handbook/product-development/r-and-d-interlock/ ##### :one: Pre-Interlock * [x] Update epic description with all relevant information * [x] Ensure all dependencies are identified * [x] Apply appropriate labels (see below) * [ ] Apply target delivery Milestone * [ ] Update interlock status as discussions progress (via label) ##### :two: Post-Interlock: once quarter begins * Update health status weekly (via label) * Document any newly identified risks or dependencies * Link to implementation epics/issues as work begins * Flag any scope or timeline changes immediately ### Target Metric/s * Improved detection of licenses resulting in an increased % of licenses accurately detected, which can lead to an increase in adoption of SCA. _(Will update with data via comment section)_