MVC Detect vulnerabilities in vendored JavaScript libraries using Gemnasium

Why are we doing this work

Following the design discussions in Design: Detect vulnerabilities in vendored JavaScript libraries using Gemnasium we want to deliver a MVC of scanning capabilities for vendored JS libraries with our Gemnasium analyzer.

Relevant links

Proposal

As a first step, leverage Retire.js to make Gemnasium scan vendored JS libaries. Gemnasium executes retire --js against the root directory of the git repo being scanned. It does NOT install the project dependencies prior to the scan.

In gemnasium codebase, We introduce new Go struct type, functions, and methods to:

  • Find vendored libraries.
  • Find the vulnerabilities matching these vendored libraries, using Gemnasium's vulnerability database.
  • Add these vulnerabilities to the Dependency Scanning reports.

We can't reuse the existing high-level functions and type structs because they operate on dependency files, and not vendored libraries. Unlike dep. files, vendored libraries don't have list dependencies, and they're not associated with a package manager.

Please note that vendored libraries are not listed in the dependency_files field of the Dependency Scanning report b/c they're not dependency files, and shouldn't be presented as such in the UI.

The exact path of a vendored library can be used in the location.file field of the vulnerabilities, in the report.

Limitations

Gemnasium won't be able to report vulnerabilities for a vendored library if Retire.js doesn't have vulnerabilities for this library, in jsrepository.json.

Gemnasium accurately report the file path in .vulnerabilities[].location.file. This was not the case with the retire.js analyzer though: location.file was always set to package.json even when the vulnerable file was part of the git repo. As a result, vulnerability feedback might lost.

Gemnasium doesn't find vulnerable JS files in installed packages. This reduces the noise but it might be seen as a limitation.

Documentation

Performance

retire --js takes 40 seconds to process 540 MB and 63k files. See #288330 (comment 813516860)

Testing

Integration tests must cover these scenarios:

  • Scan of libraries is enabled.
  • Scan of libraries is enabled but some JS files are excluded.
  • retire command fails. (Error is reported.)
  • Scan of libraries is NOT enabled.

Implementation plan

To be implemented in the gemnasium project:

  • Add the retire command to the gemnasium image via the Dockerfile.
  • Introduce a new Go struct type that describes vendored libraries; it combines a file path with a package (type, name, and version).
  • Add Go package that finds JS vendored libraries using the retire --js command.
  • Update scanner package to scan vendored libraries, and report their vulnerabilities.
  • Update convert package to report vulnerabilities found in vendored libraries to the Dependency Scanning report.
  • Update main package to find, scan, and report JS vendored library.
  • Add an image test integration to check the scanning of JS vendored libraries.
  • Enable the feature in the CI template.
  • Update documentation.

Auto-Summary 🤖

Discoto Usage

Points

Discussion points are declared by headings, list items, and single lines that start with the text (case-insensitive) point:. For example, the following are all valid points:

  • #### POINT: This is a point
  • * point: This is a point
  • + Point: This is a point
  • - pOINT: This is a point
  • point: This is a **point**

Note that any markdown used in the point text will also be propagated into the topic summaries.

Topics

Topics can be stand-alone and contained within an issuable (epic, issue, MR), or can be inline.

Inline topics are defined by creating a new thread (discussion) where the first line of the first comment is a heading that starts with (case-insensitive) topic:. For example, the following are all valid topics:

  • # Topic: Inline discussion topic 1
  • ## TOPIC: **{+A Green, bolded topic+}**
  • ### tOpIc: Another topic

Quick Actions

Action Description
/discuss sub-topic TITLE Create an issue for a sub-topic. Does not work in epics
/discuss link ISSUABLE-LINK Link an issuable as a child of this discussion

Last updated by this job

Discoto Settings 
---
summary:
  max_items: -1
  sort_by: created
  sort_direction: ascending

See the settings schema for details.

Edited by Fabien Catteau