Skip to content
GitLab
Next
    • GitLab: the DevOps platform
    • Explore GitLab
    • Install GitLab
    • How GitLab compares
    • Get started
    • GitLab docs
    • GitLab Learn
  • Pricing
  • Talk to an expert
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
    Projects Groups Topics Snippets
  • Register
  • Sign in
  • GitLab GitLab
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
    • Locked files
  • Issues 52,121
    • Issues 52,121
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 1,553
    • Merge requests 1,553
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Artifacts
    • Schedules
    • Test cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Metrics
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.orgGitLab.org
  • GitLabGitLab
  • Issues
  • #321315
Closed
Open
Issue created Feb 11, 2021 by Igor Frenkel@ifrenkelDeveloper

RCA of Dependency Scanning vulnerability database issue

Please note: if the incident relates to sensitive data, or is security related consider labeling this issue with security and mark it confidential.


Summary

Dependency Scanning had a bug which caused its internal vulnerability database to lag behind the live vulnerability database gemnasium-db. This resulted in users having dependency scans with outdated results.

  • Service(s) affected : Dependency Scanning Category:Dependency Scanning
  • Team attribution : Composition Analysis groupcomposition analysis
  • Minutes downtime or degradation : This issue has been present in the code since 2020-03-30

Impact & Metrics

Question Answer
What was the impact Projects running dependency scans had intermittently outdated data.
Who was impacted Users with projects running Dependency Scanning.
How did this impact customers Customers could have received false negatives.

The vulnerability database got updated intermittently (rather than on every scan) every time a new version of the gemnasium analyzer was released.

Note: The gemnasium-python and gemnasium-maven analyzers use gemnasium's advisory package for updating their vulnerability database and were affected by the bug as well. For a list of affected languages and package managers see the gemnasium, gemnasium-maven, and gemnasium-python entries in the docs.

Detection & Response

Start with the following:

Question Answer
When was the incident detected? 2020-12-17
How was the incident detected? Vulnerability Research team noticed detection of newly added advisories not working.
Did alarming work as expected? A bug was opened with the groupcomposition analysis team.
How long did it take from the start of the incident to its detection? 11 months
How long did it take from detection to remediation? 1 release
What steps were taken to remediate? Reproduce issue, fix, issue new releases for affected analyzers.

Timeline

gemnasium analyzer affected versions

Date new version released gemnasium version # of days vuln db stale
2021-02-03T07:28:43.847Z v2.28.1 bug fixed
2021-01-22T08:24:53.957Z v2.28.0 7 days
2021-01-15T07:08:16.494Z v2.27.0 1 days
2021-01-14T07:06:17.311Z v2.26.2 0 days
2021-01-13T15:19:17.147Z v2.26.1 1 days
2021-01-12T15:07:59.584Z v2.26.0 5 days
2021-01-06T19:15:24.291Z v2.25.1 0 days
2021-01-06T08:05:07.486Z v2.25.0 19 days
2020-12-17T16:13:24.819Z bug reported
2020-12-17T16:13:44.819Z v2.24.1 1 days
2020-12-16T08:44:42.274Z v2.24.0 5 days
2020-12-10T17:40:14.599Z v2.23.0 6 days
2020-12-04T06:46:08.514Z v2.22.0 20 days
2020-11-13T15:00:29.945Z v2.21.0 20 days
2020-10-23T15:05:14.239Z v2.20.0 7 days
2020-10-16T13:20:38.888Z v2.19.0 17 days
2020-09-29T12:36:31.184Z v2.18.1 6 days
2020-09-22T15:44:28.344Z v2.18.0 8 days
2020-09-14T12:29:40.605Z v2.17.1 10 days
2020-09-04T01:40:20.718Z v2.17.0 3 days
2020-08-31T06:21:17.408Z v2.16.0 17 days
2020-08-14T02:51:45.277Z v2.15.0 0 days
2020-08-13T03:31:34.714Z v2.14.0 9 days
2020-08-03T08:47:19.417Z v2.13.0 3 days
2020-07-31T08:26:52.483Z v2.12.1 27 days
2020-07-03T10:27:28.568Z v2.12.0 0 days
2020-07-03T06:07:47.475Z v2.11.0 21 days
2020-06-11T09:21:38.226Z v2.10.1 63 days
2020-04-08T13:15:49.478Z v2.9.0 9 days
2020-03-30T17:18:16.299Z v2.8.1 bug introduced

gemnasium-python analyzer affected versions

Date new version released gemnasium-python version gemnasium version # of days vuln db stale
2021-02-10T14:36:45.793Z v2.17.3 v2.28.3 bug fixed
2021-01-13T15:47:51.007Z v2.17.2 v2.26.1 6 days
2021-01-06T19:31:48.797Z v2.17.1 v2.25.1 21 days
2020-12-16T14:22:01.735Z v2.17.0 v2.24.0 9 days
2020-12-07T08:44:28.327Z v2.16.0 v2.22.0 90 days
2020-09-07T12:26:23.253Z v2.15.0 v2.11.0 24 days
2020-08-14T06:07:55.503Z v2.14.1 v2.11.0 1 days
2020-08-13T03:36:11.208Z v2.14.0 v2.11.0 40 days
2020-07-03T07:35:16.222Z v2.13.0 v2.11.0 21 days
2020-06-11T11:56:12.576Z v2.12.1 v2.10.1 20 days
2020-05-22T10:12:14.872Z v2.12.0 v2.10.0 6 days
2020-05-16T06:06:11.340Z v2.11.0 v2.10.0 bug introduced

gemnasium-maven analyzer affected versions

Date new version released gemnasium-maven version gemnasium version # of days vuln db stale
2021-02-10T14:50:00.651Z v2.20.4 v2.28.2 bug fixed
2021-01-13T16:11:25.403Z v2.20.3 v2.26.1 6 days
2021-01-07T15:55:15.418Z v2.20.2 v2.25.1 0 days
2021-01-07T07:51:44.841Z v2.20.1 v2.25.1 21 days
2020-12-16T14:21:13.335Z v2.20.0 v2.24.0 0 days
2020-12-15T22:53:13.404Z v2.19.1 v2.22.0 7 days
2020-12-08T07:00:28.830Z v2.19.0 v2.22.0 53 days
2020-10-15T11:55:05.767Z v2.18.4 v2.11.0 26 days
2020-09-18T14:21:07.317Z v2.18.3 v2.11.0 0 days
2020-09-18T07:35:10.523Z v2.18.2 v2.11.0 1 days
2020-09-16T18:41:39.161Z v2.18.1 v2.11.0 9 days
2020-09-07T12:20:17.759Z v2.18.0 v2.11.0 20 days
2020-08-17T13:01:48.808Z v2.17.2 v2.11.0 3 days
2020-08-14T07:50:58.331Z v2.17.1 v2.11.0 1 days
2020-08-13T03:29:11.396Z v2.17.0 v2.11.0 33 days
2020-07-10T07:08:07.620Z v2.16.1 v2.11.0 6 days
2020-07-03T08:13:14.829Z v2.16.0 v2.11.0 1 days
2020-07-02T07:21:23.896Z v2.15.0 v2.10.2-0.20200702060642-7e4851913865 20 days
2020-06-11T11:57:37.228Z v2.14.2 v2.10.1 13 days
2020-05-29T09:24:02.313Z v2.14.1 v2.10.0 13 days
2020-05-16T06:16:33.118Z v2.14.0 v2.10.0 29 days
2020-04-17T01:58:44.357Z v2.13.0 v2.10.0 bug introduced

Bug fix rollout

A fix for this bug was released in gemnasium v2.28.1 and subsequently in gemnasium-maven v2.20.4 and gemnasium-python v2.17.3. The default Dependency Scanning template always points to the latest major tag of the analyzer (i.e. gemnasium:2, gemnasium-maven:2, gemnasium-python:2). This means that when a new version of the analyzer is released, the major tag is moved to point at that new version. Users that are using gitlab.com runners, container registry, and original Dependency Scanning template will have the new analyzer version available to them on the very next scan after the bug fix release.

Users in the following categories would need to take a manual action of pulling the relevant analyzer docker version into their container registry:

  • gitlab.com users with their own docker runners and a pull-policy other than always
  • self managed users with gitlab runners using pull-policy with a setting other than always
  • users with an edited Dependency Scanning template with analyzers pinned to a non-major-only tag (say gemnasium:2.27.0) pointing to a version between v2.8.1 and `v2.28.0

Note: If a customer is on an affected version but is moving their own gemnasium-db tags to update the advisory database, then they are unaffected by this bug.

Root Cause Analysis

A bug was introduced in the Dependency Scanning detection analyzer gemnasium. Specifically, the issue was in the advisory package which uses gemnasium-db git repo as the data source and refreshes this data via git commands. advisory.Update syncs with its advisory db remote at scan-time in order to pull in the freshest version of the database.

The feature delivered with version v2.8.1 added functionality for using commit refs and tags alongside branches. In this version bug was introduced in the way git handles updates: if the build-time branch matched the scan-time branch (i.e. the branch was already checked out), no update would be conducted because git would assume it has the latest copy.

In order to have a full set of advisories and not pull data needlessly on every scan, gemnasium also builds with its own version of the advisory database. It then also tries to refresh this database at scan-time using the GEMNASIUM_DB_REF_NAME environment variable - this was the part affected. Thus, the "freshness" of advisories was dictated by the analyzer's last build time (see the timeline section).

What went well

  • Good team collaboration to get issue tested and thoroughly reviewed.
  • QA test environment caught a possible regression in the release process (i.e. we added tests more some facets of this vuln db update behavior).

What can be improved

  • We need to add a class of integration tests for the update logic of the vulnerability database.
  • The projects we use for integration tests never change unless we add new scanning capabilities and need the test project to reflect this. This makes simulation of "real" user scenarios more difficult. We should discuss how to add a "realistic" subset to test projects.

Corrective actions

  • Add a set of test scenarios specifically for the vulnerability database feature (regression test with better coverage was added alongside the fix).

Guidelines

  • Blameless RCA Guideline
  • 5 whys
Edited Feb 17, 2021 by Olivier Gonzalez
Assignee
Assign to
Time tracking