Skip to content
GitLab
Next
Closed
Issue created by Julian Thome@julianthome⌨Developer

gemnasium does not always use the latest state/version of gemnasium-db

Release notes

Dependency Scanning is designed to check for the latest version of the vulnerability database (called the gemnasium--db) each time the analyzer is run, however starting [DATE] a bug was introduced and the vulnerability database was only updated when the analyzer was updated (which happens monthly). As a result, since the introduction of the bug, scan results could be identifying only CVE found earlier than one month ago. We are concerned by this bug and will be performing a review of it in order to prevent it, or things similar, from going undetected again.

If you regularly update to the current docker image for Dependency Scanning the bug will be resolved as soon as you are using the latest image. If you are more deliberate in your updates you should X.

Issue

We became aware of the following issue by accident through an advisory request created by @joernchen in this slack discussion.

Summary

The gemnasium dependency scanner leverages the advisory data provided by gemnasium-db which is updated on a daily basis.

There is some logic in the gemnasium dependency scanner for fetching the latest updates from gemnasium-db. As it turned out, gemnasium always fetches the latest updates from gemnasium-db, but the HEAD is not updated to point to the state of the most recent origin/master. Therefore, although updates are fetched from gemnasium-db, gemnasium uses the state of gemnasium from the time when the docker image was created.

This affects all Gemnasium-based analyzers: gemnasium, gemnasium-maven, and gemnasium-python.

This regression was introduced when implementing #196868 (closed)

Further details

Gemnasium updates its local git repo of gemnasium-db by running git commands. The Shell equivalent of these comments would be:

git remote set-url origin $GEMNASIUM_DB_REMOTE_URL
git remote update
git checkout $GEMNASIUM_DB_REF_NAME

See repo.go

This code is shared by gemnasium, gemnasium-maven, and gemnasium-python.

Steps to reproduce

You can reproduce the issue by using the sample go project testproject that contains a vulnerability for which an advisory (CVE-2020-29652.yml) was added recently to gemnasium-db (gemnasium-db version: v1.2.383).

Gemnasium v2.24.0 was released before v1.2.383 based on the older state of gemnasium-db v1.2.377.

# clone the test project
> git clone https://gitlab.com/julianthome/testadbupdate
> cd testadbupdate
# run gemnasium 2.24.0
> docker run -it --volume "$PWD:/tmp/project" \
    --env SECURE_LOG_LEVEL=debug \
    --env CI_PROJECT_DIR=/tmp/project \
    registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium:2.24.0
# CVE-2020-29652 not found
> grep -m 1 -o CVE-2020-29652 gl-dependency-scanning-report.json || echo "nothing found"
nothing found
# run the same version of gemnasium but this time by explicitly defining the latest version v1.2.383
> docker run -it --volume "$PWD:/tmp/project" \
    --env GEMNASIUM_DB_REF_NAME=v1.2.383 \
    --env SECURE_LOG_LEVEL=debug \
    --env CI_PROJECT_DIR=/tmp/project \
    registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium:2.24.0
> grep -m 1 -o CVE-2020-29652 gl-dependency-scanning-report.json || echo "nothing found"
CVE-2020-29652

Example Project

The test project that uses the DS template is available under https://gitlab.com/julianthome/testadbupdate. You can see that CVE-2020-29652 does not show up in the vulnerability report. However there are two branches/MR where CVE-2020-29652 shows up in the vulnerability report:

  1. https://gitlab.com/julianthome/testadbupdate/-/merge_requests/1: by specifying GEMNASIUM_DB_REF_NAME: "v1.2.383"
  2. https://gitlab.com/julianthome/testadbupdate/-/merge_requests/2: by specifying GEMNASIUM_DB_REF_NAME: "origin/master"

What is the current bug behavior?

Recently added advisories to gemnasium-db do not show up in the vulnerability report because gemnasium uses the state of gemnasium-db from the time where the docker image was created.

What is the expected correct behavior?

Recently added advisories to gemnasium-db should show up instantaneously in the vulnerability report.

Relevant logs and/or screenshots

You can find more details in the logs for the two jobs that were running on master and the MRs:

[DEBU] [Gemnasium] [2020-12-17T17:27:54Z] ▶ /usr/bin/git -C /gemnasium-db remote set-url origin https://gitlab.com/gitlab-org/security-products/gemnasium-db.git
[DEBU] [Gemnasium] [2020-12-17T17:27:55Z] ▶ /usr/bin/git -C /gemnasium-db remote update
Fetching origin
From https://gitlab.com/gitlab-org/security-products/gemnasium-db
   493a7e116..f16104eea  master              -> origin/master
 * [new branch]          remove-spellchecker -> origin/remove-spellchecker
 * [new tag]             v1.2.383            -> v1.2.383
 * [new tag]             v1.2.380            -> v1.2.380
 * [new tag]             v1.2.381            -> v1.2.381
 * [new tag]             v1.2.382            -> v1.2.382
[DEBU] [Gemnasium] [2020-12-17T17:27:55Z] ▶ /usr/bin/git -C /gemnasium-db checkout master
Already on 'master'
Your branch is behind 'origin/master' by 46 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
[DEBU] [Gemnasium] [2020-12-17T17:27:55Z] ▶ /usr/bin/git -C /builds/julianthome/testadbupdate status
HEAD detached at 177fdbe
nothing to commit, working tree clean
[DEBU] [Gemnasium] [2020-12-17T17:37:18Z] ▶ /usr/bin/git -C /gemnasium-db remote set-url origin https://gitlab.com/gitlab-org/security-products/gemnasium-db.git
[DEBU] [Gemnasium] [2020-12-17T17:37:18Z] ▶ /usr/bin/git -C /gemnasium-db remote update
Fetching origin
From https://gitlab.com/gitlab-org/security-products/gemnasium-db
   493a7e116..f16104eea  master              -> origin/master
 * [new branch]          remove-spellchecker -> origin/remove-spellchecker
 * [new tag]             v1.2.383            -> v1.2.383
 * [new tag]             v1.2.380            -> v1.2.380
 * [new tag]             v1.2.381            -> v1.2.381
 * [new tag]             v1.2.382            -> v1.2.382
[DEBU] [Gemnasium] [2020-12-17T17:37:18Z] ▶ /usr/bin/git -C /gemnasium-db checkout origin/master
Note: checking out 'origin/master'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
  git checkout -b <new-branch-name>
HEAD is now at f16104eea Merge branch 'add-CVE-2020-29652' into 'master'

Possible fixes

See discussion

Implementation plan

Credits

Thanks a lot to @joernchen and @plafoucriere for helping me to understand the nature of this issue.

Edited by Igor Frenkel