RCA of Dependency Scanning vulnerability database issue
Please note: if the incident relates to sensitive data, or is security related consider labeling this issue with security and mark it confidential.
Summary
Dependency Scanning had a bug which caused its internal vulnerability database to lag behind the live vulnerability database gemnasium-db. This resulted in users having dependency scans with outdated results.
- Service(s) affected : Dependency Scanning Category:Dependency Scanning
- Team attribution : Composition Analysis groupcomposition analysis
- Minutes downtime or degradation : This issue has been present in the code since 2020-03-30
Impact & Metrics
| Question | Answer |
|---|---|
| What was the impact | Projects running dependency scans had intermittently outdated data. |
| Who was impacted | Users with projects running Dependency Scanning. |
| How did this impact customers | Customers could have received false negatives. |
The vulnerability database got updated intermittently (rather than on every scan) every time a new version of the gemnasium analyzer was released.
Note: The gemnasium-python and gemnasium-maven analyzers use gemnasium's advisory package for updating their vulnerability database and were affected by the bug as well. For a list of affected languages and package managers see the gemnasium, gemnasium-maven, and gemnasium-python entries in the docs.
Detection & Response
Start with the following:
| Question | Answer |
|---|---|
| When was the incident detected? | 2020-12-17 |
| How was the incident detected? | Vulnerability Research team noticed detection of newly added advisories not working. |
| Did alarming work as expected? | A bug was opened with the groupcomposition analysis team. |
| How long did it take from the start of the incident to its detection? | 11 months |
| How long did it take from detection to remediation? | 1 release |
| What steps were taken to remediate? | Reproduce issue, fix, issue new releases for affected analyzers. |
Timeline
gemnasium analyzer affected versions
| Date new version released | gemnasium version | # of days vuln db stale |
|---|---|---|
| 2021-02-03T07:28:43.847Z | v2.28.1 | bug fixed |
| 2021-01-22T08:24:53.957Z | v2.28.0 | 7 days |
| 2021-01-15T07:08:16.494Z | v2.27.0 | 1 days |
| 2021-01-14T07:06:17.311Z | v2.26.2 | 0 days |
| 2021-01-13T15:19:17.147Z | v2.26.1 | 1 days |
| 2021-01-12T15:07:59.584Z | v2.26.0 | 5 days |
| 2021-01-06T19:15:24.291Z | v2.25.1 | 0 days |
| 2021-01-06T08:05:07.486Z | v2.25.0 | 19 days |
| 2020-12-17T16:13:24.819Z | bug reported | |
| 2020-12-17T16:13:44.819Z | v2.24.1 | 1 days |
| 2020-12-16T08:44:42.274Z | v2.24.0 | 5 days |
| 2020-12-10T17:40:14.599Z | v2.23.0 | 6 days |
| 2020-12-04T06:46:08.514Z | v2.22.0 | 20 days |
| 2020-11-13T15:00:29.945Z | v2.21.0 | 20 days |
| 2020-10-23T15:05:14.239Z | v2.20.0 | 7 days |
| 2020-10-16T13:20:38.888Z | v2.19.0 | 17 days |
| 2020-09-29T12:36:31.184Z | v2.18.1 | 6 days |
| 2020-09-22T15:44:28.344Z | v2.18.0 | 8 days |
| 2020-09-14T12:29:40.605Z | v2.17.1 | 10 days |
| 2020-09-04T01:40:20.718Z | v2.17.0 | 3 days |
| 2020-08-31T06:21:17.408Z | v2.16.0 | 17 days |
| 2020-08-14T02:51:45.277Z | v2.15.0 | 0 days |
| 2020-08-13T03:31:34.714Z | v2.14.0 | 9 days |
| 2020-08-03T08:47:19.417Z | v2.13.0 | 3 days |
| 2020-07-31T08:26:52.483Z | v2.12.1 | 27 days |
| 2020-07-03T10:27:28.568Z | v2.12.0 | 0 days |
| 2020-07-03T06:07:47.475Z | v2.11.0 | 21 days |
| 2020-06-11T09:21:38.226Z | v2.10.1 | 63 days |
| 2020-04-08T13:15:49.478Z | v2.9.0 | 9 days |
| 2020-03-30T17:18:16.299Z | v2.8.1 | bug introduced |
gemnasium-python analyzer affected versions
| Date new version released | gemnasium-python version | gemnasium version | # of days vuln db stale |
|---|---|---|---|
| 2021-02-10T14:36:45.793Z | v2.17.3 | v2.28.3 | bug fixed |
| 2021-01-13T15:47:51.007Z | v2.17.2 | v2.26.1 | 6 days |
| 2021-01-06T19:31:48.797Z | v2.17.1 | v2.25.1 | 21 days |
| 2020-12-16T14:22:01.735Z | v2.17.0 | v2.24.0 | 9 days |
| 2020-12-07T08:44:28.327Z | v2.16.0 | v2.22.0 | 90 days |
| 2020-09-07T12:26:23.253Z | v2.15.0 | v2.11.0 | 24 days |
| 2020-08-14T06:07:55.503Z | v2.14.1 | v2.11.0 | 1 days |
| 2020-08-13T03:36:11.208Z | v2.14.0 | v2.11.0 | 40 days |
| 2020-07-03T07:35:16.222Z | v2.13.0 | v2.11.0 | 21 days |
| 2020-06-11T11:56:12.576Z | v2.12.1 | v2.10.1 | 20 days |
| 2020-05-22T10:12:14.872Z | v2.12.0 | v2.10.0 | 6 days |
| 2020-05-16T06:06:11.340Z | v2.11.0 | v2.10.0 | bug introduced |
gemnasium-maven analyzer affected versions
| Date new version released | gemnasium-maven version | gemnasium version | # of days vuln db stale |
|---|---|---|---|
| 2021-02-10T14:50:00.651Z | v2.20.4 | v2.28.2 | bug fixed |
| 2021-01-13T16:11:25.403Z | v2.20.3 | v2.26.1 | 6 days |
| 2021-01-07T15:55:15.418Z | v2.20.2 | v2.25.1 | 0 days |
| 2021-01-07T07:51:44.841Z | v2.20.1 | v2.25.1 | 21 days |
| 2020-12-16T14:21:13.335Z | v2.20.0 | v2.24.0 | 0 days |
| 2020-12-15T22:53:13.404Z | v2.19.1 | v2.22.0 | 7 days |
| 2020-12-08T07:00:28.830Z | v2.19.0 | v2.22.0 | 53 days |
| 2020-10-15T11:55:05.767Z | v2.18.4 | v2.11.0 | 26 days |
| 2020-09-18T14:21:07.317Z | v2.18.3 | v2.11.0 | 0 days |
| 2020-09-18T07:35:10.523Z | v2.18.2 | v2.11.0 | 1 days |
| 2020-09-16T18:41:39.161Z | v2.18.1 | v2.11.0 | 9 days |
| 2020-09-07T12:20:17.759Z | v2.18.0 | v2.11.0 | 20 days |
| 2020-08-17T13:01:48.808Z | v2.17.2 | v2.11.0 | 3 days |
| 2020-08-14T07:50:58.331Z | v2.17.1 | v2.11.0 | 1 days |
| 2020-08-13T03:29:11.396Z | v2.17.0 | v2.11.0 | 33 days |
| 2020-07-10T07:08:07.620Z | v2.16.1 | v2.11.0 | 6 days |
| 2020-07-03T08:13:14.829Z | v2.16.0 | v2.11.0 | 1 days |
| 2020-07-02T07:21:23.896Z | v2.15.0 | v2.10.2-0.20200702060642-7e4851913865 | 20 days |
| 2020-06-11T11:57:37.228Z | v2.14.2 | v2.10.1 | 13 days |
| 2020-05-29T09:24:02.313Z | v2.14.1 | v2.10.0 | 13 days |
| 2020-05-16T06:16:33.118Z | v2.14.0 | v2.10.0 | 29 days |
| 2020-04-17T01:58:44.357Z | v2.13.0 | v2.10.0 | bug introduced |
Bug fix rollout
A fix for this bug was released in gemnasium v2.28.1 and subsequently in gemnasium-maven v2.20.4 and gemnasium-python v2.17.3. The default Dependency Scanning template always points to the latest major tag of the analyzer (i.e. gemnasium:2, gemnasium-maven:2, gemnasium-python:2). This means that when a new version of the analyzer is released, the major tag is moved to point at that new version. Users that are using gitlab.com runners, container registry, and original Dependency Scanning template will have the new analyzer version available to them on the very next scan after the bug fix release.
Users in the following categories would need to take a manual action of pulling the relevant analyzer docker version into their container registry:
-
gitlab.comusers with their own docker runners and a pull-policy other thanalways - self managed users with gitlab runners using pull-policy with a setting other than
always - users with an edited Dependency Scanning template with analyzers pinned to a non-major-only tag (say
gemnasium:2.27.0) pointing to a version betweenv2.8.1and `v2.28.0
Note: If a customer is on an affected version but is moving their own gemnasium-db tags to update the advisory database, then they are unaffected by this bug.
Root Cause Analysis
A bug was introduced in the Dependency Scanning detection analyzer gemnasium. Specifically, the issue was in the advisory package which uses gemnasium-db git repo as the data source and refreshes this data via git commands. advisory.Update syncs with its advisory db remote at scan-time in order to pull in the freshest version of the database.
The feature delivered with version v2.8.1 added functionality for using commit refs and tags alongside branches. In this version bug was introduced in the way git handles updates: if the build-time branch matched the scan-time branch (i.e. the branch was already checked out), no update would be conducted because git would assume it has the latest copy.
In order to have a full set of advisories and not pull data needlessly on every scan, gemnasium also builds with its own version of the advisory database. It then also tries to refresh this database at scan-time using the GEMNASIUM_DB_REF_NAME environment variable - this was the part affected. Thus, the "freshness" of advisories was dictated by the analyzer's last build time (see the timeline section).
What went well
- Good team collaboration to get issue tested and thoroughly reviewed.
- QA test environment caught a possible regression in the release process (i.e. we added tests more some facets of this vuln db update behavior).
What can be improved
- We need to add a class of integration tests for the update logic of the vulnerability database.
- The projects we use for integration tests never change unless we add new scanning capabilities and need the test project to reflect this. This makes simulation of "real" user scenarios more difficult. We should discuss how to add a "realistic" subset to test projects.
Corrective actions
- Add a set of test scenarios specifically for the vulnerability database feature (regression test with better coverage was added alongside the fix).