Increase the frequency of the package metadata export
Why are we doing this work
With the advent of CVS, vulnerabilities are being identified in projects as soon as the advisory DB is processed by GitLab.
While the schedule for package_metadata_advisories_sync_worker
and package_metadata_licenses_sync_worker
is "every 5 minutes", the License-DB (aka package metadata) exports only happens every 24 hours.
In order to take full advantage of CVS, the exporter should run as frequently as possible to reduce the time between an advisory being published and a vulnerability being created.
Relevant links
Non-functional requirements
-
Documentation: -
Feature flag: -
Performance: The performance of global advisory scans should improve due to the reduced resource contention, and lock contention in Postgres. -
Testing:
Implementation plan
We don't want to rate limit ourselves, so we will take a staggered approach to the interval reduction.
- Update the scheduled feeder and exporter job schedule in the following intervals. After every modification, verify that we are not running into increased errors or putting excessive pressure on the backend with vulnerability scans.
1hr
30m
15m
Verification steps
- After every export interval adjustment let an hour pass and verify that the logs don't show an increase in rate limiting
429
status codes. - Verify the CVS dashboard to ensure that we don't see an increase worker or SQL errors, and that we don't have an increase in job duration.
Edited by Oscar Tovar