Increase the frequency of the package metadata export

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Work on this issue
• Close this issue

Why are we doing this work

With the advent of CVS, vulnerabilities are being identified in projects as soon as the advisory DB is processed by GitLab.

While the schedule for package_metadata_advisories_sync_worker and package_metadata_licenses_sync_worker is "every 5 minutes", the License-DB (aka package metadata) exports only happens every 24 hours.

In order to take full advantage of CVS, the exporter should run as frequently as possible to reduce the time between an advisory being published and a vulnerability being created.

Relevant links

Non-functional requirements

• Documentation:
• Feature flag:
• Performance: The performance of global advisory scans should improve due to the reduced resource contention, and lock contention in Postgres.
• Testing:

Implementation plan

We don't want to rate limit ourselves, so we will take a staggered approach to the interval reduction.

1. Update the scheduled feeder and exporter job schedule in the following intervals. After every modification, verify that we are not running into increased errors or putting excessive pressure on the backend with vulnerability scans.
   1. 1hr
   2. 30m
   3. 15m

Verification steps

1. After every export interval adjustment let an hour pass and verify that the logs don't show an increase in rate limiting 429 status codes.
2. Verify the CVS dashboard to ensure that we don't see an increase worker or SQL errors, and that we don't have an increase in job duration.