[PMDB] Exporter should throttle advisories export above a certain threshold
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Problem description
The root cause was identified as an unusually large batch of OS-related security advisories being exported from Trivy DB, which triggered the PackageMetadata::GlobalAdvisoryScanWorker jobs. This was determined to be a normal but rare occurrence when external sources publish many updates simultaneously, not a system issue.
The result was to have hundreds of queued jobs in the Rails app. This is quite an edge case that normally doesn't happen
Proposal
In order to make PMDB more robust we should alter the exporter logic so that it doesn't export in one go all the advisories if they are above a certain threahsold. If the threshold is 100 advisories and we have 1000 advisories we could export 10 batches with delays of 5-10minutes. This is not ideal though since we will create more exported timestamp dirs in the public buckets. This could be mitigated once we have automatic compaction implemented. Another disadvantage is that the exporter job could timeout.
Related issues
https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/team/-/issues/8795+