AST: Package Metadata DB: improve offline synchronization experience
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Release notes
Problem to solve
Enabling the Package Metadata Database for air gapped systemms is complex, due to current methods expecting access to shared storage. This does not operate well in cloud native GitLab installations.
Proposal
Implement and document a method for isolating the tasks into Rake and/or isolated Sidekiq queues, so that these can be directly loaded from localized object storage or a filesystem attached to a specific container in a cloud native GitLab installation. While this may remain a manual process, as "sneaker net" shuttling of the Package Metadata DB is mandatory in these environments, we can significantly improve the experience of the GitLab instance Operators, and several other SAST / continuous scanning consumer personas.
detailed workaround
- Get Package Metadata files off of GCP
- Add a sufficiently sized PV/PVC to the toolbox pod mounted at
/srv/gitlab/vendor/package_metadata - Copy the licenses and advisories folders into the PVC (
kubectl cpor similar) - Run the following on the toolbox pod (this is out of Duo):
gitlab-rails runner "lease = Gitlab::ExclusiveLease.new('package_metadata_licenses_sync', timeout: 360); lease.try_obtain && PackageMetadata::SyncService.execute(data_type: 'licenses', lease: lease) && lease.cancel"gitlab-rails runner "lease = Gitlab::ExclusiveLease.new('package_metadata_advisories_sync', timeout: 360); lease.try_obtain && PackageMetadata::SyncService.execute(data_type: 'advisories', lease: lease) && lease.cancel"
Note: Based on this code & this code, it appears PackageMetadata::SyncService.execute will use the local files before going to GCP.
Intended users
- Sidney (Systems Administrator)
- Ingrid (Infrastructure Operator)
- Isaac (Infrastructure Security Engineer)
- Alex (Security Operations Engineer)
- Amy (Application Security Engineer)
Feature Usage Metrics
N/A