Create an EPSS ingestion service
Introduction
The flow of package_metadata on the GitLab side is:
- Cronjob executes the relevant data type worker (licenses, advisories, epss).
- The worker runs the
SyncServicewhich handles thepackage_metadataflow for each purl type or for epss SyncServiceretrieves aSyncConfigurationfor the relevant epss.SyncServiceuses the relevant connector (offline or GCP) to iterate over all new files (chunks) in the bucket since the last checkpoint.SyncServiceexecutesIngestionServicefor the given data type.- The
IngestionServiceruns a set ofIngestionTask. - Each
IngestionTaskparses and upserts the given data. - The checkpoint is updated to reflect that we have progressed and data has been ingested.
- Continue until all data has been inserted or a stop signal is received.
This issue is responsible for implementing the ingestion part of the whole flow.
Implementation Plan
- Create
epss.rbmodel inee/app/models/package_metadatasimilarly to other objects in the directory. Note the requirement forinclude BulkInsertSafe(explained in this video at 10:21). (Implemented in Add EPSS to GitLab database and enable querying... (#470835 - closed)). - Create a
cve_enrichmentdirectory underee/app/services/package_metadata/ingestion. - Implement
ee/app/services/package_metadata/ingestion/cve_enrichment/cve_enrichment_ingestion_task.rbsimilarly toee/app/services/package_metadata/ingestion/advisory/advisory_ingestion_task.rb. to upsert EPSS data. This is also where parsing happens. - Implement
ee/app/services/package_metadata/ingestion/cve_enrichment/ingestion_service.rbsimilarly toee/app/services/package_metadata/ingestion/advisory/ingestion_service.rb. -
Use a feature flag to enable/disable ingestion.(This is done in Resolve "Introduce a sync mechanism for EPSS sc... (!164348 - merged))
Test! You may create a CVE Enrichment object in ee/spec/factories/package_metadata similarly to ee/spec/factories/package_metadata/advisory_data_objects.rb.
- Implement
ee/spec/services/package_metadata/ingestion/cve_enrichment/cve_enrichment_ingestion_task_spec.rbsimilarly to nearby tests. - Implement
ee/spec/services/package_metadata/ingestion/cve_enrichment/ingestion_service_spec.rb
Edited by Yasha Rise