Create an EPSS ingestion service
Introduction
The flow of package_metadata on the GitLab side is:
- Cronjob executes the relevant data type worker (licenses, advisories, epss).
- The worker runs the
SyncServicewhich handles thepackage_metadataflow for each purl type or for epss -
SyncServiceretrieves aSyncConfigurationfor the relevant epss. -
SyncServiceuses the relevant connector (offline or GCP) to iterate over all new files (chunks) in the bucket since the last checkpoint. -
SyncServiceexecutesIngestionServicefor the given data type. - The
IngestionServiceruns a set ofIngestionTask. - Each
IngestionTaskparses and upserts the given data. - The checkpoint is updated to reflect that we have progressed and data has been ingested.
- Continue until all data has been inserted or a stop signal is received.
This issue is responsible for implementing the ingestion part of the whole flow.
Implementation Plan
-
Create epss.rbmodel inee/app/models/package_metadatasimilarly to other objects in the directory. Note the requirement forinclude BulkInsertSafe(explained in this video at 10:21). (Implemented in Add EPSS to GitLab database and enable querying... (#470835 - closed)). -
Create a cve_enrichmentdirectory underee/app/services/package_metadata/ingestion. -
Implement ee/app/services/package_metadata/ingestion/cve_enrichment/cve_enrichment_ingestion_task.rbsimilarly toee/app/services/package_metadata/ingestion/advisory/advisory_ingestion_task.rb. to upsert EPSS data. This is also where parsing happens. -
Implement ee/app/services/package_metadata/ingestion/cve_enrichment/ingestion_service.rbsimilarly toee/app/services/package_metadata/ingestion/advisory/ingestion_service.rb. -
Use a feature flag to enable/disable ingestion.(This is done in Resolve "Introduce a sync mechanism for EPSS sc... (!164348 - merged))
Test! You may create a CVE Enrichment object in ee/spec/factories/package_metadata similarly to ee/spec/factories/package_metadata/advisory_data_objects.rb.
-
Implement ee/spec/services/package_metadata/ingestion/cve_enrichment/cve_enrichment_ingestion_task_spec.rbsimilarly to nearby tests. -
Implement ee/spec/services/package_metadata/ingestion/cve_enrichment/ingestion_service_spec.rb
Edited by Yasha Rise