Update security report processing to handle metrics

Update the security report processing done in the monolith to log metrics using our internal events API. Perform validation of the event prior to logging. Validation should include checking an allow list of event names. It might also be good to have some thresholds for how many events are expected in a single report.

Tracing current report handling

The following is a trace of pipeline completing to report storage. It's not known if there is post-db-insertion processing that occurs.

Click to expand

EE::Ci::Pipeline

	state_machine ->
		after_transition => ::Ci::Pipeline.completed_with_manual_statuses ->
			Security::StoreScansWorker.perform_async(pipeline.id)
		
			or
		
			Sbom::ScheduleIngestReportsService.new(pipeline).execute

Security::StoreScansWorker (ApplicationWorker)
	perform(pipeline_id) ->
		Security::StoreScansService.execute(pipeline)

Security::StoreScansService
store_scans_service.rb
	This service takes in a pipeline and stores all the security reports.
	
	execute(pipeline) ->
		StoreGroupedScansService.execute(artifacts)


Security::StoreGroupedScansService
	This service stores one or more security report artifacts using Security::StoreScanService
	
	execute ->
		store_scan_for -> 
			StoreScanService.execute
	

Security::StoreScanService
store_scan_service.rb
	This service stores the `Security::Scan` and `Security::Finding` records for the given job artifact.

Implementation plan

• Metrics data doesn't need to be stored along with the security report, only provided to the events API.
  • Makes most sense to process the events directly from the pipeline
• Processing errors will be reported via an event defined for this purpose.
• Validation is needed to prevent attackers from changing events not related to analyzers and easily polluting the data.
  • Allow list for event names
  • Threshold counts (how many times an event can be in a report)
  • Action: Don't ingest any events and report via event

Plan:

1. Create a feature flag to put this work behind
2. Create a new worker to perform the event processing modeled after Security::StoreScansWorker
3. Implement worker such that:
   1. Process each event:
      1. Verify event name is in allow list. If not continue to next event.
      2. Verify the number of events provided for specific names doesn't cross defined threshold
      3. Record the event using the internal events API
4. Call worker when ::Security::StoreScansWorker is called from EE::Ci::Pipeline
5. Write tests