Skip to content

Implement very large file export mechanism

What does this MR do and why?

This MR implements a relatively generic mechanism for exporting large amounts of vulnerabilities at the group level by breaking the export into multiple export parts, queuing multiple export workers which process the parts of the export before a finalisation job knits these resultant parts into the final export file for the user.

The existing mechanisms have been struggling with the size of vulnerabilities some groups may be attempting to export. Additionally, due to the sheer size of these exports, loading them into memory had become a hazard for our Sidekiq workers. As such this MR seeks to mitigate these problems by using the GitLab filestore mechanism to stream the data to files during reading and writing to avoid memory constraints.

Finally, as a safety precaution, this mechanic is both feature flagged, and has a limited number of workers for data reading to strike a balancing point between stability and speed of the export. More workers can export the report more quickly, but this comes at the cost of putting additional strain on the database to read all of these vulnerabilities out in rapid succession.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

This is a little tricky to validate, because it's entirely backend and shouldn't look any different, and is mitigating a circumstance you're unlikely to reproduce locally (several 10's of thousands of vulnerabilities)

As such, at most what you can do for validation is:

  1. Globally enable the segmented_vulnerability_report_export feature flag.
  2. Make sure you have a Group with at least 1 project with valid vulnerabilities in it
  3. Modify the VULNERABILITY_READS_PARTIAL_FILE_BATCH_SIZE constant in ee/app/services/vulnerability_exports/export_service.rb to a small number. If you're feeling adventurous, you can set it to 1. It should work.
  4. Go to the Group vulnerability report page containing the project and click the export.
  5. The export should be correct and as expected.

Related to #440163

Edited by Gregory Havenga

Merge request reports