Create X-Ray scanner uploader job
Description:
The metadata JSON produced by the X-Ray scanner job needs to be imported into the GitLab Rails PostgreSQL database for persistence and further usage.
This should happen asynchronously via a background job triggered upon a successful X-Ray scanner CI job completion.
Requirements:
- Add a background job to handle upload
- Job should be idempotent to allow retry logic
- Create a new database table to store the scan data
- Could use a jsonb column to store raw JSON data at first
- Apply unique constraint on project/repo ID
- Use UPSERT to only persist latest scan data per project
The existing pipeline security scan data integration can be used as a reference:
- Pipeline completion schedules
StoreScansWorkerhttps://gitlab.com/gitlab-org/gitlab/blob/a065bee3ff5e2fa7c03276dca5e59cd0b6cdab0b/ee/app/models/ee/ci/pipeline.rb#L67 - Calls
StoreScansServiceto save artifacts in batches https://gitlab.com/gitlab-org/gitlab/blob/41f2a30c9881ad8aaf2352e5ad9792263fdb3d36/ee/app/services/security/store\_scans\_service.rb#L16 -
StoreGroupedScansServiceparses and saves each one
Testing and error handling should follow established patterns.
Please advise if any part of the database integration needs clarification or adjustment.
Edited by Mikołaj Wawrzyniak