Skip to content

Create X-Ray scanner uploader job

Description:

The metadata JSON produced by the X-Ray scanner job needs to be imported into the GitLab Rails PostgreSQL database for persistence and further usage.

This should happen asynchronously via a background job triggered upon a successful X-Ray scanner CI job completion.

Requirements:

  • Add a background job to handle upload
  • Job should be idempotent to allow retry logic
  • Create a new database table to store the scan data
    • Could use a jsonb column to store raw JSON data at first
  • Apply unique constraint on project/repo ID
  • Use UPSERT to only persist latest scan data per project

The existing pipeline security scan data integration can be used as a reference:

  1. Pipeline completion schedules StoreScansWorker https://gitlab.com/gitlab-org/gitlab/blob/a065bee3ff5e2fa7c03276dca5e59cd0b6cdab0b/ee/app/models/ee/ci/pipeline.rb#L67
  2. Calls StoreScansService to save artifacts in batches https://gitlab.com/gitlab-org/gitlab/blob/41f2a30c9881ad8aaf2352e5ad9792263fdb3d36/ee/app/services/security/store\_scans\_service.rb#L16
  3. StoreGroupedScansService parses and saves each one

Testing and error handling should follow established patterns.

Please advise if any part of the database integration needs clarification or adjustment.

Edited by Mikołaj Wawrzyniak