Skip to content

Introduce Pipeline Artifact Uploader

Max Orefice requested to merge mo-pipeline-artifact-uploader into master

Part of #211410 (closed)

Ref: #234037 (closed), #234036 (closed)

Step 2 to make code coverage available for everyone.

This feature is behind a feature flag (coverage_report_view). The full feature has been developed on this POC branch.

The team decided to implement this feature with this approach according to this proposal validated by @ayufan.

What does this MR do?

This MR creates a new uploader for our new model introduced in !37969 (merged).

It includes the following items:

  • Adds Ci::PipelineArtifactUploader
  • Mount our new uploader to Ci::PipelineArtifact

Why are we doing this?

Today our Code Coverage feature does not scale properly for large projects containing a lot of code to analyze.

Processing data in memory and parsing big chunk of data have some ~performance implication as explained on this issue.

Here a diagram of tomorrow's architecture:

sequenceDiagram
    participant MergeRequest
    participant Pipeline
    participant PipelineArtifact
    MergeRequest->>Pipeline: has_coverage_reports?
    Note right of Pipeline: When pipeline is completed
    loop PipelineArtifactService
        Pipeline->>PipelineArtifact: Persist coverage report with object storage
    end
    Note right of MergeRequest: No more ReactiveCaching
    Pipeline->>MergeRequest: Read file from object storage

How the feature looks like?

You can visualize how this feature is showing up on the UI on our demo project.

Context

This MR represents a vertical slice of functionality for improving the ~performance of our Code coverate feature. This is slice 2/4.

This work will occur in 4 slices:

  1. Create data model - !37969 (merged)
  2. Introduce a new uploader - !38221 (merged)
  3. Process and persist reports when a pipeline is finished -
  4. Fetch report from new abstraction -

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team
Edited by Max Orefice

Merge request reports