Skip to content

Add PipelineArtifact table and model

Max Orefice requested to merge mo-introduct-pipeline-artifact into master

Part of #211410 (closed)

Step 1 to make code coverage available for everyone.

This feature is behind a feature flag (coverage_report_view). The full feature has been developed on this POC branch.

The team decided to implement this feature with this approach according to this proposal validated by @ayufan.

What does this MR do?

This MR creates a new data model which will be used to persist generated report consumable by gitlab frontend in a pipeline context.

It includes the following items:

  • Adds table ci_pipeline_artifact table
  • Adds relationship to Ci::Pipeline model
  • Adds relationship to Project (in case we want to partition our data later on)
  • Creates very skeleton for Ci::PipelineArtifact model

Why are we doing this?

Today our Code Coverage feature does not scale properly for large projects containing a lot of code to analyze.

Processing data in memory and parsing big chunk of data have some ~performance implication as explained on this issue.

Here a diagram of tomorrow's architecture:

sequenceDiagram
    participant MergeRequest
    participant Pipeline
    participant PipelineArtifact
    MergeRequest->>Pipeline: has_coverage_reports?
    Note right of Pipeline: When pipeline is completed
    loop PipelineArtifactService
        Pipeline->>PipelineArtifact: Persist coverage report with object storage
    end
    Note right of MergeRequest: No more ReactiveCaching
    Pipeline->>MergeRequest: Read file from object storage

How the feature looks like?

You can visualize how this feature is showing up on the UI on our demo project.

Context

This MR represents a vertical slice of functionality for improving the ~performance of our Code coverate feature. This is slice 1/4.

This work will occur in 4 slices:

  1. Create data model - !37969 (merged)
  2. Introduce a new uploader -
  3. Process and persist reports when a pipeline is finished -
  4. Fetch report from new abstraction -

Database Review

-----Click here for supplemental database review materials------

Up migration output: Table creation

$ bin/rails db:migrate
== 20200805150316 CreateCiPipelineArtifact: migrating =========================
-- table_exists?(:ci_pipeline_artifacts)
   -> 0.0004s
-- create_table(:ci_pipeline_artifacts)
   -> 0.0110s
-- transaction_open?()
   -> 0.0000s
-- execute("ALTER TABLE ci_pipeline_artifacts\nADD CONSTRAINT check_191b5850ec\nCHECK ( char_length(file) <= 255 )\nNOT VALID;\n")
   -> 0.0006s
-- execute("ALTER TABLE ci_pipeline_artifacts VALIDATE CONSTRAINT check_191b5850ec;")
   -> 0.0008s
== 20200805150316 CreateCiPipelineArtifact: migrated (0.0221s) ================


$ bin/rails dbconsole
psql (11.7)
Type "help" for help.

gitlabhq_development=# \d ci_pipeline_artifacts
                                       Table "public.ci_pipeline_artifacts"
   Column    |           Type           | Collation | Nullable |                      Default
-------------+--------------------------+-----------+----------+---------------------------------------------------
 id          | bigint                   |           | not null | nextval('ci_pipeline_artifacts_id_seq'::regclass)
 created_at  | timestamp with time zone |           | not null |
 updated_at  | timestamp with time zone |           | not null |
 pipeline_id | bigint                   |           | not null |
 project_id  | bigint                   |           | not null |
 size        | integer                  |           | not null |
 file_store  | integer                  |           | not null |
 file_type   | smallint                 |           | not null |
 file_format | smallint                 |           | not null |
 file        | text                     |           |          |
Indexes:
    "ci_pipeline_artifacts_pkey" PRIMARY KEY, btree (id)
    "index_ci_pipeline_artifacts_on_pipeline_id_and_file_type" UNIQUE, btree (pipeline_id, file_type)
    "index_ci_pipeline_artifacts_on_pipeline_id" btree (pipeline_id)
    "index_ci_pipeline_artifacts_on_project_id" btree (project_id)
Check constraints:
    "check_191b5850ec" CHECK (char_length(file) <= 255)
Foreign-key constraints:
    "fk_rails_4a70390ca6" FOREIGN KEY (project_id) REFERENCES projects(id) ON DELETE CASCADE
    "fk_rails_a9e811a466" FOREIGN KEY (pipeline_id) REFERENCES ci_pipelines(id) ON DELETE CASCADE

Rollback table creation

$ bin/rails db:rollback
== 20200805150316 CreateCiPipelineArtifact: reverting =========================
-- drop_table(:ci_pipeline_artifacts)
   -> 0.0112s
== 20200805150316 CreateCiPipelineArtifact: reverted (0.0113s) ================

Adding foreign key for pipeline

== 20200805151001 AddForeignKeyToPipelineIdOnPipelineArtifact: migrating ======
-- add_foreign_key(:ci_pipeline_artifacts, :ci_pipelines, {:column=>:pipeline_id, :on_delete=>:cascade})
   -> 0.0026s
== 20200805151001 AddForeignKeyToPipelineIdOnPipelineArtifact: migrated (0.0055s)

Rollback foreign key for pipeline

== 20200805151001 AddForeignKeyToPipelineIdOnPipelineArtifact: reverting ======
-- remove_foreign_key(:ci_pipeline_artifacts, {:column=>:pipeline_id})
   -> 0.0044s
== 20200805151001 AddForeignKeyToPipelineIdOnPipelineArtifact: reverted (0.0071s)

Adding foreign key for project

$ bin/rails db:migrate
== 20200805151726 AddForeignKeyToProjectIdOnPipelineArtifact: migrating =======
-- add_foreign_key(:ci_pipeline_artifacts, :projects, {:column=>:project_id, :on_delete=>:cascade})
   -> 0.0038s
== 20200805151726 AddForeignKeyToProjectIdOnPipelineArtifact: migrated (0.0065s)

Rollback foreign key for project

$ bin/rails db:rollback
== 20200805151726 AddForeignKeyToProjectIdOnPipelineArtifact: reverting =======
-- remove_foreign_key(:ci_pipeline_artifacts, {:column=>:project_id})
   -> 0.0099s
== 20200805151726 AddForeignKeyToProjectIdOnPipelineArtifact: reverted (0.0171s)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team
Edited by Max Orefice

Merge request reports