Add PipelineArtifact table and model
Part of #211410 (closed)
Step 1 to make code coverage available for everyone.
This feature is behind a feature flag (coverage_report_view
). The full feature has been developed on this POC branch.
The team decided to implement this feature with this approach according to this proposal validated by @ayufan.
What does this MR do?
This MR creates a new data model which will be used to persist generated report consumable by gitlab frontend in a pipeline context.
It includes the following items:
-
Adds table ci_pipeline_artifact
table -
Adds relationship to Ci::Pipeline
model -
Adds relationship to Project
(in case we want to partition our data later on) -
Creates very skeleton for Ci::PipelineArtifact
model
Why are we doing this?
Today our Code Coverage
feature does not scale properly for large projects containing a lot of code to analyze.
Processing data in memory and parsing big chunk of data have some ~performance implication as explained on this issue.
Here a diagram of tomorrow's architecture:
sequenceDiagram
participant MergeRequest
participant Pipeline
participant PipelineArtifact
MergeRequest->>Pipeline: has_coverage_reports?
Note right of Pipeline: When pipeline is completed
loop PipelineArtifactService
Pipeline->>PipelineArtifact: Persist coverage report with object storage
end
Note right of MergeRequest: No more ReactiveCaching
Pipeline->>MergeRequest: Read file from object storage
How the feature looks like?
You can visualize how this feature is showing up on the UI on our demo project.
Context
This MR represents a vertical slice of functionality for improving the ~performance of our Code coverate
feature. This is slice 1/4.
This work will occur in 4 slices:
- Create data model - !37969 (merged)
- Introduce a new uploader -
- Process and persist reports when a pipeline is finished -
- Fetch report from new abstraction -
Database Review
-----Click here for supplemental database review materials------
Up migration output: Table creation
$ bin/rails db:migrate
== 20200805150316 CreateCiPipelineArtifact: migrating =========================
-- table_exists?(:ci_pipeline_artifacts)
-> 0.0004s
-- create_table(:ci_pipeline_artifacts)
-> 0.0110s
-- transaction_open?()
-> 0.0000s
-- execute("ALTER TABLE ci_pipeline_artifacts\nADD CONSTRAINT check_191b5850ec\nCHECK ( char_length(file) <= 255 )\nNOT VALID;\n")
-> 0.0006s
-- execute("ALTER TABLE ci_pipeline_artifacts VALIDATE CONSTRAINT check_191b5850ec;")
-> 0.0008s
== 20200805150316 CreateCiPipelineArtifact: migrated (0.0221s) ================
$ bin/rails dbconsole
psql (11.7)
Type "help" for help.
gitlabhq_development=# \d ci_pipeline_artifacts
Table "public.ci_pipeline_artifacts"
Column | Type | Collation | Nullable | Default
-------------+--------------------------+-----------+----------+---------------------------------------------------
id | bigint | | not null | nextval('ci_pipeline_artifacts_id_seq'::regclass)
created_at | timestamp with time zone | | not null |
updated_at | timestamp with time zone | | not null |
pipeline_id | bigint | | not null |
project_id | bigint | | not null |
size | integer | | not null |
file_store | integer | | not null |
file_type | smallint | | not null |
file_format | smallint | | not null |
file | text | | |
Indexes:
"ci_pipeline_artifacts_pkey" PRIMARY KEY, btree (id)
"index_ci_pipeline_artifacts_on_pipeline_id_and_file_type" UNIQUE, btree (pipeline_id, file_type)
"index_ci_pipeline_artifacts_on_pipeline_id" btree (pipeline_id)
"index_ci_pipeline_artifacts_on_project_id" btree (project_id)
Check constraints:
"check_191b5850ec" CHECK (char_length(file) <= 255)
Foreign-key constraints:
"fk_rails_4a70390ca6" FOREIGN KEY (project_id) REFERENCES projects(id) ON DELETE CASCADE
"fk_rails_a9e811a466" FOREIGN KEY (pipeline_id) REFERENCES ci_pipelines(id) ON DELETE CASCADE
Rollback table creation
$ bin/rails db:rollback
== 20200805150316 CreateCiPipelineArtifact: reverting =========================
-- drop_table(:ci_pipeline_artifacts)
-> 0.0112s
== 20200805150316 CreateCiPipelineArtifact: reverted (0.0113s) ================
Adding foreign key for pipeline
== 20200805151001 AddForeignKeyToPipelineIdOnPipelineArtifact: migrating ======
-- add_foreign_key(:ci_pipeline_artifacts, :ci_pipelines, {:column=>:pipeline_id, :on_delete=>:cascade})
-> 0.0026s
== 20200805151001 AddForeignKeyToPipelineIdOnPipelineArtifact: migrated (0.0055s)
Rollback foreign key for pipeline
== 20200805151001 AddForeignKeyToPipelineIdOnPipelineArtifact: reverting ======
-- remove_foreign_key(:ci_pipeline_artifacts, {:column=>:pipeline_id})
-> 0.0044s
== 20200805151001 AddForeignKeyToPipelineIdOnPipelineArtifact: reverted (0.0071s)
Adding foreign key for project
$ bin/rails db:migrate
== 20200805151726 AddForeignKeyToProjectIdOnPipelineArtifact: migrating =======
-- add_foreign_key(:ci_pipeline_artifacts, :projects, {:column=>:project_id, :on_delete=>:cascade})
-> 0.0038s
== 20200805151726 AddForeignKeyToProjectIdOnPipelineArtifact: migrated (0.0065s)
Rollback foreign key for project
$ bin/rails db:rollback
== 20200805151726 AddForeignKeyToProjectIdOnPipelineArtifact: reverting =======
-- remove_foreign_key(:ci_pipeline_artifacts, {:column=>:project_id})
-> 0.0099s
== 20200805151726 AddForeignKeyToProjectIdOnPipelineArtifact: reverted (0.0171s)
Does this MR meet the acceptance criteria?
Conformity
- [-] Changelog entry - Changelog to be included with feature flag removal.
- [-] Documentation (if required) - Documentation to be included with feature flag removal.
-
Code review guidelines - [-] Merge request performance guidelines
-
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
- [-] Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
- [-] Tested in all supported browsers
- [-] Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
- [-] Label as security and @ mention
@gitlab-com/gl-security/appsec
- [-] The MR includes necessary changes to maintain consistency between UI API, email, or other methods
- [-] Security reports checked/validated by a reviewer from the AppSec team