Skip to content

Remove expired pipeline artifacts

Max Orefice requested to merge mo-add-destroy-artifact-service into master

Ref: #234042 (closed)

Final step to clean up expired pipeline artifacts.

This feature is behind a feature flag (coverage_report_view).

What does this MR do?

This MR creates a new service which destroys expired pipeline_artifacts.

It includes the following items:

  • Adds a new index for expire_at to ci_pipeline_artifacts
  • Modifies DestroyExpiredJobArtifactsService to clean up Ci::PipelineArtifact

Why are we doing this?

You can visualize how much data is being stored on with your gitlab instance by visiting this page.

Following up !39114 (merged), where we introduced a new column (expire_at defaulted to 1 week).

As pointed by @dosuken123 when introducing our new model in !37969 (merged).

Our system was missing an expiration mechanism:

We've introduced Pipeline Artifacts but have not implemented an expiration mechanism for them. We should implement a mechanism for expiration in order to not put pressure on our storage.

This MR implements the necessary logic to destroy pipeline_artifacts older than 1 week to reduce user' storage capacity.

Migration output

$ bin/rails db:migrate
== 20200827150057 AddIndexExpireAtToPipelineArtifacts: migrating ==============
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:ci_pipeline_artifacts, :expire_at, {:name=>"index_ci_pipeline_artifacts_on_expire_at", :algorithm=>:concurrently})
   -> 0.0032s
-- add_index(:ci_pipeline_artifacts, :expire_at, {:name=>"index_ci_pipeline_artifacts_on_expire_at", :algorithm=>:concurrently})
   -> 0.0033s
== 20200827150057 AddIndexExpireAtToPipelineArtifacts: migrated (0.0069s) =====

Migration Rollback

$ bin/rails db:rollback
== 20200827150057 AddIndexExpireAtToPipelineArtifacts: reverting ==============
-- transaction_open?()
   -> 0.0000s
-- indexes(:ci_pipeline_artifacts)
   -> 0.0037s
-- remove_index(:ci_pipeline_artifacts, {:algorithm=>:concurrently, :name=>"index_ci_pipeline_artifacts_on_expire_at"})
   -> 0.0020s
== 20200827150057 AddIndexExpireAtToPipelineArtifacts: reverted (0.0061s) =====

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team
Edited by Max Orefice

Merge request reports