Skip to content

Add "keep latest artifact" option for projects

Jose Ivan Vargas requested to merge jivanvl-keep-latest-artifact-project-level into master

What does this MR do?

This adds the necessary backend code and a database migration to add the "Keep latest artifact" option toggle to the Project -> Settings -> CI/CD page.

If the checkbox is unchecked the application will no longer keep the artifacts for the latest pipeline on a given branch when new pipelines are created or when new pipelines succeed. We will later provide a separate way to clear out historic artifacts that are 'locked'.

#241026 (closed)

Technical Details

Artifact Locking logic

This preserves the existing behavior for users by keeping the latest artifacts enforced by defaulting the new column to true at the database level.

Locked pipeline artifacts will not be deleted even if they expire. Unlocked pipeline artifacts will be deleted when they expire.

When a pipeline is created currently we rely on the database default to set the value for 'locked' to true. This MR instead sets the value based on keep_latest_artifact. We then unlock the relevant artifacts at each transition.

To update settings via a mutation:

mutation {
  ciCdSettingsUpdate(input: {fullPath: "root/merge-trains", keepLatestArtifact: false}) {
    errors
  }
}

To fetch settings via a graphql query:

query {
  project(fullPath: "root/merge-trains") {
    id,
    ciCdSettings {
      keepLatestArtifact
    }
  }
}

Database Migration Output

Up
   (0.3ms)  BEGIN
   (0.2ms)  SET LOCAL lock_timeout TO '100ms'
-- add_column(:project_ci_cd_settings, :keep_latest_artifact, :boolean, {:default=>true, :null=>false})
   (15.5ms)  ALTER TABLE "project_ci_cd_settings" ADD "keep_latest_artifact" boolean DEFAULT TRUE NOT NULL
   -> 0.0168s
   (1.0ms)  COMMIT
   (0.3ms)  RESET idle_in_transaction_session_timeout; RESET lock_timeout

database lab:

exec ALTER TABLE "project_ci_cd_settings" ADD "keep_latest_artifact" boolean DEFAULT TRUE NOT NULL
The query has been executed. Duration: 68.000 ms (edited)
Down
   (0.2ms)  BEGIN
   (0.1ms)  SET LOCAL lock_timeout TO '100ms'
-- remove_column(:project_ci_cd_settings, :keep_latest_artifact)
   (2.9ms)  ALTER TABLE "project_ci_cd_settings" DROP COLUMN "keep_latest_artifact"
   -> 0.0036s
   (0.5ms)  COMMIT
   (0.2ms)  RESET idle_in_transaction_session_timeout; RESET lock_timeout

database lab:

exec ALTER TABLE "project_ci_cd_settings" DROP COLUMN "keep_latest_artifact"
The query has been executed. Duration: 17.000 ms (edited)

Guard clause Database output

From prod I found a ref with many pipelines:

 ci_ref_id | pipeline_count
-----------+--------
       499 | 132326

This is what I plan to use for the guard clause:

ci_ref.pipelines.where(locked: :artifacts_locked).exists?

generated query:

SELECT 1 AS one FROM "ci_pipelines" WHERE "ci_pipelines"."ci_ref_id" = 499 AND "ci_pipelines"."locked" = 1 LIMIT 1

from database lab:

Time: 14.072 ms
  - planning: 0.231 ms
  - execution: 13.841 ms
    - I/O read: 13.685 ms
    - I/O write: N/A
 Limit  (cost=0.56..0.66 rows=1 width=4) (actual time=11.668..11.670 rows=1 loops=1)
   Buffers: shared hit=3 read=5
   I/O Timings: read=11.395
   ->  Index Only Scan using idx_ci_pipelines_artifacts_locked on public.ci_pipelines  (cost=0.56..1568.26 rows=16903 width=4) (actual time=11.666..11.666 rows=1 loops=1)
         Index Cond: (ci_pipelines.ci_ref_id = 499)
         Heap Fetches: 0
         Buffers: shared hit=3 read=5
         I/O Timings: read=11.395

Screenshots (strongly suggested)

N/A

Does this MR meet the acceptance criteria?

Conformity

Edited by Allison Browne

Merge request reports