Skip to content

Resolve ""Keep latest artifacts" keeps artifacts of failed pipelines"

Sam Kim requested to merge 266958-keep-latest-artifacts-keeps-artifacts-of-failed-pipelines into master

What does this MR do and why?

Currently when we have "Keep latest artifacts" enabled and have recent failing pipelines, the failing pipelines are considered to be "latest pipelines" until a successful pipeline is run. This means that we currently store all artifacts from the failing pipelines.

The purpose of this MR is to remove artifacts of failed pipelines even if they are considered "latest". The behaviour we want when we have failed pipelines is to not store artifacts for any failed pipelines.

This is consistent with what our documentation, as the section here is titled “Keep from most recent successful jobs”: https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#keep-artifacts-from-most-recent-successful-jobs

A quick summary of the behaviour that we want:

image

Image of comment is from #382019 (comment 1228431417)

Screenshots or screen recordings

Please see the following video for a walkthrough of the current problem we are facing: https://gitlab.com/gitlab-org/gitlab/uploads/96a1af15e17b15ac98edb2cf9be0b41e/GitLabKeepLatestArtifactFailingPipeline.mp4

How to set up and validate locally

  1. Create a project and setup a CI/CD pipeline with some artifacts
  2. Add a job which fails, for example by setting the script to exit 1
  3. Run the pipeline a few times

Database testing

  1. This MR adds the method last_successful_pipeline_id:
Query 
SELECT "ci_pipelines"."id" FROM "ci_pipelines" WHERE "ci_pipelines"."ci_ref_id" = 499 AND ("ci_pipelines"."source" IN (1, 2, 3, 4, 5, 6, 7, 8, 10, 11) OR "ci_pipelines"."source" IS NULL) AND ("ci_pipelines"."status" IN ('success')) ORDER BY "ci_pipelines"."id" DESC LIMIT 1
Plan & Summary 
 Limit  (cost=0.57..0.63 rows=1 width=4) (actual time=87.129..87.131 rows=1 loops=1)
   Buffers: shared hit=4 read=780
   I/O Timings: read=79.376 write=0.000
   ->  Index Only Scan using index_ci_pipelines_on_ci_ref_id_and_more on public.ci_pipelines  (cost=0.57..12691.26 rows=226032 width=4) (actual time=87.126..87.127 rows=1 loops=1)
         Index Cond: ((ci_pipelines.ci_ref_id = 499) AND (ci_pipelines.status = 'success'::text))
         Heap Fetches: 0
         Filter: ((ci_pipelines.source = ANY ('{1,2,3,4,5,6,7,8,10,11}'::integer[])) OR (ci_pipelines.source IS NULL))
         Rows Removed by Filter: 0
         Buffers: shared hit=4 read=780
         I/O Timings: read=79.376 write=0.000
Time: 87.734 ms
  - planning: 0.562 ms
  - execution: 87.172 ms
    - I/O read: 79.376 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 4 (~32.00 KiB) from the buffer pool
  - reads: 780 (~6.10 MiB) from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0
  1. This MR changes the unlock_pipelines_query method with a conditional that has two separate queries depending on ref_failed:

if ref_failed the after_pipeline method is used:

Query 
SELECT "ci_pipelines".* FROM "ci_pipelines" WHERE "ci_pipelines"."ci_ref_id" = 499 AND "ci_pipelines"."locked" = 1 AND "ci_pipelines"."id" > 150 AND "ci_pipelines"."id" NOT IN (WITH RECURSIVE "base_and_descendants" AS ((SELECT "ci_pipelines".* FROM "ci_pipelines" WHERE "ci_pipelines"."id" = 150)
UNION
(SELECT "ci_pipelines".* FROM "ci_pipelines", "base_and_descendants", "ci_sources_pipelines" WHERE "ci_sources_pipelines"."pipeline_id" = "ci_pipelines"."id" AND "ci_sources_pipelines"."source_pipeline_id" = "base_and_descendants"."id" AND "ci_sources_pipelines"."source_project_id" = "ci_sources_pipelines"."project_id")) SELECT "id" FROM "base_and_descendants" AS "ci_pipelines")
Plan & Summary 
 Index Scan using idx_ci_pipelines_artifacts_locked on public.ci_pipelines  (cost=466.29..161536.01 rows=64715 width=347) (actual time=20.847..604243.879 rows=308462 loops=1)
   Index Cond: ((ci_pipelines.ci_ref_id = 499) AND (ci_pipelines.id > 150))
   Filter: (NOT (hashed SubPlan 2))
   Rows Removed by Filter: 0
   Buffers: shared hit=10084 read=297637 dirtied=3169
   I/O Timings: read=598099.034 write=0.000
   SubPlan 2
     ->  CTE Scan on base_and_descendants ci_pipelines_3  (cost=465.47..465.69 rows=11 width=4) (actual time=0.040..0.072 rows=0 loops=1)
           Buffers: shared hit=4
           I/O Timings: read=0.000 write=0.000
           CTE base_and_descendants
             ->  Recursive Union  (cost=0.57..465.47 rows=11 width=347) (actual time=0.039..0.060 rows=0 loops=1)
                   Buffers: shared hit=4
                   I/O Timings: read=0.000 write=0.000
                   ->  Index Scan using ci_pipelines_pkey on public.ci_pipelines ci_pipelines_1  (cost=0.57..3.59 rows=1 width=347) (actual time=0.031..0.033 rows=0 loops=1)
                         Index Cond: (ci_pipelines_1.id = 150)
                         Buffers: shared hit=4
                         I/O Timings: read=0.000 write=0.000
                   ->  Nested Loop  (cost=1.14..46.17 rows=1 width=347) (actual time=0.005..0.019 rows=0 loops=1)
                         I/O Timings: read=0.000 write=0.000
                         ->  Nested Loop  (cost=0.56..42.65 rows=1 width=4) (actual time=0.004..0.013 rows=0 loops=1)
                               I/O Timings: read=0.000 write=0.000
                               ->  WorkTable Scan on base_and_descendants  (cost=0.00..0.20 rows=10 width=4) (actual time=0.003..0.005 rows=0 loops=1)
                                     I/O Timings: read=0.000 write=0.000
                               ->  Index Scan using index_ci_sources_pipelines_on_source_pipeline_id on public.ci_sources_pipelines  (cost=0.56..4.23 rows=1 width=8) (actual time=0.000..0.000 rows=0 loops=0)
                                     Index Cond: (ci_sources_pipelines.source_pipeline_id = base_and_descendants.id)
                                     Filter: (ci_sources_pipelines.source_project_id = ci_sources_pipelines.project_id)
                                     Rows Removed by Filter: 0
                                     I/O Timings: read=0.000 write=0.000
                         ->  Index Scan using ci_pipelines_pkey on public.ci_pipelines ci_pipelines_2  (cost=0.57..3.52 rows=1 width=347) (actual time=0.000..0.000 rows=0 loops=0)
                               Index Cond: (ci_pipelines_2.id = ci_sources_pipelines.pipeline_id)
                               I/O Timings: read=0.000 write=0.000

Time: 10.076 min
  - planning: 6.047 ms
  - execution: 10.076 min
    - I/O read: 9.968 min
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 10084 (~78.80 MiB) from the buffer pool
  - reads: 297637 (~2.30 GiB) from the OS file cache, including disk I/O
  - dirtied: 3169 (~24.80 MiB)
  - writes: 0

otherwise the before_pipeline method is used:

Query 
SELECT "ci_pipelines".* FROM "ci_pipelines" WHERE "ci_pipelines"."ci_ref_id" = 499 AND "ci_pipelines"."locked" = 1 AND "ci_pipelines"."id" < 150 AND "ci_pipelines"."id" NOT IN (WITH RECURSIVE "base_and_descendants" AS ((SELECT "ci_pipelines".* FROM "ci_pipelines" WHERE "ci_pipelines"."id" = 150)
UNION
(SELECT "ci_pipelines".* FROM "ci_pipelines", "base_and_descendants", "ci_sources_pipelines" WHERE "ci_sources_pipelines"."pipeline_id" = "ci_pipelines"."id" AND "ci_sources_pipelines"."source_pipeline_id" = "base_and_descendants"."id" AND "ci_sources_pipelines"."source_project_id" = "ci_sources_pipelines"."project_id")) SELECT "id" FROM "base_and_descendants" AS "ci_pipelines")
Plan & Summary 
 Index Scan using idx_ci_pipelines_artifacts_locked on public.ci_pipelines  (cost=466.29..469.32 rows=1 width=347) (actual time=0.357..0.359 rows=0 loops=1)
   Index Cond: ((ci_pipelines.ci_ref_id = 499) AND (ci_pipelines.id < 150))
   Filter: (NOT (hashed SubPlan 2))
   Rows Removed by Filter: 0
   Buffers: shared hit=3 read=4
   I/O Timings: read=0.314 write=0.000
   SubPlan 2
     ->  CTE Scan on base_and_descendants ci_pipelines_3  (cost=465.47..465.69 rows=11 width=4) (actual time=0.000..0.000 rows=0 loops=0)
           I/O Timings: read=0.000 write=0.000
           CTE base_and_descendants
             ->  Recursive Union  (cost=0.57..465.47 rows=11 width=347) (actual time=0.000..0.000 rows=0 loops=0)
                   I/O Timings: read=0.000 write=0.000
                   ->  Index Scan using ci_pipelines_pkey on public.ci_pipelines ci_pipelines_1  (cost=0.57..3.59 rows=1 width=347) (actual time=0.000..0.000 rows=0 loops=0)
                         Index Cond: (ci_pipelines_1.id = 150)
                         I/O Timings: read=0.000 write=0.000
                   ->  Nested Loop  (cost=1.14..46.17 rows=1 width=347) (actual time=0.000..0.000 rows=0 loops=0)
                         I/O Timings: read=0.000 write=0.000
                         ->  Nested Loop  (cost=0.56..42.65 rows=1 width=4) (actual time=0.000..0.000 rows=0 loops=0)
                               I/O Timings: read=0.000 write=0.000
                               ->  WorkTable Scan on base_and_descendants  (cost=0.00..0.20 rows=10 width=4) (actual time=0.000..0.000 rows=0 loops=0)
                                     I/O Timings: read=0.000 write=0.000
                               ->  Index Scan using index_ci_sources_pipelines_on_source_pipeline_id on public.ci_sources_pipelines  (cost=0.56..4.23 rows=1 width=8) (actual time=0.000..0.000 rows=0 loops=0)
                                     Index Cond: (ci_sources_pipelines.source_pipeline_id = base_and_descendants.id)
                                     Filter: (ci_sources_pipelines.source_project_id = ci_sources_pipelines.project_id)
                                     Rows Removed by Filter: 0
                                     I/O Timings: read=0.000 write=0.000
                         ->  Index Scan using ci_pipelines_pkey on public.ci_pipelines ci_pipelines_2  (cost=0.57..3.52 rows=1 width=347) (actual time=0.000..0.000 rows=0 loops=0)
                               Index Cond: (ci_pipelines_2.id = ci_sources_pipelines.pipeline_id)
                               I/O Timings: read=0.000 write=0.000
Time: 14.239 ms  
  - planning: 13.518 ms  
  - execution: 0.721 ms  
    - I/O read: 0.314 ms  
    - I/O write: 0.000 ms  
  
Shared buffers:  
  - hits: 3 (~24.00 KiB) from the buffer pool  
  - reads: 4 (~32.00 KiB) from the OS file cache, including disk I/O  
  - dirtied: 0  
  - writes: 0

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #266958 (closed)

Edited by Sam Kim

Merge request reports