Clean up old Maven snapshot builds with the cleanup policy (!266409) · Merge requests · GitLab.org / GitLab

What does this MR do and why?

The cleanup policy rule Number of duplicated assets to keep never removes old Maven snapshot builds. Each snapshot build puts a timestamp and build number in its file names, like my-app-1.0-20220922.125608-3.jar. The rule only matches files with exactly the same name, so no two builds ever count as duplicates. Projects that publish snapshots from CI keep every build forever.

This MR applies the rule to Maven snapshot versions (-SNAPSHOT):

Rank each file, not each build. The policy strips the [timestamp, build number] from each file name, so my-app-1.0-20240102.090000-11.jar and my-app-1.0-20240101.100000-10.jar are both copies of my-app-1.0.jar. Gitlab::Regex::Packages::MAVEN_SNAPSHOT_DYNAMIC_PARTS is the same regex the upload path uses for its duplicate check.
Keep the newest N copies of each file. For each stripped name, the policy keeps the copies from the N newest builds. Older copies are marked pending_destruction. A file that only older builds have, like a classifier deployed from a separate job, keeps its newest copy.
Order by name, not upload order. Copies are sorted by build timestamp, then build number as an integer, so build 10 sorts after build 9. Retries and partial uploads can reorder file IDs, so IDs are not used.
Files without a build in their name are unchanged. maven-metadata.xml stays on the existing exact-name rule, which keeps the newest copy.
Release versions are unchanged. Only packages whose version ends in -SNAPSHOT (case-sensitive) are read.
Time budget. The work runs inside the existing package batches from !264239 (merged) and checks the service deadline while it lists files and while it marks them.

Behind the packages_cleanup_policy_maven_snapshot_builds feature flag, off by default. Without the flag, every project with a cleanup policy would lose its older snapshot builds on the first run after deploy, and deleted files cannot be restored.

References

Closes #375065 (closed)
Rollout issue: #637187
Builds on !264239 (merged) (batch the cleanup policy by package)
Parent epic: gitlab-org#8021

Screenshots or screen recordings

Not applicable: no UI change.

How to set up and validate locally

In a Rails console, create a project with a policy that keeps one duplicate and a snapshot package with three builds:

require 'rack/test'
FactoryBot::SyntaxRunner.define_method(:fixture_file_upload) { |*args, **kwargs| Rack::Test::UploadedFile.new(*args, **kwargs) }

project = FactoryBot.create(:project)
policy = FactoryBot.create(:packages_cleanup_policy, project: project, keep_n_duplicated_package_files: '1')
package = FactoryBot.create(:maven_package, project: project, version: '1.0-SNAPSHOT', package_files: [])
%w[20240101.100000-9 20240101.100000-10 20240102.090000-11].each do |build|
  %w[jar pom].each do |ext|
    FactoryBot.create(:package_file, :jar, package: package, file_name: "my-app-1.0-#{build}.#{ext}")
  end
end

Enable the flag and run the service:

Feature.enable(:packages_cleanup_policy_maven_snapshot_builds, project)
Packages::Cleanup::ExecutePolicyService.new(policy).execute

Check that only the newest build remains:

package.package_files.installable.pluck(:file_name)
# => ["my-app-1.0-20240102.090000-11.jar", "my-app-1.0-20240102.090000-11.pom"]

Database

New queries run by Packages::Cleanup::ExecutePolicyService for each batch of 500 packages, only when the packages_cleanup_policy_maven_snapshot_builds flag is on and Number of duplicate assets to keep is not All. All plans ran with EXPLAIN (ANALYZE, BUFFERS) on a gitlab-production-main thin clone.

<project_id> is a project with about 14k Maven packages, almost all snapshots.
<package_id> is the package with the second most installable files on GitLab.com (about 160k). It is a generic package, so the plans show the worst case for the file queries. The query shape does not depend on the package type. The largest Maven snapshot package found in the sample had 139 files.

1. Snapshot packages with old builds, loaded by id

The snapshot filter (version LIKE '%-SNAPSHOT', case-sensitive) now runs inside query 4. This query loads only the packages that query 4 returns, at most one batch of 500.

SELECT "packages_packages".* FROM "packages_packages"
WHERE "packages_packages"."id" IN (<package ids from query 4>)

Plan (gitlab-production-main thin clone, plan, with query 4 as the id source): Index Scan on index_packages_packages_on_id_and_created_at, one lookup per id. 131 rows, about 40 ms (0.3 ms per id), buffers: 608 hit / 47 read. The other 156 ms in that run is query 4.

2. Installable files of a snapshot package, in batches of 1,000

Start query:

SELECT "packages_package_files"."id" FROM "packages_package_files"
WHERE "packages_package_files"."package_id" = <package_id> AND "packages_package_files"."status" = 0
ORDER BY "packages_package_files"."id" ASC LIMIT 1

Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/79727/commands/187782 (Index Only Scan on index_packages_package_files_on_package_id_status_and_id, 1 row, 0.07 ms warm, buffers: 11 hit / 0 read)

Boundary query:

SELECT "packages_package_files"."id" FROM "packages_package_files"
WHERE "packages_package_files"."package_id" = <package_id> AND "packages_package_files"."status" = 0
  AND "packages_package_files"."id" >= <start_id>
ORDER BY "packages_package_files"."id" ASC LIMIT 1 OFFSET 1000

Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/79727/commands/187783 (Index Only Scan on index_packages_package_files_on_package_id_status_and_id, 1,001 rows scanned, 1.0 ms cold, buffers: 43 hit / 1 read)

pluck(:id, :file_name) for the batch:

SELECT "packages_package_files"."id", "packages_package_files"."file_name" FROM "packages_package_files"
WHERE "packages_package_files"."package_id" = <package_id> AND "packages_package_files"."status" = 0
  AND "packages_package_files"."id" >= <start_id> AND "packages_package_files"."id" < <end_id>

Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/79727/commands/187784 (Index Only Scan on idx_pkgs_installable_package_files_on_package_id_id_file_name, 1,000 rows, 33 ms cold, buffers: 24 hit / 25 read, 0 heap fetches)

3. Mark old copies for destruction, 200 ids per call

MarkPackageFilesForDestructionService on Packages::PackageFile.id_in(ids). These are the existing service's queries with a new relation.

Start query:

SELECT "packages_package_files"."id" FROM "packages_package_files"
WHERE "packages_package_files"."id" IN (<200 ids>)
ORDER BY "packages_package_files"."id" ASC LIMIT 1

Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/79727/commands/187785 (Index Only Scan on packages_package_files_pkey, 1 row, 7.8 ms cold, buffers: 1 hit / 4 read)

Boundary query:

SELECT "packages_package_files"."id" FROM "packages_package_files"
WHERE "packages_package_files"."id" IN (<200 ids>) AND "packages_package_files"."id" >= <start_id>
ORDER BY "packages_package_files"."id" ASC LIMIT 1 OFFSET 200

Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/79727/commands/187786 (Index Only Scan on packages_package_files_pkey, 200 rows scanned, 2.9 ms cold, buffers: 26 hit / 8 read)

update_all:

UPDATE "packages_package_files" SET "status" = 1
WHERE "packages_package_files"."id" IN (<200 ids>) AND "packages_package_files"."id" >= <start_id>

Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/79727/commands/187787 (Index Scan on packages_package_files_pkey, 200 rows, 915 ms cold / 537 ms on a second run, buffers: 12,775 hit / 1,108 read. The scan takes 181 ms; the rest is index and WAL upkeep for 200 rows. This matches the existing duplicate-file path, which also updates 200 rows per statement.)

4. Snapshot packages with old builds, one aggregate per batch

Added after daniyalAD's review. It runs once per 500-package batch. Only the packages it returns get their files listed (queries 2 and 3).

SELECT "packages_package_files"."package_id" FROM "packages_package_files"
WHERE "packages_package_files"."status" = 0
  AND "packages_package_files"."package_id" IN (
    SELECT "packages_packages"."id" FROM "packages_packages"
    WHERE "packages_packages"."project_id" = <project_id>
      AND "packages_packages"."id" >= <start_id> AND "packages_packages"."id" < <end_id>
      AND "packages_packages"."package_type" = 1
      AND "packages_packages"."version" LIKE '%-SNAPSHOT')
  AND (file_name ~ '-\d{8}\.\d{6}-\d+')
GROUP BY "packages_package_files"."package_id",
  regexp_replace("packages_package_files"."file_name", '-\d{8}\.\d{6}-\d+', '')
HAVING (COUNT(*) > <keep_n>)

Plan: gitlab-production-main thin clone, cold plan and warm plan. The full plans are below.

Shape: Index Scan on idx_installable_helm_pkgs_on_project_id_id for the batch (155 snapshot packages), then a nested loop of Index Only Scans on idx_pkgs_installable_package_files_on_package_id_id_file_name, then an Incremental Sort and a GroupAggregate. No Seq Scan.
Cold: 2.13 s execution, 1.91 s of it I/O. Buffers: 3,350 hit / 1,991 read.
Warm: 154 ms. Buffers: 5,341 hit / 0 read.
Batch: the same public project as plans 1 to 3, a single batch with 362 packages. Its 155 snapshot packages hold 20,941 build files (about 135 each), so this is a heavy case. daniyalAD's 496-package batch with two files per package took 11 ms warm and 2,654 buffers (188804).
Compared with before: the old path listed the same files for every snapshot package, with three statements per package (465 for this batch). The aggregate reads them once, in one statement, and skips the listing for packages with nothing to delete.

Cold plan (189710)

 GroupAggregate  (cost=86.39..257.05 rows=1 width=40) (actual time=22.122..2132.117 rows=895 loops=1)
   Group Key: packages_package_files.package_id, (regexp_replace((packages_package_files.file_name)::text, '-\d{8}\.\d{6}-\d+'::text, ''::text))
   Filter: (count(*) > 1)
   Rows Removed by Filter: 180
   Buffers: shared hit=3350 read=1991 dirtied=577
   WAL: records=748 fpi=577 bytes=4578431
   I/O Timings: read=1914.727 write=0.000
   ->  Incremental Sort  (cost=86.39..256.99 rows=3 width=40) (actual time=22.113..2129.475 rows=20941 loops=1)
         Sort Key: packages_package_files.package_id, (regexp_replace((packages_package_files.file_name)::text, '-\d{8}\.\d{6}-\d+'::text, ''::text))
         Presorted Key: packages_package_files.package_id
         Full-sort Groups: 121  Sort Method: quicksort  Average Memory: 29kB  Peak Memory: 29kB
         Pre-sorted Groups: 95  Sort Method: quicksort  Average Memory: 36kB  Peak Memory: 38kB
         Buffers: shared hit=3350 read=1991 dirtied=577
         WAL: records=748 fpi=577 bytes=4578431
         I/O Timings: read=1914.727 write=0.000
         ->  Nested Loop  (cost=1.14..256.86 rows=3 width=40) (actual time=12.522..2100.616 rows=20941 loops=1)
               Buffers: shared hit=3344 read=1991 dirtied=577
               WAL: records=748 fpi=577 bytes=4578431
               I/O Timings: read=1914.727 write=0.000
               ->  Index Scan using idx_installable_helm_pkgs_on_project_id_id on public.packages_packages  (cost=0.56..174.93 rows=3 width=8) (actual time=6.633..449.653 rows=155 loops=1)
                     Index Cond: ((packages_packages.project_id = 62043363) AND (packages_packages.id >= 32027890) AND (packages_packages.id < 57184557))
                     Filter: (((packages_packages.version)::text ~~ '%-SNAPSHOT'::text) AND (packages_packages.package_type = 1))
                     Rows Removed by Filter: 207
                     Buffers: shared hit=4 read=367 dirtied=37
                     WAL: records=37 fpi=37 bytes=297685
                     I/O Timings: read=445.735 write=0.000
               ->  Index Only Scan using idx_pkgs_installable_package_files_on_package_id_id_file_name on public.packages_package_files  (cost=0.57..26.73 rows=58 width=48) (actual time=0.933..10.148 rows=135 loops=155)
                     Index Cond: (packages_package_files.package_id = packages_packages.id)
                     Heap Fetches: 1110
                     Filter: ((packages_package_files.file_name)::text ~ '-\d{8}\.\d{6}-\d+'::text)
                     Rows Removed by Filter: 45
                     Buffers: shared hit=3340 read=1624 dirtied=540
                     WAL: records=711 fpi=540 bytes=4280746
                     I/O Timings: read=1468.992 write=0.000
Settings: work_mem = '230MB', jit = 'off', random_page_cost = '1.5', seq_page_cost = '4', effective_cache_size = '472585MB'
Query ID: 483856996987667622

Warm plan (189711)

 GroupAggregate  (cost=86.39..257.05 rows=1 width=40) (actual time=0.665..153.696 rows=895 loops=1)
   Group Key: packages_package_files.package_id, (regexp_replace((packages_package_files.file_name)::text, '-\d{8}\.\d{6}-\d+'::text, ''::text))
   Filter: (count(*) > 1)
   Rows Removed by Filter: 180
   Buffers: shared hit=5341
   I/O Timings: read=0.000 write=0.000
   ->  Incremental Sort  (cost=86.39..256.99 rows=3 width=40) (actual time=0.657..151.428 rows=20941 loops=1)
         Sort Key: packages_package_files.package_id, (regexp_replace((packages_package_files.file_name)::text, '-\d{8}\.\d{6}-\d+'::text, ''::text))
         Presorted Key: packages_package_files.package_id
         Full-sort Groups: 121  Sort Method: quicksort  Average Memory: 29kB  Peak Memory: 29kB
         Pre-sorted Groups: 95  Sort Method: quicksort  Average Memory: 36kB  Peak Memory: 38kB
         Buffers: shared hit=5341
         I/O Timings: read=0.000 write=0.000
         ->  Nested Loop  (cost=1.14..256.86 rows=3 width=40) (actual time=0.070..125.622 rows=20941 loops=1)
               Buffers: shared hit=5335
               I/O Timings: read=0.000 write=0.000
               ->  Index Scan using idx_installable_helm_pkgs_on_project_id_id on public.packages_packages  (cost=0.56..174.93 rows=3 width=8) (actual time=0.020..0.529 rows=155 loops=1)
                     Index Cond: ((packages_packages.project_id = 62043363) AND (packages_packages.id >= 32027890) AND (packages_packages.id < 57184557))
                     Filter: (((packages_packages.version)::text ~~ '%-SNAPSHOT'::text) AND (packages_packages.package_type = 1))
                     Rows Removed by Filter: 207
                     Buffers: shared hit=371
                     I/O Timings: read=0.000 write=0.000
               ->  Index Only Scan using idx_pkgs_installable_package_files_on_package_id_id_file_name on public.packages_package_files  (cost=0.57..26.73 rows=58 width=48) (actual time=0.007..0.342 rows=135 loops=155)
                     Index Cond: (packages_package_files.package_id = packages_packages.id)
                     Heap Fetches: 1110
                     Filter: ((packages_package_files.file_name)::text ~ '-\d{8}\.\d{6}-\d+'::text)
                     Rows Removed by Filter: 45
                     Buffers: shared hit=4964
                     I/O Timings: read=0.000 write=0.000
Settings: work_mem = '230MB', jit = 'off', random_page_cost = '1.5', seq_page_cost = '4', effective_cache_size = '472585MB'
Query ID: 483856996987667622

Rollout and risks

Data loss on enable. Enabling the flag for a project marks its older snapshot builds for destruction on the next run. The rollout issue starts with internal projects.
First-run volume. Projects with long-lived snapshot versions can mark many files in one run. The signal is marked_package_files_total_count and duration_s for Packages::Cleanup::ExecutePolicyWorker.
Known limitation: failed partial deploys. If a deploy fails after it uploads some files, and the policy runs before the next successful deploy, the policy can keep the copies from the failed build and delete the copies that maven-metadata.xml lists. Clients then get a 404 for those files. The next successful deploy fixes it, because it uploads a new copy of every file.
Memory. For each snapshot package the service holds [id, file_name] for its files, read 1,000 at a time. The largest package on GitLab.com has about 158k files.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Snapshot file listing. The Maven snapshot pass lists each snapshot package's installable files separately (start, boundary and pluck queries per 1,000 files). Each one is an index-only scan on idx_pkgs_installable_package_files_on_package_id_id_file_name / index_packages_package_files_on_package_id_status_and_id, at 0.1–0.14 ms on a Database Lab clone. A full pass over 14,000 snapshot packages is about 42,000 queries, or roughly 40–70 s including round trips, inside the 250 s deadline. Listing the whole 500-package batch at once was slower: each each_batch step re-reads and sorts every file of the batch (about 17–19 ms per query at 10,000 files), and the cost grows with the number of files.

Snapshot batch query (after sylviashen's review).

SQL and plans

Superseded: query 4 now selects the snapshot packages; see query 1 above.

1. Snapshot packages in the batch

SELECT "packages_packages".* FROM "packages_packages"
WHERE "packages_packages"."project_id" = <project_id>
  AND "packages_packages"."id" >= <start_id> AND "packages_packages"."id" < <end_id>
  AND "packages_packages"."package_type" = 1
  AND "packages_packages"."version" ILIKE '%-SNAPSHOT'

Plan (gitlab-production-main thin clone, Joe command 568213): one Index Scan on idx_installable_helm_pkgs_on_project_id_id, with package_type and the ILIKE as filters on the batch rows. 155 rows, 4.8 ms, buffers: 371 hit / 0 read.

Old shape on the same IDs (id IN (SELECT id ...), Joe command 568215): Nested Loop with an extra Index Scan on index_packages_packages_on_id_and_created_at per id. 155 rows, 6.2 ms warm (683 ms cold), buffers: 1,933 hit.

The IDs are one full batch of a public project with 334 Maven packages, 155 of them snapshots. The original <project_id> behind plan 187740 couldn't be recovered from Joe (the command has expired), so both plans here use the same new IDs.

New plan

 Index Scan using idx_installable_helm_pkgs_on_project_id_id on public.packages_packages  (cost=0.56..176.12 rows=2 width=120) (actual time=0.031..0.699 rows=155 loops=1)
   Index Cond: ((packages_packages.project_id = <project_id>) AND (packages_packages.id >= <start_id>) AND (packages_packages.id < <end_id>))
   Filter: (((packages_packages.version)::text ~~* '%-SNAPSHOT'::text) AND (packages_packages.package_type = 1))
   Rows Removed by Filter: 207
   Buffers: shared hit=371
   I/O Timings: read=0.000 write=0.000

Old plan, same IDs

 Nested Loop  (cost=1.13..511.89 rows=2 width=120) (actual time=0.043..1.538 rows=155 loops=1)
   Buffers: shared hit=1933
   I/O Timings: read=0.000 write=0.000
   ->  Index Only Scan using idx_installable_helm_pkgs_on_project_id_id on public.packages_packages packages_packages_1  (cost=0.56..23.99 rows=136 width=8) (actual time=0.029..0.168 rows=362 loops=1)
         Index Cond: ((packages_packages_1.project_id = <project_id>) AND (packages_packages_1.id >= <start_id>) AND (packages_packages_1.id < <end_id>))
         Heap Fetches: 15
         Buffers: shared hit=123
         I/O Timings: read=0.000 write=0.000
   ->  Index Scan using index_packages_packages_on_id_and_created_at on public.packages_packages  (cost=0.56..3.59 rows=1 width=120) (actual time=0.004..0.004 rows=0 loops=362)
         Index Cond: (packages_packages.id = packages_packages_1.id)
         Filter: (((packages_packages.version)::text ~~* '%-SNAPSHOT'::text) AND (packages_packages.package_type = 1))
         Rows Removed by Filter: 1
         Buffers: shared hit=1810
         I/O Timings: read=0.000 write=0.000

Edited Nov 06, 2026 by Tim Rizzi

Clean up old Maven snapshot builds with the cleanup policy