Update Maven snapshot cleanup slices directly instead of through each_batch
What does this MR do and why?
The Maven snapshot pass in Packages::Cleanup::ExecutePolicyService marked old snapshot files through Packages::MarkPackageFilesForDestructionService. That service runs each_batch over a relation that already holds a single 200-id slice, so each slice paid a start query and a boundary query before the UPDATE.
Each slice is now updated directly, matching mark_packages_for_destruction:
old_ids_by_name.values.flatten.each_slice(MARK_PACKAGE_FILES_FOR_DESTRUCTION_SERVICE_BATCH_SIZE) do |ids|
break :timeout if batch_deadline.past?
@counts[:marked_package_files_total_count] +=
::Packages::PackageFile.id_in(ids).update_all(status: :pending_destruction)
endThe snapshot timeout spec stubs batch_deadline instead of the destruction service.
Closes #639257
How to set up and validate locally
bin/rspec spec/services/packages/cleanup/execute_policy_service_spec.rb -e 'maven snapshot packages'Database query
Each slice runs a primary-key UPDATE (up to 200 ids, pending_destruction = 1). This is the same Packages::PackageFile.id_in(ids).update_all(status: :pending_destruction) already used by mark_packages_for_destruction in this service.
A reviewer captured these plans on Database Lab (gitlab-production-main) with a real slice: the first 200 ids this pass marks for the com/gitlab/eclipse/gitlab-language-server.cocoa.macosx.aarch64 0.3.4-SNAPSHOT package in gitlab-org/editor-extensions/gitlab-eclipse-plugin. That package has 601 installable files, and with keep_n_duplicated_package_files: 1 the pass marks 442 of them in slices of 200, 200 and 42. The links open for GitLab team members only, so the full plans are in the collapsed section below.
After: one statement per slice
UPDATE "packages_package_files"
SET "status" = 1
WHERE "packages_package_files"."id" IN (/* slice of up to 200 ids */);Query plan: Index Scan using packages_package_files_pkey, 200 rows, 968 ms on a cold clone.
Before: three statements per slice
MarkPackageFilesForDestructionService ran each_batch over each slice, which added a start query and a boundary query before the UPDATE:
SELECT "packages_package_files"."id" FROM "packages_package_files"
WHERE "packages_package_files"."id" IN (/* slice */)
ORDER BY "packages_package_files"."id" ASC LIMIT 1;Query plan: Index Only Scan using packages_package_files_pkey, 6 ms.
SELECT "packages_package_files"."id" FROM "packages_package_files"
WHERE "packages_package_files"."id" IN (/* slice */) AND "packages_package_files"."id" >= /* start id */
ORDER BY "packages_package_files"."id" ASC LIMIT 1 OFFSET 200;Query plan: Index Only Scan using packages_package_files_pkey, 92 ms.
UPDATE "packages_package_files" SET "status" = 1
WHERE "packages_package_files"."id" IN (/* slice */) AND "packages_package_files"."id" >= /* start id */;Query plan: the same plan as the new UPDATE, 596 ms.
The UPDATE plan is unchanged, and this MR removes the two lookups. The UPDATE time is mostly the clone's cold I/O for index maintenance: a second run of the new UPDATE on a warmed clone took 760 ms with the same plan, and every UPDATE run touched about the same buffers (12.5k–13.6k hits, 600–800 reads, about 3.5k WAL records).
Plans and the 200 ids
After: UPDATE
Update on public.packages_package_files (cost=0.57..475.85 rows=0 width=0) (actual time=964.152..964.153 rows=0 loops=1)
Buffers: shared hit=12492 read=707 dirtied=580 written=48
WAL: records=3550 fpi=513 bytes=4325159
I/O Timings: read=898.886 write=29.231
-> Index Scan using packages_package_files_pkey on public.packages_package_files (cost=0.57..475.85 rows=200 width=8) (actual time=16.196..398.763 rows=200 loops=1)
Index Cond: (packages_package_files.id = ANY ('{<200 ids below>}'::bigint[]))
Buffers: shared hit=124 read=273 dirtied=1
WAL: records=1 fpi=1 bytes=7873
I/O Timings: read=397.440 write=0.000
Trigger trigger_206cbe2dc1a2: time=1.208 calls=200Before: start, boundary and UPDATE (start id 173157555)
Limit (cost=0.57..2.68 rows=1 width=8) (actual time=3.797..3.798 rows=1 loops=1)
Buffers: shared read=5
I/O Timings: read=3.704 write=0.000
-> Index Only Scan using packages_package_files_pkey on public.packages_package_files (cost=0.57..421.09 rows=200 width=8) (actual time=3.795..3.796 rows=1 loops=1)
Index Cond: (packages_package_files.id = ANY ('{<200 ids below>}'::bigint[]))
Heap Fetches: 0
Buffers: shared read=5
I/O Timings: read=3.704 write=0.000
Limit (cost=420.05..423.64 rows=1 width=8) (actual time=89.276..89.278 rows=0 loops=1)
Buffers: shared hit=138 read=92 dirtied=1
WAL: records=1 fpi=1 bytes=7873
I/O Timings: read=88.440 write=0.000
-> Index Only Scan using packages_package_files_pkey on public.packages_package_files (cost=0.57..420.05 rows=117 width=8) (actual time=0.071..89.254 rows=200 loops=1)
Index Cond: ((packages_package_files.id = ANY ('{<200 ids below>}'::bigint[])) AND (packages_package_files.id >= 173157555))
Heap Fetches: 2
Buffers: shared hit=138 read=92 dirtied=1
WAL: records=1 fpi=1 bytes=7873
I/O Timings: read=88.440 write=0.000
Update on public.packages_package_files (cost=0.57..453.79 rows=0 width=0) (actual time=592.837..592.838 rows=0 loops=1)
Buffers: shared hit=12579 read=609 dirtied=579 written=48
WAL: records=3540 fpi=512 bytes=4316272
I/O Timings: read=511.106 write=50.224
-> Index Scan using packages_package_files_pkey on public.packages_package_files (cost=0.57..453.79 rows=117 width=8) (actual time=1.072..170.765 rows=200 loops=1)
Index Cond: ((packages_package_files.id = ANY ('{<200 ids below>}'::bigint[])) AND (packages_package_files.id >= 173157555))
Buffers: shared hit=207 read=193
I/O Timings: read=169.757 write=0.000
Trigger trigger_206cbe2dc1a2: time=0.866 calls=200The 200 ids, in the order the service builds them
177998337, 177956790, 177937240, 177929648, 177865309, 177762221, 177756394, 177750724, 177442222, 177400826, 177386599, 177151640, 176983612, 176764688, 176702266, 176543921, 176535292, 176534422, 176505923, 176447884, 176244501, 176024013, 176023911, 176023461, 175739303, 175621643, 175448534, 175267570, 175211109, 175016616, 174984418, 174955324, 174737216, 174720856, 174710119, 174448497, 174433977, 174402334, 174039300, 173998649, 173842213, 173611431, 173573495, 173558869, 173395993, 173334969, 173317184, 173199652, 173157555, 177998342, 177956796, 177937248, 177929660, 177865318, 177762231, 177756400, 177750725, 177442232, 177400829, 177386612, 177151643, 176983622, 176764690, 176702282, 176543928, 176535297, 176534425, 176505932, 176447889, 176244503, 176024026, 176023919, 176023464, 175739307, 175621650, 175448539, 175267572, 175211116, 175016619, 174984441, 174955343, 174737219, 174720861, 174710127, 174448513, 174433999, 174402349, 174039306, 173998666, 173842216, 173611440, 173573508, 173558873, 173395998, 173334973, 173317193, 173199653, 173157556, 177998347, 177956803, 177937253, 177929665, 177865321, 177762238, 177756402, 177750727, 177442238, 177400830, 177386619, 177151645, 176983630, 176764691, 176702286, 176543932, 176535299, 176534431, 176505935, 176447891, 176244504, 176024030, 176023925, 176023468, 175739309, 175621656, 175448541, 175267574, 175211122, 175016622, 174984449, 174955351, 174737222, 174720864, 174710129, 174448524, 174434013, 174402362, 174039308, 173998675, 173842218, 173611445, 173573512, 173558878, 173396001, 173334975, 173317196, 173199654, 173157557, 177998353, 177956808, 177937256, 177929672, 177865326, 177762242, 177756404, 177750729, 177442243, 177400833, 177386624, 177151648, 176983635, 176764694, 176702291, 176543936, 176535300, 176534433, 176505944, 176447893, 176244505, 176024032, 176023927, 176023469, 175739313, 175621659, 175448545, 175267575, 175211128, 175016626, 174984460, 174955357, 174737225, 174720868, 174710138, 174448532, 174434033, 174402370, 174039313, 173998681, 173842221, 173611447, 173573518, 173558883, 173396003, 173334978, 173317202, 173199655, 173157560, 177998357, 177956815, 177937272, 177929682MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.