Add service to sync builds to ClickHouse
requested to merge pedropombeiro/421200/2-ingest-finished_build_ch_sync_events-data-in-ch into master
| Sequence | MR |
|---|---|
| 1 | Create p_ci_finished_build_ch_sync_events table (!131902 - merged) |
| 2 |
Add service to sync builds to ClickHouse (!132010 - merged) |
What does this MR do and why?
This MR:
- introduces the
generate_ci_finished_builds_sync_eventsandci_data_ingestion_to_click_houseFFs. - populates the
p_ci_finished_build_ch_sync_eventstable whenever a build finishes (gated bygenerate_ci_finished_builds_sync_events). - introduces a service that will be leveraged in Cron worker to ingest data from ci_builds to Cl... (#421201 - closed) to periodically sync the build information to ClickHouse. The service is gated by the
ci_data_ingestion_to_click_houseFF. As each batch is processed, the respective sync event records are marked as processed. When the current partition has records that are more than a day old and all records in that partition are marked as processed, the partition manager will create a new partition and mark it the default one. The previously active partition will be detached automatically.
EE: true
Part of #421200 (closed)
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Screen capture of service syncing sync event records to ClickHouse across 3 workers (on tab 5, running ::ClickHouse::CiFinishedBuildsSyncWorker.new.perform) as they are being created from ci_builds on another pane (tab 3). The ClickHouse client in tab 1 shows the number of CH records in ci_finished_builds increasing:
Kapture_2023-09-19_at_14.35.21
=> {:status=>:processed, :records_inserted=>139500, :reached_end_of_table=>false}How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
-
Go to the shell in your GDK
gitlabdirectory and runbundle exec rake "gitlab:seed:runner_fleet". This will seed your GDK with some runners and jobs required for testing this MR. -
Enabled the FFs:
Feature.enable(:generate_ci_finished_builds_sync_events) Feature.enable(:ci_data_ingestion_to_click_house) -
Create a new
Ci::FinishedBuildChSyncEventrecord for each finished build on the GDK console:Ci::Build.include EachBatch Ci::Build.finished.where.not(finished_at: nil).order(finished_at: :asc).each_batch(of: 5000) { |batch| batch = batch.map { |build| Ci::FinishedBuildChSyncEvent.new(build_id: build.id, build_finished_at: build.finished_at) }; Ci::FinishedBuildChSyncEvent.transaction { batch.each(&:save) } } -
Invoke the service to import all the finished builds:
::ClickHouse::DataIngestion::CiFinishedBuildsSyncService.new.execute
Database query plans
In order to test this, I created the p_ci_finished_build_ch_sync_events and initial partition. Afterwards, I imported 100K finished builds from ci_builds:
INSERT INTO p_ci_finished_build_ch_sync_events (build_id, build_finished_at)
SELECT id, finished_at
FROM ci_builds
WHERE status IN ('success', 'failed', 'canceled')
AND finished_at IS NOT NULL
LIMIT 100000
ON CONFLICT
DO NOTHING;Get batch in each_batch
https://console.postgres.ai/gitlab/gitlab-production-ci/sessions/22648/commands/73042
SELECT *
FROM (WITH RECURSIVE "array_cte" AS MATERIALIZED (
SELECT "build_id_partition"
FROM generate_series(50, 100) AS "p_ci_finished_build_ch_sync_events" (build_id_partition)), "recursive_keyset_cte" AS ((
SELECT NULL::bigint AS build_id, array_cte_build_id_partition_array, p_ci_finished_build_ch_sync_events_build_id_array,
0::bigint AS count
FROM (
SELECT ARRAY_AGG("array_cte"."build_id_partition") AS array_cte_build_id_partition_array,
ARRAY_AGG("p_ci_finished_build_ch_sync_events"."build_id") AS
p_ci_finished_build_ch_sync_events_build_id_array
FROM (
SELECT "array_cte"."build_id_partition"
FROM array_cte) array_cte
LEFT JOIN LATERAL (
SELECT "p_ci_finished_build_ch_sync_events"."build_id" AS build_id
FROM "p_ci_finished_build_ch_sync_events"
WHERE "p_ci_finished_build_ch_sync_events"."processed" = FALSE
AND (build_id % 100) = "array_cte"."build_id_partition"
AND ("p_ci_finished_build_ch_sync_events"."build_id" > 21341)
ORDER BY "p_ci_finished_build_ch_sync_events"."build_id" ASC
LIMIT 1) p_ci_finished_build_ch_sync_events ON TRUE
WHERE "p_ci_finished_build_ch_sync_events"."build_id" IS NOT NULL) array_scope_lateral_query
LIMIT 1)
UNION ALL (
SELECT recursive_keyset_cte.p_ci_finished_build_ch_sync_events_build_id_array[position],
array_cte_build_id_partition_array,
recursive_keyset_cte.p_ci_finished_build_ch_sync_events_build_id_array[:position_query.position - 1] ||
next_cursor_values.build_id ||
recursive_keyset_cte.p_ci_finished_build_ch_sync_events_build_id_array[position_query.position + 1:],
recursive_keyset_cte.count + 1
FROM recursive_keyset_cte, LATERAL (
SELECT build_id, position
FROM UNNEST(p_ci_finished_build_ch_sync_events_build_id_array)
WITH ORDINALITY AS u (build_id, position)
WHERE build_id IS NOT NULL
ORDER BY 1 ASC
LIMIT 1) AS position_query, LATERAL (
SELECT "record"."build_id"
FROM (
VALUES (NULL)) AS nulls
LEFT JOIN (
SELECT "p_ci_finished_build_ch_sync_events"."build_id" AS build_id
FROM "p_ci_finished_build_ch_sync_events"
WHERE "p_ci_finished_build_ch_sync_events"."processed" = FALSE
AND (build_id % 100) = recursive_keyset_cte.array_cte_build_id_partition_array[position]
AND ("p_ci_finished_build_ch_sync_events"."build_id" >
recursive_keyset_cte.p_ci_finished_build_ch_sync_events_build_id_array[position])
ORDER BY "p_ci_finished_build_ch_sync_events"."build_id" ASC
LIMIT 1) record ON TRUE
LIMIT 1) AS next_cursor_values))
SELECT build_id
FROM "recursive_keyset_cte" AS "p_ci_finished_build_ch_sync_events"
WHERE (count <> 0)) p_ci_finished_build_ch_sync_events
LIMIT 500 Limit (cost=99.63..102.90 rows=100 width=8) (actual time=1.229..23.974 rows=500 loops=1)
Buffers: shared hit=1661
I/O Timings: read=0.000 write=0.000
-> CTE Scan on recursive_keyset_cte p_ci_finished_build_ch_sync_events (cost=99.63..101.90 rows=100 width=8) (actual time=1.228..23.886 rows=500 loops=1)
Filter: (p_ci_finished_build_ch_sync_events.count <> 0)
Rows Removed by Filter: 1
Buffers: shared hit=1661
I/O Timings: read=0.000 write=0.000
CTE array_cte
-> Function Scan on generate_series p_ci_finished_build_ch_sync_events_1 (cost=0.00..0.51 rows=51 width=4) (actual time=0.018..0.025 rows=51 loops=1)
I/O Timings: read=0.000 write=0.000
CTE recursive_keyset_cte
-> Recursive Union (cost=25.17..99.12 rows=101 width=80) (actual time=1.070..22.808 rows=501 loops=1)
Buffers: shared hit=1661
I/O Timings: read=0.000 write=0.000
-> Limit (cost=25.17..25.19 rows=1 width=80) (actual time=1.068..1.074 rows=1 loops=1)
Buffers: shared hit=157
I/O Timings: read=0.000 write=0.000
-> Subquery Scan on array_scope_lateral_query (cost=25.17..25.19 rows=1 width=80) (actual time=1.067..1.073 rows=1 loops=1)
Buffers: shared hit=157
I/O Timings: read=0.000 write=0.000
-> Aggregate (cost=25.17..25.18 rows=1 width=64) (actual time=1.066..1.070 rows=1 loops=1)
Buffers: shared hit=157
I/O Timings: read=0.000 write=0.000
-> Nested Loop (cost=0.42..24.91 rows=51 width=12) (actual time=0.159..1.034 rows=50 loops=1)
Buffers: shared hit=157
I/O Timings: read=0.000 write=0.000
-> CTE Scan on array_cte (cost=0.00..1.02 rows=51 width=4) (actual time=0.019..0.041 rows=51 loops=1)
I/O Timings: read=0.000 write=0.000
-> Subquery Scan on p_ci_finished_build_ch_sync_events_2 (cost=0.42..0.46 rows=1 width=8) (actual time=0.019..0.019 rows=1 loops=51)
Filter: (p_ci_finished_build_ch_sync_events_2.build_id IS NOT NULL)
Rows Removed by Filter: 0
Buffers: shared hit=157
I/O Timings: read=0.000 write=0.000
-> Limit (cost=0.42..0.45 rows=1 width=8) (actual time=0.018..0.018 rows=1 loops=51)
Buffers: shared hit=157
I/O Timings: read=0.000 write=0.000
-> Index Only Scan using p_ci_finished_build_ch_sync_events_1_expr_build_id_idx on gitlab_partitions_dynamic.p_ci_finished_build_ch_sync_events_1 p_ci_finished_build_ch_sync_events_3 (cost=0.42..25.78 rows=893 width=8) (actual time=0.018..0.018 rows=1 loops=51)
Index Cond: ((((p_ci_finished_build_ch_sync_events_3.build_id % '100'::bigint)) = array_cte.build_id_partition) AND (p_ci_finished_build_ch_sync_events_3.build_id > 21341))
Heap Fetches: 0
Buffers: shared hit=157
I/O Timings: read=0.000 write=0.000
-> Nested Loop (cost=0.58..7.19 rows=10 width=80) (actual time=0.042..0.043 rows=1 loops=500)
Buffers: shared hit=1504
I/O Timings: read=0.000 write=0.000
-> Nested Loop (cost=0.15..1.95 rows=10 width=80) (actual time=0.027..0.028 rows=1 loops=500)
Buffers: shared hit=3
I/O Timings: read=0.000 write=0.000
-> WorkTable Scan on recursive_keyset_cte (cost=0.00..0.20 rows=10 width=72) (actual time=0.000..0.000 rows=1 loops=500)
I/O Timings: read=0.000 write=0.000
-> Limit (cost=0.15..0.16 rows=1 width=16) (actual time=0.027..0.027 rows=1 loops=500)
Buffers: shared hit=3
I/O Timings: read=0.000 write=0.000
-> Sort (cost=0.15..0.18 rows=10 width=16) (actual time=0.026..0.026 rows=1 loops=500)
Sort Key: u.build_id
Sort Method: top-N heapsort Memory: 25kB
Buffers: shared hit=3
I/O Timings: read=0.000 write=0.000
-> Function Scan on unnest u (cost=0.00..0.10 rows=10 width=16) (actual time=0.007..0.014 rows=50 loops=500)
Filter: (u.build_id IS NOT NULL)
Rows Removed by Filter: 0
I/O Timings: read=0.000 write=0.000
-> Limit (cost=0.42..0.48 rows=1 width=8) (actual time=0.011..0.011 rows=1 loops=500)
Buffers: shared hit=1501
I/O Timings: read=0.000 write=0.000
-> Nested Loop Left Join (cost=0.42..0.48 rows=1 width=8) (actual time=0.010..0.010 rows=1 loops=500)
Buffers: shared hit=1501
I/O Timings: read=0.000 write=0.000
-> Result (cost=0.00..0.01 rows=1 width=0) (actual time=0.000..0.000 rows=1 loops=500)
I/O Timings: read=0.000 write=0.000
-> Limit (cost=0.42..0.45 rows=1 width=8) (actual time=0.010..0.010 rows=1 loops=500)
Buffers: shared hit=1501
I/O Timings: read=0.000 write=0.000
-> Index Only Scan using p_ci_finished_build_ch_sync_events_1_expr_build_id_idx on gitlab_partitions_dynamic.p_ci_finished_build_ch_sync_events_1 p_ci_finished_build_ch_sync_events_4 (cost=0.42..10.03 rows=330 width=8) (actual time=0.009..0.009 rows=1 loops=500)
Index Cond: ((((p_ci_finished_build_ch_sync_events_4.build_id % '100'::bigint)) = (recursive_keyset_cte.array_cte_build_id_partition_array)[u."position"]) AND (p_ci_finished_build_ch_sync_events_4.build_id > (recursive_keyset_cte.p_ci_finished_build_ch_sync_events_build_id_array)[u."position"]))
Heap Fetches: 0
Buffers: shared hit=1501
I/O Timings: read=0.000 write=0.000Fetch joined data for each batch (lambda in `process_batch`)
https://console.postgres.ai/gitlab/gitlab-production-ci/sessions/22648/commands/73053
SELECT "ci_builds"."finished_at", "ci_builds"."id", "ci_builds"."project_id", "ci_builds"."commit_id",
"ci_builds"."status", "ci_builds"."runner_id", EXTRACT(epoch FROM ci_builds.created_at) AS casted_created_at,
EXTRACT(epoch FROM ci_builds.queued_at) AS casted_queued_at, EXTRACT(epoch FROM ci_builds.started_at) AS
casted_started_at, EXTRACT(epoch FROM ci_builds.finished_at) AS casted_finished_at, ci_runners.run_untagged AS
runner_run_untagged, ci_runners.runner_type AS runner_type, ci_runner_machines.system_xid AS
runner_manager_system_xid,
ci_runner_machines.version AS runner_manager_version, ci_runner_machines.revision AS runner_manager_revision,
ci_runner_machines.platform AS runner_manager_platform, ci_runner_machines.architecture AS
runner_manager_architecture
FROM "ci_builds"
LEFT OUTER JOIN "ci_runners" ON "ci_runners"."id" = "ci_builds"."runner_id"
LEFT OUTER JOIN "p_ci_runner_machine_builds" ON "p_ci_runner_machine_builds"."build_id" = "ci_builds"."id"
LEFT OUTER JOIN "ci_runner_machines" ON "ci_runner_machines"."id" = "p_ci_runner_machine_builds"."runner_machine_id"
WHERE "ci_builds"."type" = 'Ci::Build'
AND "ci_builds"."id" IN (2568800, 266801, 267701, 324101, 352101, 373101, 377701, 3462401, 3940901, 341502, 483902,
561502, 2423702, 377403, 386903, 264104, 407204, 608804, 3041304, 6087804, 309205, 342305, 392205, 549905, 266506,
294406, 407206, 545306, 615006, 3725506, 5933706, 6166206, 6329306, 375307, 2174407, 2239707, 3940907, 6171207,
6419207, 569108, 3289808, 4396808, 6099208, 6134508, 7119108, 637109, 3726909, 6165109, 6166009, 6303109, 114910,
279910, 2455110, 5669210, 6309910, 539011, 2249912, 2423712, 6303912, 269013, 365213, 546713, 334414, 367214,
476014, 555214, 4396814, 5669214, 6311414, 6840214, 7079414, 319215, 348315, 3377015, 6668515, 6982315, 339116,
575316, 6519616, 6840216, 52617, 382017, 435417, 539217, 348018, 476318, 5854418, 608119, 1888819, 3500319,
6231819, 366120, 377120, 542020, 811020, 6311620, 113021, 4396921, 6210121, 6590021, 346822, 511622, 539522,
5683822, 6165422, 334323, 348823, 372423, 378223, 391023, 480623, 495223, 571423, 2588523, 5622323, 6417823,
341224, 363824, 378924, 476124, 484424, 585324, 558725, 598225, 2423725, 541826, 5916826, 6185626, 266927, 342027,
348127, 355227, 388727, 489727, 538627, 550827, 612227, 3476327, 6165427, 319128, 506828, 573028, 3725328, 6419228,
459029, 477129, 542229, 595629, 2423729, 6231429, 7036029, 7200829, 308530, 6165130, 389231, 543231, 1089931,
5700531, 5736431, 6236731, 6946131, 264032, 351632, 385532, 458732, 659032, 2423332, 2568932, 351133, 363033,
1215133, 3710833, 4396933, 5575733, 318634, 360134, 425834, 2424334, 4396834, 6215334, 322835, 350235, 353535,
543635, 4396835, 6171535, 484436, 524236, 571336, 4396836, 5683836, 6168036, 477237, 572737, 769537, 3547137,
4396837, 436138, 208439, 330039, 264240, 363640, 516440, 2423340, 6224340, 21341, 351141, 541741, 611641, 2483741,
21342, 349342, 1115642, 2412742, 3731542, 6168542, 341943, 439243, 540443, 603543, 3738243, 3808443, 4143243,
6419443, 263844, 264544, 382244, 491244, 6165244, 374245, 447945, 476145, 546445, 575245, 596145, 2568845, 277946,
378846, 4396946, 6366746, 318247, 338547, 420847, 569447, 4396947, 377148, 402348, 563148, 2408148, 6168048,
425449, 442749, 541849, 4396849, 5566249, 7232249, 354650, 369850, 403650, 630350, 2445350, 6166350, 305051,
2494151, 300252, 347052, 347652, 368052, 593052, 611452, 2610952, 6165452, 7157852, 377353, 6138753, 6168953,
279754, 343654, 614654, 2588554, 3506754, 5877154, 6166354, 341355, 363755, 6167155, 6307755, 376356, 465456,
540556, 2423356, 6312756, 2350457, 5921057, 6165457, 6432757, 429558, 475958, 540058, 569458, 660958, 5724458,
6099258, 41159, 304259, 342359, 365859, 506059, 538559, 3595859, 6165559, 6500559, 363760, 550760, 1043460,
3730960, 317361, 345061, 585661, 6316361, 302062, 385762, 484162, 494362, 5708162, 208963, 310763, 349963, 356763,
461463, 563863, 6602363, 6607763, 264064, 344064, 373564, 539664, 555764, 3738164, 6166364, 410265, 544265, 607665,
5610265, 366166, 377966, 541366, 2064866, 2568866, 3725566, 7059266, 513967, 768567, 3289667, 263968, 343568,
564668, 548669, 582869, 603569, 1244369, 3493269, 6165569, 6166569, 347070, 372970, 387470, 3886270, 6165570,
6448870, 451871, 517571, 542671, 607771, 5701471, 362672, 543872, 565172, 3800772, 6311472, 6315572, 382573,
414173, 465673, 6165473, 6337573, 6963673, 372674, 475074, 6172274, 282275, 349975, 475475, 552475, 2664675,
2829575, 4246575, 6998475, 242776, 265176, 382076, 386776, 2392376, 3508276, 3739276, 5729676, 6676276, 52477,
208677, 354177, 362077, 372577, 475777, 592177, 2568377, 3730977, 265078, 313178, 486678, 608478, 3930878, 7060778,
337579, 341579, 373079, 4396979, 6166479, 303080, 319080, 3099980, 6210280, 6519980, 6918480, 452381, 2580381,
3490981, 6165381, 5815182, 307883, 593483, 355284, 538784, 2568784, 2588584, 6165584, 530985, 3433685, 5534885,
2568886, 6311786, 346187, 369287, 377687, 3977387, 538388, 553988, 564788, 360189, 388489, 543389, 5720389,
6311589, 363390, 3739290, 6311790, 293891, 342691, 426391, 3738491, 6165991, 6305091, 275592, 365392, 509692,
5586992, 6168892, 2412493, 113094, 536394, 563694, 811794, 3670094, 6419194, 264095, 355095, 360195, 565195,
6180995, 318496, 359696, 477496, 3391996, 3940896, 6167896, 370997, 5722497, 7119097, 447898, 6341698, 539399) Nested Loop Left Join (cost=2.02..6426.10 rows=470 width=204) (actual time=66.232..2918.483 rows=405 loops=1)
Buffers: shared hit=3543 read=1081 dirtied=37
I/O Timings: read=2811.130 write=0.000
-> Nested Loop Left Join (cost=1.59..4984.25 rows=470 width=70) (actual time=52.922..2889.032 rows=405 loops=1)
Buffers: shared hit=3543 read=1081 dirtied=37
I/O Timings: read=2811.130 write=0.000
-> Nested Loop Left Join (cost=1.02..3284.02 rows=470 width=62) (actual time=40.677..2857.520 rows=405 loops=1)
Buffers: shared hit=1927 read=1077 dirtied=37
I/O Timings: read=2799.008 write=0.000
-> Index Scan using ci_builds_pkey on public.ci_builds (cost=0.58..1667.20 rows=470 width=59) (actual time=35.870..2377.046 rows=405 loops=1)
Index Cond: (ci_builds.id = ANY ('{2568800,266801,267701,324101,352101,373101,377701,3462401,3940901,341502,483902,561502,2423702,377403,386903,264104,407204,608804,3041304,6087804,309205,342305,392205,549905,266506,294406,407206,545306,615006,3725506,5933706,6166206,6329306,375307,2174407,2239707,3940907,6171207,6419207,569108,3289808,4396808,6099208,6134508,7119108,637109,3726909,6165109,6166009,6303109,114910,279910,2455110,5669210,6309910,539011,2249912,2423712,6303912,269013,365213,546713,334414,367214,476014,555214,4396814,5669214,6311414,6840214,7079414,319215,348315,3377015,6668515,6982315,339116,575316,6519616,6840216,52617,382017,435417,539217,348018,476318,5854418,608119,1888819,3500319,6231819,366120,377120,542020,811020,6311620,113021,4396921,6210121,6590021,346822,511622,539522,5683822,6165422,334323,348823,372423,378223,391023,480623,495223,571423,2588523,5622323,6417823,341224,363824,378924,476124,484424,585324,558725,598225,2423725,541826,5916826,6185626,266927,342027,348127,355227,388727,489727,538627,550827,612227,3476327,6165427,319128,506828,573028,3725328,6419228,459029,477129,542229,595629,2423729,6231429,7036029,7200829,308530,6165130,389231,543231,1089931,5700531,5736431,6236731,6946131,264032,351632,385532,458732,659032,2423332,2568932,351133,363033,1215133,3710833,4396933,5575733,318634,360134,425834,2424334,4396834,6215334,322835,350235,353535,543635,4396835,6171535,484436,524236,571336,4396836,5683836,6168036,477237,572737,769537,3547137,4396837,436138,208439,330039,264240,363640,516440,2423340,6224340,21341,351141,541741,611641,2483741,21342,349342,1115642,2412742,3731542,6168542,341943,439243,540443,603543,3738243,3808443,4143243,6419443,263844,264544,382244,491244,6165244,374245,447945,476145,546445,575245,596145,2568845,277946,378846,4396946,6366746,318247,338547,420847,569447,4396947,377148,402348,563148,2408148,6168048,425449,442749,541849,4396849,5566249,7232249,354650,369850,403650,630350,2445350,6166350,305051,2494151,300252,347052,347652,368052,593052,611452,2610952,6165452,7157852,377353,6138753,6168953,279754,343654,614654,2588554,3506754,5877154,6166354,341355,363755,6167155,6307755,376356,465456,540556,2423356,6312756,2350457,5921057,6165457,6432757,429558,475958,540058,569458,660958,5724458,6099258,41159,304259,342359,365859,506059,538559,3595859,6165559,6500559,363760,550760,1043460,3730960,317361,345061,585661,6316361,302062,385762,484162,494362,5708162,208963,310763,349963,356763,461463,563863,6602363,6607763,264064,344064,373564,539664,555764,3738164,6166364,410265,544265,607665,5610265,366166,377966,541366,2064866,2568866,3725566,7059266,513967,768567,3289667,263968,343568,564668,548669,582869,603569,1244369,3493269,6165569,6166569,347070,372970,387470,3886270,6165570,6448870,451871,517571,542671,607771,5701471,362672,543872,565172,3800772,6311472,6315572,382573,414173,465673,6165473,6337573,6963673,372674,475074,6172274,282275,349975,475475,552475,2664675,2829575,4246575,6998475,242776,265176,382076,386776,2392376,3508276,3739276,5729676,6676276,52477,208677,354177,362077,372577,475777,592177,2568377,3730977,265078,313178,486678,608478,3930878,7060778,337579,341579,373079,4396979,6166479,303080,319080,3099980,6210280,6519980,6918480,452381,2580381,3490981,6165381,5815182,307883,593483,355284,538784,2568784,2588584,6165584,530985,3433685,5534885,2568886,6311786,346187,369287,377687,3977387,538388,553988,564788,360189,388489,543389,5720389,6311589,363390,3739290,6311790,293891,342691,426391,3738491,6165991,6305091,275592,365392,509692,5586992,6168892,2412493,113094,536394,563694,811794,3670094,6419194,264095,355095,360195,565195,6180995,318496,359696,477496,3391996,3940896,6167896,370997,5722497,7119097,447898,6341698,539399}'::bigint[]))
Filter: ((ci_builds.type)::text = 'Ci::Build'::text)
Rows Removed by Filter: 95
Buffers: shared hit=1578 read=921
I/O Timings: read=2337.969 write=0.000
-> Memoize (cost=0.44..3.45 rows=1 width=7) (actual time=1.181..1.181 rows=0 loops=405)
Buffers: shared hit=349 read=156 dirtied=37
I/O Timings: read=461.039 write=0.000
-> Index Scan using ci_runners_pkey on public.ci_runners (cost=0.43..3.44 rows=1 width=7) (actual time=3.751..3.751 rows=1 loops=127)
Index Cond: (ci_runners.id = ci_builds.runner_id)
Buffers: shared hit=349 read=156 dirtied=37
I/O Timings: read=461.039 write=0.000
-> Append (cost=0.57..3.60 rows=2 width=16) (actual time=0.074..0.074 rows=0 loops=405)
Buffers: shared hit=1616 read=4
I/O Timings: read=12.122 write=0.000
-> Index Scan using ci_runner_machine_builds_100_pkey on gitlab_partitions_dynamic.ci_runner_machine_builds_100 p_ci_runner_machine_builds_1 (cost=0.57..3.59 rows=1 width=16) (actual time=0.065..0.065 rows=0 loops=405)
Index Cond: (p_ci_runner_machine_builds_1.build_id = ci_builds.id)
Buffers: shared hit=1616 read=4
I/O Timings: read=12.122 write=0.000
-> Seq Scan on gitlab_partitions_dynamic.ci_runner_machine_builds_101 p_ci_runner_machine_builds_2 (cost=0.00..0.00 rows=1 width=16) (actual time=0.001..0.001 rows=0 loops=405)
Filter: (p_ci_runner_machine_builds_2.build_id = ci_builds.id)
Rows Removed by Filter: 0
I/O Timings: read=0.000 write=0.000
-> Index Scan using ci_runner_machines_pkey on public.ci_runner_machines (cost=0.42..3.06 rows=1 width=46) (actual time=0.001..0.001 rows=0 loops=405)
Index Cond: (ci_runner_machines.id = p_ci_runner_machine_builds.runner_machine_id)
I/O Timings: read=0.000 write=0.000Mark sync events as processed
https://console.postgres.ai/gitlab/gitlab-production-ci/sessions/22648/commands/73059
UPDATE
"p_ci_finished_build_ch_sync_events"
SET "processed" = TRUE
WHERE "p_ci_finished_build_ch_sync_events"."build_id" IN (2568800, 266801, 267701, 324101, 352101, 373101, 377701, 3462401, 3940901, 341502, 483902,
561502, 2423702, 377403, 386903, 264104, 407204, 608804, 3041304, 6087804, 309205, 342305, 392205, 549905, 266506,
294406, 407206, 545306, 615006, 3725506, 5933706, 6166206, 6329306, 375307, 2174407, 2239707, 3940907, 6171207,
6419207, 569108, 3289808, 4396808, 6099208, 6134508, 7119108, 637109, 3726909, 6165109, 6166009, 6303109, 114910,
279910, 2455110, 5669210, 6309910, 539011, 2249912, 2423712, 6303912, 269013, 365213, 546713, 334414, 367214,
476014, 555214, 4396814, 5669214, 6311414, 6840214, 7079414, 319215, 348315, 3377015, 6668515, 6982315, 339116,
575316, 6519616, 6840216, 52617, 382017, 435417, 539217, 348018, 476318, 5854418, 608119, 1888819, 3500319,
6231819, 366120, 377120, 542020, 811020, 6311620, 113021, 4396921, 6210121, 6590021, 346822, 511622, 539522,
5683822, 6165422, 334323, 348823, 372423, 378223, 391023, 480623, 495223, 571423, 2588523, 5622323, 6417823,
341224, 363824, 378924, 476124, 484424, 585324, 558725, 598225, 2423725, 541826, 5916826, 6185626, 266927, 342027,
348127, 355227, 388727, 489727, 538627, 550827, 612227, 3476327, 6165427, 319128, 506828, 573028, 3725328, 6419228,
459029, 477129, 542229, 595629, 2423729, 6231429, 7036029, 7200829, 308530, 6165130, 389231, 543231, 1089931,
5700531, 5736431, 6236731, 6946131, 264032, 351632, 385532, 458732, 659032, 2423332, 2568932, 351133, 363033,
1215133, 3710833, 4396933, 5575733, 318634, 360134, 425834, 2424334, 4396834, 6215334, 322835, 350235, 353535,
543635, 4396835, 6171535, 484436, 524236, 571336, 4396836, 5683836, 6168036, 477237, 572737, 769537, 3547137,
4396837, 436138, 208439, 330039, 264240, 363640, 516440, 2423340, 6224340, 21341, 351141, 541741, 611641, 2483741,
21342, 349342, 1115642, 2412742, 3731542, 6168542, 341943, 439243, 540443, 603543, 3738243, 3808443, 4143243,
6419443, 263844, 264544, 382244, 491244, 6165244, 374245, 447945, 476145, 546445, 575245, 596145, 2568845, 277946,
378846, 4396946, 6366746, 318247, 338547, 420847, 569447, 4396947, 377148, 402348, 563148, 2408148, 6168048,
425449, 442749, 541849, 4396849, 5566249, 7232249, 354650, 369850, 403650, 630350, 2445350, 6166350, 305051,
2494151, 300252, 347052, 347652, 368052, 593052, 611452, 2610952, 6165452, 7157852, 377353, 6138753, 6168953,
279754, 343654, 614654, 2588554, 3506754, 5877154, 6166354, 341355, 363755, 6167155, 6307755, 376356, 465456,
540556, 2423356, 6312756, 2350457, 5921057, 6165457, 6432757, 429558, 475958, 540058, 569458, 660958, 5724458,
6099258, 41159, 304259, 342359, 365859, 506059, 538559, 3595859, 6165559, 6500559, 363760, 550760, 1043460,
3730960, 317361, 345061, 585661, 6316361, 302062, 385762, 484162, 494362, 5708162, 208963, 310763, 349963, 356763,
461463, 563863, 6602363, 6607763, 264064, 344064, 373564, 539664, 555764, 3738164, 6166364, 410265, 544265, 607665,
5610265, 366166, 377966, 541366, 2064866, 2568866, 3725566, 7059266, 513967, 768567, 3289667, 263968, 343568,
564668, 548669, 582869, 603569, 1244369, 3493269, 6165569, 6166569, 347070, 372970, 387470, 3886270, 6165570,
6448870, 451871, 517571, 542671, 607771, 5701471, 362672, 543872, 565172, 3800772, 6311472, 6315572, 382573,
414173, 465673, 6165473, 6337573, 6963673, 372674, 475074, 6172274, 282275, 349975, 475475, 552475, 2664675,
2829575, 4246575, 6998475, 242776, 265176, 382076, 386776, 2392376, 3508276, 3739276, 5729676, 6676276, 52477,
208677, 354177, 362077, 372577, 475777, 592177, 2568377, 3730977, 265078, 313178, 486678, 608478, 3930878, 7060778,
337579, 341579, 373079, 4396979, 6166479, 303080, 319080, 3099980, 6210280, 6519980, 6918480, 452381, 2580381,
3490981, 6165381, 5815182, 307883, 593483, 355284, 538784, 2568784, 2588584, 6165584, 530985, 3433685, 5534885,
2568886, 6311786, 346187, 369287, 377687, 3977387, 538388, 553988, 564788, 360189, 388489, 543389, 5720389,
6311589, 363390, 3739290, 6311790, 293891, 342691, 426391, 3738491, 6165991, 6305091, 275592, 365392, 509692,
5586992, 6168892, 2412493, 113094, 536394, 563694, 811794, 3670094, 6419194, 264095, 355095, 360195, 565195,
6180995, 318496, 359696, 477496, 3391996, 3940896, 6167896, 370997, 5722497, 7119097, 447898, 6341698, 539399) ModifyTable on public.p_ci_finished_build_ch_sync_events (cost=0.42..1183.55 rows=0 width=0) (actual time=11.601..11.602 rows=0 loops=1)
Buffers: shared hit=5893 dirtied=367 written=6
I/O Timings: read=0.000 write=0.000
-> Index Scan using p_ci_finished_build_ch_sync_events_1_pkey on gitlab_partitions_dynamic.p_ci_finished_build_ch_sync_events_1 (cost=0.42..1183.55 rows=500 width=11) (actual time=0.138..2.812 rows=500 loops=1)
Index Cond: (p_ci_finished_build_ch_sync_events_1.build_id = ANY ('{2568800,266801,267701,324101,352101,373101,377701,3462401,3940901,341502,483902,561502,2423702,377403,386903,264104,407204,608804,3041304,6087804,309205,342305,392205,549905,266506,294406,407206,545306,615006,3725506,5933706,6166206,6329306,375307,2174407,2239707,3940907,6171207,6419207,569108,3289808,4396808,6099208,6134508,7119108,637109,3726909,6165109,6166009,6303109,114910,279910,2455110,5669210,6309910,539011,2249912,2423712,6303912,269013,365213,546713,334414,367214,476014,555214,4396814,5669214,6311414,6840214,7079414,319215,348315,3377015,6668515,6982315,339116,575316,6519616,6840216,52617,382017,435417,539217,348018,476318,5854418,608119,1888819,3500319,6231819,366120,377120,542020,811020,6311620,113021,4396921,6210121,6590021,346822,511622,539522,5683822,6165422,334323,348823,372423,378223,391023,480623,495223,571423,2588523,5622323,6417823,341224,363824,378924,476124,484424,585324,558725,598225,2423725,541826,5916826,6185626,266927,342027,348127,355227,388727,489727,538627,550827,612227,3476327,6165427,319128,506828,573028,3725328,6419228,459029,477129,542229,595629,2423729,6231429,7036029,7200829,308530,6165130,389231,543231,1089931,5700531,5736431,6236731,6946131,264032,351632,385532,458732,659032,2423332,2568932,351133,363033,1215133,3710833,4396933,5575733,318634,360134,425834,2424334,4396834,6215334,322835,350235,353535,543635,4396835,6171535,484436,524236,571336,4396836,5683836,6168036,477237,572737,769537,3547137,4396837,436138,208439,330039,264240,363640,516440,2423340,6224340,21341,351141,541741,611641,2483741,21342,349342,1115642,2412742,3731542,6168542,341943,439243,540443,603543,3738243,3808443,4143243,6419443,263844,264544,382244,491244,6165244,374245,447945,476145,546445,575245,596145,2568845,277946,378846,4396946,6366746,318247,338547,420847,569447,4396947,377148,402348,563148,2408148,6168048,425449,442749,541849,4396849,5566249,7232249,354650,369850,403650,630350,2445350,6166350,305051,2494151,300252,347052,347652,368052,593052,611452,2610952,6165452,7157852,377353,6138753,6168953,279754,343654,614654,2588554,3506754,5877154,6166354,341355,363755,6167155,6307755,376356,465456,540556,2423356,6312756,2350457,5921057,6165457,6432757,429558,475958,540058,569458,660958,5724458,6099258,41159,304259,342359,365859,506059,538559,3595859,6165559,6500559,363760,550760,1043460,3730960,317361,345061,585661,6316361,302062,385762,484162,494362,5708162,208963,310763,349963,356763,461463,563863,6602363,6607763,264064,344064,373564,539664,555764,3738164,6166364,410265,544265,607665,5610265,366166,377966,541366,2064866,2568866,3725566,7059266,513967,768567,3289667,263968,343568,564668,548669,582869,603569,1244369,3493269,6165569,6166569,347070,372970,387470,3886270,6165570,6448870,451871,517571,542671,607771,5701471,362672,543872,565172,3800772,6311472,6315572,382573,414173,465673,6165473,6337573,6963673,372674,475074,6172274,282275,349975,475475,552475,2664675,2829575,4246575,6998475,242776,265176,382076,386776,2392376,3508276,3739276,5729676,6676276,52477,208677,354177,362077,372577,475777,592177,2568377,3730977,265078,313178,486678,608478,3930878,7060778,337579,341579,373079,4396979,6166479,303080,319080,3099980,6210280,6519980,6918480,452381,2580381,3490981,6165381,5815182,307883,593483,355284,538784,2568784,2588584,6165584,530985,3433685,5534885,2568886,6311786,346187,369287,377687,3977387,538388,553988,564788,360189,388489,543389,5720389,6311589,363390,3739290,6311790,293891,342691,426391,3738491,6165991,6305091,275592,365392,509692,5586992,6168892,2412493,113094,536394,563694,811794,3670094,6419194,264095,355095,360195,565195,6180995,318496,359696,477496,3391996,3940896,6167896,370997,5722497,7119097,447898,6341698,539399}'::bigint[]))
Buffers: shared hit=1986
I/O Timings: read=0.000 write=0.000MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.