Export commit notes via repository walk to avoid timeouts

Rodrigo Tomonarirequested to merge
rodrigo/commit-notes-export into master

What does this MR do and why?

Exporting the commit_notes relation paginated the notes table with WHERE noteable_type='Commit' ORDER BY id, times out on large projects as there is no index supporting the keyset scan and we cannot add one to the notes table.

Instead, this change updated the export process to walk the project's commits and resolve notes by commit_id:

  • CommitNotesBatcher walks every commit reachable from the repository's refs (Gitaly ListCommits --all) with a bounded retry on transient Gitaly errors.
  • Notes are resolved with commit_id IN (...), which uses index_notes_on_commit_id, and filtered to the project's commit notes.

Both export paths use it, behind the commit_notes_export_via_repo flag:

  • Import/Export: CommitNotesSaver replaces the default RelationSaver.
  • Direct Transfer: CommitNotesExportService, with the batched path grouping resolved note IDs into batches.

References

#554953

Screenshots or screen recordings

Before After

Database review

New queries introduced by this MR. Query plans were captured on gitlab-org/gitlab (project 278964).

Query 1 — Commit-notes existence guard (CommitNotesSaver#commit_notes_present?, CommitNotesExportService#commit_notes_present?). Runs once per export to skip the repository walk when a project has no commit notes.

Plan: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/52649/commands/154321

Plan details 
SELECT 1 AS one FROM notes
WHERE notes.project_id = 278964 AND notes.noteable_type = 'Commit' LIMIT 1
 Limit  (cost=0.58..0.61 rows=1 width=4) (actual time=2.190..2.191 rows=1 loops=1)
   Buffers: shared read=5
   ->  Index Only Scan using index_notes_on_project_id_and_noteable_type on public.notes  (cost=0.58..26330.33 rows=819395 width=4) (actual time=2.188..2.188 rows=1 loops=1)
         Index Cond: ((notes.project_id = 278964) AND (notes.noteable_type = 'Commit'::text))
         Heap Fetches: 0
         Buffers: shared read=5
Settings: work_mem = '230MB', seq_page_cost = '4', effective_cache_size = '472585MB', jit = 'off', random_page_cost = '1.5'
Time: 25.433 ms
  - planning: 23.183 ms
  - execution: 2.250 ms
    - I/O read: N/A
    - I/O write: N/A

Shared buffers:
  - hits: 0 from the buffer pool
  - reads: 5 (~40.00 KiB) from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

Query 2 — Resolve note IDs from a SHA batch (Note.commit_note_ids_for_shas; the file-based saver and the non-batched export path select only id). Runs once per SHA page during the walk. noteable_type is in Ruby.

Plan: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/52698/commands/154419

SELECT notes.id, notes.noteable_type FROM notes
WHERE notes.commit_id IN ('<sha1>', '<sha2>', ...); -- up to the commit-notes batch size (CommitNotesBatcher::DEFAULT_BATCH_SIZE, 500) SHAs
Plan details 
 Index Scan using index_notes_on_commit_id on public.notes  (cost=0.58..1185722.83 rows=784398 width=23) (actual time=0.729..11.209 rows=602 loops=1)
   Index Cond: ((notes.commit_id)::text = ANY ('{3cb217ea6a0862ddec319fd4ae6a4b36b511b411,4b71aaec847aaa0f9cd2417a22812430cd65d373,4094b149a4530cdd217a5a8744697a1f9ade0dd4,ecbebe298a3c08234c2f69e9faa4966edf98faa8,46748686af0d528dc60daf98284bf059dce846da,255a06ca6c2c93f600294c8a62f40dbefeaa9239,417a61e80317ddd193f8e4c5721c349cce139489,1018e390b2649d8055f9ae71d4f6a1746d8e5291,126603720798ba6b96c5b495209fe2a6f814fd2d,c4a746098f80306bcc1a7ed655975254df62d90e,bf590358c097e229d5b13cadfbf310360543a44a,a07b634e9120cf7b8a130e693d66767709e579d3,98547c1296a69893ccc23f553beedbd57076980d,bf0451f9de252e10dc25f4dad880733de9a59400,7f658b21e2004b0547bb7ddc28ac1f0044bf838e,4a4c18b194bf7b890222d0551d72460fd795eadb,3464314ee3f722823c887a540362cf05fc57c45b,e2952d84df0d5eac96e4095d001e7a5e605dda0e,4ad48a98f5a248afc0aa9e7f738ffe53f628281c,7f642ddd43b8475a39d2f635d1a1aeb883d31899,cc819a618946af52be9b083d516eb789f3d71d94,022429d161a4c1719efe3211f9c66dcfa6b40b6a,dc29ff2ac8095a3b0316306a5d910458d218d952,249539215aaf4384ce24eaa4a4828ed3f7d77e3d,5b89209910e86ed2d9764857d1458b75cd2c2308,4cceab70f1c9f1fb303a17f532be730d76f07d78,7dd45b89a0ce9445529e9787ae27100754837cf2,782d79cd4d15a91eb8366f2582ba17c3c819ff43,25e5b6feeaea3e4b581d6ecdc73981d211f43738,9738bb3120b84dce4d791b7dc706cff63735d04d,051f84b56ecb7a81e80c037c972bf7347a3ac3ba,45fc066ec21ee8e5bb9382d0fd7796978b40ee16,580a2cd5d0bd444a039afbe80f628ee9ae26d701,c4755c38a7a7c3f93ea10af34fed16eb66779b5f,f24826e05e7bf6d44f7b05adb383bef76f452c73,f56cf4f94ecdbe816f2369c1b774fc47a2581d7d,dc50d8403383ab0b607c52704fb3ae3c06549d6d,a7a282de9c85d01632fd46d266119fca876d2cff,14d82ec719b3e23dd55c37462e90c9a4721db8ff,2722493f4f5d46daf0e4bb7f95dac2da63224dba,62e15f9dd2229330cc227d88fd6bf85df0434985,c71db0606e2ee0b35ed21849fcd79fd4b81adc2e,03ab39aa865141eced64d600b9049847e7f6e9aa,195a329af692e7decacbd4f539a7523f1634c6e4,0f70af7c82b6afc5bf3bdb85d224e9c4958e8d5e,7acd9ca489082292f0f5b3455c45451c40790988,e967e0a802c74ccc0c3610829408bf2b33f2dbdf,299c7c375d26b47c25463820a9ef7ec770219796,9e0b5278b8a227a0844046d7ebb02bfe3cea484a,ff88ac7c00ea8ad2262b0ded52cc7836d08ac325,ea032ad8a0ce4cf2b5d6a6887e6e923cbd09139f,7864f3106f8c2afad3a1327e98b1acdcbe782e30,346804be34e39a5326dc6a835394e2715507dc6d,e6759fc4746fa65eb919e136a8e72d274b69f1e0,5dca1031a19f185da112a0f017fe440897cfb45f,6f417566609f9596b85484b293b8b9407c5f24fc,64cb8d6c8d9f2cf7cd77d4c44887acfe8c8b1ad9,96ea309a948fcc0ddacbc74d142d39d69a4107ba,3da0c87507a61c0ffa7f63010da1449beb0f7e8b,988f90aba28b17dbef9d3c4776d56d8bfb397558,5ab2cb4cee12054a7818d9847212c395d08f1b79,5ed6d86759c080a49eab9c9bc18f8e0b8cd741f0,519ea45de12733a0918338a5d904277ae45457d9,31d532ef54aedea3c7f5ef60da38c0a80150fd20,2358650d33afd2c400b975cd37d7a4a348e8fdaa,1add74ea229f5705413b1ee3ca3f81f9dff2703d,6ca9141d593d964747fd0af13f83f0e54ebe5180,bac1dfa4523fb72c5bb0c0a8b2be47b5d6661fcf,d57bae8fb48c87be2ef48a704874d011d2be4f37,7bc13cae359765ebeb8125e2149e5dcc55bb71a9,cc1b7f414e11dc8739010de72d0ea17571351bbf,6f0a303adb4924219523f58154d775bf00daecd4,f1a84a27a920adb1a5d2d1db077f570a97c346a5,c7833bf91d963e866fe1f90263103fe09c90bd3c,4ebfbbf6d5fa6e22334d2c09c0614784de30bcd4,77db36a61467f6cb1e8d57e23aac43678fd252bb,44e5d5da073e210de59e3ecd3989a0e5e6fc7a51,b1f8a4ba4617968a5c198f53a71508e713eba562,5f62e1a0217968a95c36aef40066ac459c3f79b5,a22d32a70149be43dd61c70319ba86eaace4a6cb,c87b581752fd60a5e3a232643426301b84468ef4,0c4f63ed87000ab75fbdd2b0cde052e2ee7686c3,0f2a8fbd3618b23c939794d8593c29fbb0145d18,befe95106329cc3f64caa652140f7da390b1ba24,10d6c840ffa2eb81e7f6fa8011834cb5d23a174a,68e340c15c1cf84f51aba2b9ae4e24723fc6a135,a6e117a8fe2dafa2241aba4190bb7d93f9187212,b167b55782c99dcc23b8bceeb6b356c0f81fa28b,26629d54c19cd95a5d84a9dd83ba3540a62138a8,c1e239baf3f2ed191435d1c87e3de71891df90b3,ebeedc38db344cee6cccdb08b7d7586969de9de1,ad94eb3a35c0434c5a737f352b8c6af3210cbb0e,5d23cd090dd90d85403d5c1fb3c405b2849e2e36,70c28c0643c455634dea454ef2c9ef331552e70d,f5b55a9c693bb94fc83097ceb972161856e610b0,23ef9d46390a0f7710b4379a4e02bc1f94789740,93e18b235f5d786945870fd6cc70793d93b2c819,27a8f77f6b88c8f46e653702a3cd491365c9ef8a,5bf7d7929d6f4206511535f1e79f787ba2b66800,df6a33391703e8b4cdbdc4c213831e1c4c9ee5ed,ed145fe83f1c3f39de06f326a2eca937d84a79a5,45ffcf2567ed8b70ce7a35c8edb4c83440b1efc6,25029030b48029ff9953a294c4c87929d52710de,d36d4215fd1224c5e95d317b0bcddfa0faf1d07e,2611a74d2106c786119a358056c1b5aacdb07aec,ac7b7396ef897785830a1b21524804a16d855d2b,fb01e17c976ad6d716b7ad14ae16c715f813e8c3,1cd521c6127f2c0b02d54c6240d0e2a791ad6c65,fd832c35e4c1a18111ec83dbe5f4a2a02959dbde,fce27989d1dbbd1bc63667aac38540db2904212f,cbe88000f1418372940c5c5bccdba0f7e38a44c4,cc38189b940a310b665a28b700137ba10eeee37e,9db03213b3c8ffd7d919d0b915a4eb879f3dfc7e,ca97ff9c265c2804ef16674c42efcaaf4cd1caab,17316d60b9dae5f9d973131eeb4aadd105f6ea63,2f35be97f7a858e5180c85dbae4af6d82dbbb901,df821ae4a3c5e3b58f560d3b56d539922bce0e7f,07ceece617bb0088f3ddf44b123ac95929c772b9,f17661231a2f428f20b842a78f0d2f0e169f6ad3,07b7148f42add75708907dc521401ff9de1c3cbc,62312236cada8a71d798670ae7a464903e7420fd,517fec79afc18d1faa40d478617b37f24fbcf918,2d0926199740eba83c22202cd9a7dbed0a753cc2,2f3f74b91da25d303d6e00af6404c023517c8058,04165d8f98157567ad1d577d92c2812b3ef6dd66,9698f849f711f9ac981550cb39b46e2697793f6b,11e7a3264142bc49f4bfc811d56d55f7ebbead1c,b456429068fa89691e965b69c5d70b99cafc2bc2,395b029ad5c9f3d264d42dc4035177d35f2268e1,1809d6761f3121518bbb776ad755d52d63c827c2,bcbf1b418e0b5db0c04f885de517af7ba043d706,fa9a808c361a4ddb849e0d4c6f9e973386be188f,2784f3b247d3e99210f85ee8baac2bc8594ede67,d1f925a20fa8488c17dcc970071e8113e8bffcf2,26a3def0fd53617a817d93480a9161bd47f5925b,cffdecaea296057cdd1d893eec1518f3c66d207c,8be052669178e12bd7b158848004f61de5652673,f584536f00432c7d16031c5ca9c6510569dd4e38,4ec4335629a458c3616dd254b477e907829812a4,8bd4e5627bc65c0c75922f4005408660fe1cae14,47ebf8ecfda77040d5bfe63bef7d0d075f6e20da,a546a402bb4826cd8fee135187d326e2ab3bcfaf,3c65338fb27a296af81765fbbb48eb5427567883,608a9e420fd6758b93899436d2fc2a459e0abd42,c0a570853aef2d9432add4682de597c093e70c63,9664b6015b625ac395eea18c8a68c7379d71cd6d,4dd899a372767fa86cf65b4849f04e3c655f48ad,37732e60eb6f44b1120f79e89f2ccf36a6605d34,b012f94cc3666b20eddb3f5e8dda96a24415ad52,cc0aa20387c25c44266577dcb7d2a81152ac438b,9006b915937bd050436fad66af4a99100013f4f0,e58eae0a60d39b597e9f7bcedd6eca3a7c6e3916,c2e12f55c14531d00be02caeb530376e4e0d46f0,1856152761ba8b8f1c8268f549101eefb28b8d6f,95ed3fbd98f543d2012f1ef792356bd19a628589,fcac59096e3eb10bf08af44828dce9887f64a34d,5aac696b193f7c69819971448632b2c595e3d1d7,ae329e2ef0842666a48afef3d3590b1a09e65628,240f594dcc6d68e8acfde04cc3a37f725ead0c5f,e7aa4d048773a76ffdad904d00e7cac69dfec4c5,b19701e93866d0f2e84d86fca409be53dcfffdb3,33599a985a30aac0cbafbbf3531f57ba2c782102,838506f54e3fcc3f1480b4fe8dc6f288426744b3,dec0b8531bcba501cd5961bde25fc923d62b2811,65d7af551c08417162c603e1160b733881a2e7e7,94e5ff606c757679a17f9a7682ac93fe3f6fcf9d,6c15d19ea57d164d311f9aeea483f24529438eb2,a031c18fc2c8c47e9b5e25bcce032e90a67606ad,f1bfb8a25a38a9b23edf0965c00d01b262e93405,99e2402a223d8d22b60cbfdedc332491af819f83,8e38a88391e1ca05a3f2328284a4b01b93a62736,92ff845af81cd7382afd15b4ca3fd90a3a37f103,88c553b5f59b60065ee81bd3fcbceffcc95a196f,6178884a40691521447901a82365d4f0de6e50db,682045db7a85b11552214d7980314cfaec68ad17,60998a6f8d4e6d5cb58928c2f90d27018af83f54,61e2f3c228fbc717739e66879224a02d46255379,30a6076fd2372dbbb092ae2a1da615e971964f7a,c9b85c501d73b6156e146e247e31529895fb79c5,a576d06ea55d3c8f279409fbda075afd20328185,9f6b8197059525b281e49dacb1078b5764979342,0da853d86a38de1cc2887080ccdc316c72bf1090,9da2abd6b7c1d44c78bfbe4a4aa3a4b2eb81980e,05ff62952ac217ad5c0fd6984aab9d1435b456e7,76f3887214115733e7a72082747059b76fa66d37,55f9f3c343639f94de5fba4ce37e2c1c39510bbf,06bab9e7c1e213d45eb8641a3d69b074cea6e001,3d7d42e21bfd122d0eed682429babe045e4348e7,c02b14bc277df0b268a0980a64b48a57b092f0d7,5e4bdec92168d94e52f3b68374f2d8e0a47346d9,0c6902c938d97bff4daf3a9469371d4f18f70b89,52a5e7ffbc9126255c3c8d325f4e474f14080df6,5b2de9ba76ece15c95ac0455b45c068404f82a59,ce8a5593b582ae718cc4409df2274f607b55ce9c,97d8074b4b0cd4d696eb7d6f6403de61ab1fdf73,fb75636e339405b4a5340948a02fdc7cae5ddbc2,2d4202c588131fa1388ab7de82a28173fc4367c9,684e9371a755b790f28f932e5a913499f52fc9a2,1e4182e80f9e6b262640ff137b738606ecdb82a4,007f083657207f033d10846bf4d582fa444615da,1a1c0955433f4ca183d8988a4e1fa6ccf067812f,7066aaf3e102a4e1c0beeb5e09027216e7b60cbd,c81cd87203f67d4f48c8ae4ae6d53348646b10d4,c76bd396ae67986f58e411a6a54853b9d39a3c05,f221e310fafd0e9adbe605ec1d9f670ce0cf48aa,42f24b23e93f0608c0ff1bdfda4a56165d60dedc,18233e09c1318b13edc800fc603b078932d269b6,2e87e0ba6d650c74324e05cfd3932caf02cdec80,3a036f1f36d8965c82669e0b4d5f7b262dc37255,4e386bce407315bc1538c6b289b4e4e9abae9407,cc22cdef43f37945de5d63e1995266d922646ccf,76ac0a75c27142aea6ebc4a4355615654f09f7ea,d8ca7d43c121c024a13491ea2b012205b050f2d3,7d8696f42fb2cb09573459069876bfaf05bf517d,ef36f8543bb01d7a77f8e57e76a882e43044964b,4d58ff372da214178eb4a1526700ebe4c3b894ec,197b36b562d0ffd270ce3192d3cd76d45f9c2cb4,40177fa23d23f0a5c83060edbabbda292e7d6585,ac1b9de6b94655d3a73ddeca5cc1e5f8312db190,c5cc3df4ab8020d9539270f93ee3bf9756ba3b69,22ff6185fcbb7e25b4d3b5384abb11994b8e43cd,3b670cd012485bf814b6a5761777c59730b574a1,8c132d2934e5530056eec2c6a44192e7440d1930,c4edbeb54a18c1bdad2d8272b950db1286ed27e1,4cf7be1613de3bfd88e9382b3b947bf21f2d7c59,6d5b6a726eae94e9ef3c64551b5f99f1b3a50e4b,c51090341a27ab2dd3ecfb6fdfe2bbb7fb516b3c,9f3901cccd16372812abc260c365de1aa0f00bc8,da8c7670589f9ae8bc0f0553626a32e1f7d52bd0,355233c2d7367ffba4e9aeeca79ab29c77c97336,8a15ade2ee3859a328df7bc8399c04da10bd071d,5d47d6a92fc87b1b6b0fd5b14b15b58ad2d64538,167d94f64a44c373766a76e7c11cc1db41592f85,60bd48227cf08d4596453906d00ceffd84a78908,c6710c581b08406aa5a24d5516dffe2c04fc2a40,cdfdb425b72b0d74862f4168b3bde9a24e4d17ee,e761335e1f7ef2c9926d9b531b78b8379979904a,d005fbb68f3269a38a4ef6fe162476c84c196ee2,04cb34d05ef8497d5cbfe8f6b9df684a2b8412d8,ef0c4aa2c03fcda61b12054a711bb7e9ab69deee,c947a1b3b63b10b337df92660c0f5d0b27df104f,041da0d92ec85e88e88e455c3db7975aea7c069e,1ab752851c55c59c29811f2cb75a56d54078c7b9,a361160012a7f93f566cb2477f96715ec3f39eb6,a438dcaade540e0840d752d6fa6cf1a90de5c31a,52e5929321ecacdd6322f1eba6a5c1fe4b6dd13f,21a2ebf2d6efa1e0af5deace0cb5770caf01b500,4cdd7d0b726009da10ec90a82f16d19131532282,f8fd6d7916c46d8f1d78c7ee6a374cbb085d787e,7a9a5528a5e9aa93c940663065b5263b0983bba2,ca442c5566fb7f8eeb75378b2bd91e1c34db1efc,6d66eb61049179c813adad2815a39c7af1c2d900,88e8e954ad6902e4230cdda3cb41e1884616c0fa,db6137c559d267e8c7757d7f1979cdbaaeb6f28b,58bf4de2cdf0f4d7462eb4cff28f9a1f3379d592,93839fdc74e161a2ba06292fa507e031b574f98c,69a5e76059dc26a6d472a95c95840769c44e1f5e,6fde3ae70fe10d9d1d98686a8e89fef2292e5931,763ec8261e6f108af608afd96f8bd823daa73f02,e46caee47f0aed17713e2a89ee22dcc956b4f3aa,15c7a2005ab0eb0ca1b38fb307d20cadc6ab3d47,82dbff93349ec61809b98dada53431fc17902748,01be7cfd7d06d2da6ae5a04b6ad21449b60a0e8a,408dad964c2e4a0e12235ed73b57892d45f7ab44,3caa95dfd58ffa7c6fc082c9f522b2c71682c203,9801ab8b6ff0285d2450b5b87376e7038e4a3069,b912a9dd48d50b532a40d1caff0eee4cde83ed9a,7a720716f4db7f7911e8704fa07e71c464b332d7,3afc7365c6d4d61dd9a669e569bcf2ca499f3b24,5ceef63cc34811efdc4e8e259c6c72bc5dc9fa2c,59761a7197628c63f485489dae8cdcdd9b820bd9,b5e4aca2502d70fe8fac9c3ea2984ab3c74b2f88,7337b3d6a3b3f8ffc2961255da1d06fef8f96087,5d766e9095720e797f0e484047a41f306bf36ab9,80a16a25e8a43961da0ab254e297c9e1ac91f494,0ca7654f296e89c570e36eb6ff388cd5f472abc4,267744510defa2949288f735b808bab2767ecc10,b01b8031d5c00484b155820559068db6909c8b7f,95e871bc2046ca88a12974ea07d596522311b862,001e3cc320507e54d3d6f127c50f6c1eddfd72e5,2f160642e1a2038b019ecc1eddcdc465b6a1bc57,bec4543c1dec9b8860d760994d7a0df2bff6e6e2,e9a9aea961cd62558a8abef7ea10ed7edd94e975,0dbbe044f31801f468860698a81080ee782c21ff,ccbbcaccd461989d70d906561ff0a9108477d4f7,743a34985caddef65c48c3a435668cf31b36a566,5b050a13c14f912c29de14c1c8f2f6bc117610bf,295051c2b141a68604f400f99c3421367836fcc2,8ba19b4e83925f387a1e2aaebb31ca775bf56cf6,7aa213d2c07ad58c7b732cb8bcc32a59f83e3a3e,b438711a5a572e891088b6cd0015bf69df709f92,b00bcd7f921ecbe28122fa566265fd5c50cff5ee,542c68312e3960ce5d0eb2b6613a3abadcdaf324,2092184527007cfaf6f61a2eb06bd4e81921a4a6,df1d62de805829bdd0568c2c9431274856c1d282,6d6c213d2b10352e0de44228b68d4ff0bf263986,a36484280c9a6a0720d6c73341cd408ec20ae08c,1729a52b22fe68ac2d06599af0d1a63d0019c129,4f91e6a0c2dbf4a35cfbff5cf51d91dbeb126443,ed5f539d49a2be519ebeb17fa621afe1b7c1d7da,c2f73728089201bd04cf6b7f42bc96fbc47c0d9d,ff2ac3773478735206ca925c997a72af4a82bd9b,c2da9e224370619aaf13e4bc9d9325dc76c98a26,39fed535be02342cc85ac4d05e63d863d66b9ff3,fdc65e2e7da4c29daaec8115ff73826fa0943752,6c834d28895fc5544a7ef0c0991badc9c131e9b5,37a2cd5242dd0c2de1b4fe63a0954d12de832f6d,0865187b0bf51c70063ed4d1a2137dabcbb038ec,ce247f1e06c255068bc0d897f14d6e9e875fe3a8,eb6c4f5c34a627df8831ac584436d995ded4c93e,b644c6d29e0614896cd332e1c2573a727310d5e6,00026e0a69ac7dcb1bd92c9f61145f5f49e5d2a1,fb8c2f5de558584b6226fba5d5ffd43140ce7ae8,152fd74ac3dc6bf2d0bf15ca0df2bca3199df850,adbe03296b79a0b1ac76d94c55459e3598807848,a0d925269203823e8b4c5318015ac6a899c24c9d,6cddf214d1eb7310fbbbf12d395e15a2f313cf1a,baacd57e5b97a86d67c04ab6236cf17f86093a70,4a0dfe06011e667ce205143d1ab351a701dc3a67,2992f1634bcca4cca74b4322e59f857500522e1d,571245115ff9763b2c9a47996addd58aeddfbfcf,0aa3812ac1638ed2e336b349bf72a5b7e9d23d8e,7b206df057874000c52c7e45cba88a8b95aa48d8,3293db0c5e90cf0fe12c1456f6c18016579b39fe,4876550322c13e1c51f0df24f9faf7abd024e101,9a6fa088843fb74aec804a4e463646d93425f68c,a9be490a03f715053a112547489dbf60bf822b18,e68b57047d4eead6b3521a8283a5095a11f2dfbd,0939cfcb277cf9c75b4ea51f8cd423c683c2f489,439af88e01ad56e01c7f45b25b515a44fea99995,d5ec21e673c76947dc9fdb92d189c6dac4952169,35f1a701725b8c7119ee4fc94e6f8eecc709ed1b,a13a4f121d378c3adf08de8a3357f5dc32b58cf0,0161c8acf0b7121031a9c5fc68f4970e8ce1b731,1834579a35ee40b5605a734f61cdd7788ff567b0,4cd4b466ef44558c2a343be60f2b30948cb3976a,5fefda4ca1275686dc5819c1be226af48695cc9a,024418f8f944eef1062ae66af774cb89d1ccfb5f,56f584303f66b30c43d4972f0f50abee4e31446e,38eec71eeabc7ee15c3c39204fae8e675609f903,d8005be7d9fbf9b0f3f3d3d4b2ce7f9612eacac9,754428cc14f35e2673ba62b6b5016a6898d85a78,d7c2c8770b24181b323532cc9a3013c5d0907fcc,0f27d22647a3db86a778965940467d27917ce3cd,6f76eebab1a5381705300e7721834c33691c8d04,fd479d353512e210d7c9f27ef2043a164561b409,ef146112a504cd6b56aa1dc804252408ae69f8b9,78a3768d07f51da42dc7f8475521c8ce9b9ab83e,f7d3c6a86155d017fad0f0387afbd332e5b112a4,fd6fdeccb27fa169432bac7f3d0a961f0077729b,a1b4cfa7222560941fb1dd071b425cd4f47bb11b,698902511bb29faab088d0fa6b0587bfe599a8bc,5eb30283f4313b27efff902cd4622e613b0f06f0,7ac691b2a0b16acf4bbf69f0fb44074532dc117d,b3dcdc381fceed9a89748aa85fc7af04a9a7d013,b682ff7fa2cbd13bed68dfca8b23bbfed371d156,d70cc0a500fdf77f1b78ef4843f60239abfe02b7,5b2b72c4c284ca817b10c7d39434ac5a2fb2ca83,2a8efcd4e2a06951d5e37efdebff90d1f1731e5d,bb2d2e56e3cc1696de81b7b1c05d7df782f05a96,7427bde26eb381b627e348d416abac166da990d5,ea8346ad39b4736f6c47ce8508dce6f02d50c12c,132b4338ec87e0a0b835347069ace6e1eff547ef,83e273d4acc07f9211dc829ecd28d8e2442262b4,8f59f6bb00ca427c6cd5a234986516a90a2ffda7,97bc403a3ed9cc36d4f69881c7be975822c8e57f,fdccf0accb6ce97f09b17edaaadfd19ac5f9e87e,47df2940314a0626bf4c1ce3f9aa10496368a4ea,2c924c2c9deae93565355acdb7da44aa6745729d,4504df477e63ee0426202f65a58a70ba99a4e7c8,d894e7c6e0bba3d94c58cb3af42ae59416840a4c,b5d53e60b05758571ea42f78ac5a61d0ce95e9da,faa753f1110c0788cf691b0d5f5df38f84699071,7430584c602c532e720f7abd0d474f517fb0e70f,e024d0bb91cc4cc296487c943d92b4d78b555e22,16d88224ee6bc6047daa5ee63e484635b369885e,1be0a94f0719fc1f05347612afcb22311c5273d6,0ceb5d8cc008cf0aaad9c920ab2b95c538c03b52,6e30b1358c87db26ba1637a26a995382c1faf4a7,1619765f4856376ddeb28cceece4e58bbd79f756,29d4ae3d61b0cf1249e662551217c013683722e1,308aea25315c7a3be16fb926498c95b89ec83708,8d0e28360ebe931079a5ffc3460d80af82278133,93120c31e0bf91b1ad345c237f79b5e1672eb61e,5ebb7d2b5302531c45c3cbb684338b308e70c2c0,b340a502d0768041698a67cf6baae7116df163e3,6da51d98e2ff52fbbed64293ed846c56ee31e17a,7648fa35b152861569fc8c38c7943f5109153b96,f3cede6ac8bbdf36334cc6669443bbd6e19753f8,350abbbac887f5fa8bf66418ed0033dc932043cc,5c57919edc129fe102ffdde4cb8f568a7971343a,d546774375e45fe9c6433aa4820fad64016d1c89,a23093b05b9e7850a394db1518383f5a984eff64,c3b6157e5d2aecb2a21e593552f0f032c92b6d5f,b82d7129d3cab349492c523a140a88648a8e377d,5da6bce0c7db372ad168b496e960bae2e640cb6c,2b2e82bb80e27bcea4192255148886ac0a941a02,ff4c8a74091b89d22501ce2dc01d7a7f126d9629,79f0a341efaf1a26fad2c51194259d29cf5358c9,c803f92d7c55d4cc521d7ab46945577ccac3eb42,09f4d93f5315321ad9ffe81a6ddcf59c497aa328,ca9339550b3dc8383f73620b267eb0241990e5f0,5a6de018ba350030c35a51e37cbda07b8efb0fcf,d0b55d93308ce7eb512bc5c86201d16bc2b9f0f7,9156eaffcc6c5b0b443ff34e7f50a0ee1300fb0f,5c4091b0bb005eb60858535c80bb43b48312251f,871e41242196db2675abf6e6efcc854b72a914b9,c3cf16619abe2f01618f37b63469de7c197219f6,e9db314ae4d8c2534ec1fcd18bf60f0af24fd794,9727efda6c8ead257a7ed8bf7bb87d52804c7545,283dfe9a53635570769e13fdd5f84f62034240a1,3383ba1de9d914459b60398ebf3c521164ac9663,f990ffb88fa869fb482d18f107f0390665c5e7d0,c28d2abf0122afd1aa5d587b94e14d681b58ca62,d9c3fe4043b757da7c041495e36067bcc268b990,67bab056b30a8b038f19664396b1c45c85720bce,2488a8de284762bdfb6cf383f12dedc7dba40998,b534ffea5818c7a617089345f3e71609867502ad,fb5c3fff86f2f0bffb9eaaecbc1f364cada1fc2d,fbd002110336b38586dd5167fe5f02c86a821bbc,c8ee84c6451c9da1a20ac8843f4236f2e3bdb9ff,bfeca3f2e8c8f7d4d8cfd877993545a8a238857a,3aec1c13f5cc42deaf13a7a06644fe8949abdc0d,a79bb5d8a937b97537d253b67ab116bb712253c9,d227766ad671c7f658ece4a65c7bd537594dffc3,7211fa672330bd1894b1060f59cd77876ce7f679,f8d0b32eb0a6f50c89e11901379ac16583cbaf13,1b5b9c35cc74620fb9404e1f5c12642b785c40f8,c185532015350871e4030763fcda13ba4078ae9f,9e89c9d9c2146cb1693a7151ec1d9643c0d653e6,9dc6fd3e3bb44178e23c3eee6b9cb6bdee533e48,9232965cf8de10fad9512878424acad7d93b0b43,244dea8eff0ef22c1a1ac43fbf005acde49f5ea1,d63ecaef8336eee38532782fdcde793613a99c6c,798d496e0f76106c82711c6db679e41f59f571ef,daa4d3524d748ec20e1f35d7274ccd8e62508a9a,a3d583503537a7b43385c4956c0dc695fa89f029,c44258cb5b69842f678aa7b6ffb8a8925a20e3ae,fe03b53af337cc8d2d243544c4b6e08dc414c67e,98b799a359ee4b1be6697795f916270e40938a65,230182decfacdcbb35b75ea956ab86bae6b5dba5,3295ed9444ee26810ecc73c3afdddd002b9a1f26,01b07d789838d70ec839d4a2e211973dc27701d9,f57a516d5876d4b0ce19caa053026eef5e618d65,f24cd9b6cac12fbe5f10177c40ad9a5393da62b0,9f9d52dd25f90d651486083ff1a36847e6795a45,0338e36af4e4c1ba59ff43c65188936d96048b3b,711a7f39f3fd3eba76d89009f8546c9daca6e4a1,330d352ea9424b3f2c5a00aab68ba8b329b91efd,bdddc6da4fb52812256a3796a04ca926f1b67fa5,03a79a481a01ba85ec5bfd5d98d0a7eee025f33d,069452f726a6b549ebb7959cd004017e3eb3136e,6f9bb257e5863901a4d9c84ff4acdf7354ec535a,b8f7388e38414fbbaa45c670d85f71e914ba9918,356416693627b1bf7018b972957d99cf13b5f4a3,0eb7c8bc74c3e5f835e0712237db6a6acfa0e562,03674c87978535b268df3674d11f924bf46c1a51,30e9e0c75e851e69a62a9e8270babc2572fbc36e,6317ccdc029ae40187e8057c802e38661b6d709f,18ea6ddf0c59c99291e917a64c14ec931a8b95b5,db75f10eb40fa38742fbdd6a82a6496e9ffa7482,e4228f53da448216d434221b6fa7ed31ef441184,5a7f34693cad7bd4e537f3f76adb04741288736c,72c3d717ca0d07da1cb3734c8f76c97b28e5dc51,73c82f9b4971693e100302c288f2c64e3fc3a043,b093eae83add4509c2dddd30816a15fde0ba68dc,8b128bfcd6a44c360e181444c2285f180599945e,f45acb9e48fb5e71913db00548c3c0572665c4ed,03bec0ad2d983ea0f385f2e7ad36388733439e98,a695cb23df1ec2a78030bfd54db3962fe8d581a2,5a97888ff9ea705966b0cda7198c2b418cc2dc05,38cbf3f99995c0da9fce3c6fa83320369ca81ed7,d048c5ed37ec98bcb60d0ed0588597452d7f96a6,ded7f7bc6b5d71aa4281919af6c9849ee9219a15,9b574e4fc5f852153cdf501e72322966b987d1ad,1f75064bd007050f88cf3167272bb90f3ca7a2a8,29cabe3b4bd80a58f7ab4135446df51bfdb1a473,aa0207b44331b01bcaec4b7e85509cfa28244639,f9327b292a155ac1cf032f036c42619d842ef8ce,28c4ebbe4b38ba90f126a5a0c518f5b7d8bb01b8,408ac226cf33d63386c9c3b75c80bd44de1fc9c7,3a6cedb0580ce6c159145d7522c55ea29bdafedb,3c6b191eb3ab477bd1d8b6d935dea12faedc3141,50a4fe861a9117ff2ebcba24f832f229f3de573e,6991f1a3958da340a8bd6342bd9aa02fb49ab5e5,bfa10cae3e380f904c7e1a29d2d6feed1911d8d3,c0f3599faa97b6d6b2507a8e197a0776115ca80e,a93287db519a7f180ed80d4b07eb7f15b8657869,36119936a131f46f73bcdb4efc8c4c7fdefcbb96,96c90b2a08a0b7c8450260188d2bcf5a01c8b0c2,e5298f75303c1525d0453b91a05a68e80c4b8628,a7212d5449fcae5caf446f6c72b338d2a1bcdd5a,32a3c63b5aef61c0fca895bfda695cbb91d27b09,c6a892f9a5a59620b312f2a47157d2bf2f11c16a,f5cc62a2b85d803e48c5b6468b13f22a4104758a,2e8c8476bc1925b435c09e74a79e4ecef6157ae8,6eaac1a713def2191bc923fc8083794223d39569,eee7b42b689c38e89ffcf0d4382027acb6e74615,bb60f3c2d242d3fc8972e72e789357c594e2b44f,cd7d1ad0e87fbafffed1fd9efe70389e0a609294}'::text[]))
   Buffers: shared hit=2599 read=2
Settings: jit = 'off', random_page_cost = '1.5', work_mem = '230MB', seq_page_cost = '4', effective_cache_size = '472585MB'
Time: 35.521 ms
  - planning: 24.233 ms
  - execution: 11.288 ms
    - I/O read: N/A
    - I/O write: N/A

Shared buffers:
  - hits: 2599 (~20.30 MiB) from the buffer pool
  - reads: 2 (~16.00 KiB) from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

Query 3 — Commit-notes count (BatchedRelationExportService#objects_count → project.commit_notes.count). Runs once up front to size the batches. total_objects_count/batches_count are corrected to the actual enqueued values after the walk.

Plan: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/52649/commands/154326

SELECT COUNT(*) FROM notes
WHERE notes.project_id = 278964 AND notes.noteable_type = 'Commit'
Plan details 
 Aggregate  (cost=23404.28..23404.29 rows=1 width=8) (actual time=2246.987..2256.613 rows=1 loops=1)
   Buffers: shared hit=2317 read=11513
   ->  Gather  (cost=23404.06..23404.27 rows=2 width=8) (actual time=2239.711..2256.603 rows=3 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         Buffers: shared hit=2317 read=11513
         ->  Aggregate  (cost=22404.06..22404.07 rows=1 width=8) (actual time=2237.560..2237.562 rows=1 loops=3)
               Buffers: shared hit=2317 read=11513
               ->  Parallel Index Only Scan using index_notes_on_project_id_and_noteable_type on public.notes  (cost=0.58..21550.53 rows=341415 width=0) (actual time=2.086..2224.216 rows=204138 loops=3)
                     Index Cond: ((notes.project_id = 278964) AND (notes.noteable_type = 'Commit'::text))
                     Heap Fetches: 2778
                     Buffers: shared hit=2317 read=11513
Settings: seq_page_cost = '4', effective_cache_size = '472585MB', jit = 'off', random_page_cost = '1.5', work_mem = '230MB'
Time: 2.270 s
  - planning: 13.532 ms
  - execution: 2.257 s
    - I/O read: N/A
    - I/O write: N/A

Shared buffers:
  - hits: 2317 (~18.10 MiB) from the buffer pool
  - reads: 11513 (~89.90 MiB) from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

Query 4 — Serializer fetch by note IDs (StreamingSerializer#serialize_relation(batch_ids:) → project.commit_notes.where(id: note_ids)). Bounded id IN (...) lookup (≤ batch size) per batch.

Plan: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/52649/commands/154327

SELECT notes.* FROM notes
WHERE notes.project_id = 278964 AND notes.noteable_type = 'Commit'
  AND notes.id IN (?, ?, ...)
ORDER BY notes.id LIMIT ?
Plan details 
 Limit  (cost=0.58..124.16 rows=1 width=2750) (actual time=5.529..32.199 rows=40 loops=1)
   Buffers: shared hit=145 read=51
   ->  Index Scan using notes_pkey on public.notes  (cost=0.58..124.16 rows=1 width=2750) (actual time=5.526..32.179 rows=40 loops=1)
         Index Cond: (notes.id = ANY ('{3451164145,3451190502,3451219307,3451220795,3451226166,3451282638,3451322299,3451349577,3451380009,3451381310,3451397976,3451443722,3451476663,3451102945,3451122083,3451088550,3451152263,3451048531,3451066789,3451029185,3450955719,3450973636,3450938581,3450954448,3454971566,3454986451,3455004564,3450887515,3450894019,3450955924,3450827339,3450864055,3450875744,3450745954,3450804672,3450842464,3451066071,3450810002,3450815903,3450817082}'::bigint[]))
         Filter: ((notes.project_id = 278964) AND ((notes.noteable_type)::text = 'Commit'::text))
         Rows Removed by Filter: 0
         Buffers: shared hit=145 read=51
Settings: seq_page_cost = '4', effective_cache_size = '472585MB', jit = 'off', random_page_cost = '1.5', work_mem = '230MB'
Time: 177.243 ms
  - planning: 144.957 ms
  - execution: 32.286 ms
    - I/O read: N/A
    - I/O write: N/A

Shared buffers:
  - hits: 145 (~1.10 MiB) from the buffer pool
  - reads: 51 (~408.00 KiB) from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

How to set up and validate locally

Setup

  1. Enable the feature flag (gdk rails console):

    Feature.enable(:commit_notes_export_via_repo)

  2. Seed commit notes on a project that has a repository, spreading them across the repository's commits (Rails console):

    project = Project.find_by_full_path('flightjs/Flight') # any project with a repository
user    = project.first_owner

shas = project.repository.commits(nil, limit: 50, all: true).map(&:id)

2000.times do |i|
  project.notes.create!(
    author: user,
    noteable_type: 'Commit',
    commit_id: shas[i % shas.size],
    note: "Seeded commit note ##{i + 1}"
  )
end

project.notes.where(noteable_type: 'Commit').count # => 2000

    Set PROJECT_ID, TOKEN (a personal access token) and HOST for the API calls below:

    PROJECT_ID=<id>
TOKEN=<your-pat>
HOST=http://gdk.test:3000

Validate Direct Transfer (export_relations)

  1. Start a batched relations export and poll the commit_notes relation:

    curl --request POST --header "PRIVATE-TOKEN: $TOKEN" \
  "$HOST/api/v4/projects/$PROJECT_ID/export_relations?batched=true"

curl --silent --header "PRIVATE-TOKEN: $TOKEN" \
  "$HOST/api/v4/projects/$PROJECT_ID/export_relations/status?relation=commit_notes" | jq

    The payload shows "batched": true, "total_objects_count": 2000, and "batches_count": 4 (2000 ÷ 500), with no empty batches.

  2. Download a batch and confirm it holds commit notes:

    curl --silent --header "PRIVATE-TOKEN: $TOKEN" \
  "$HOST/api/v4/projects/$PROJECT_ID/export_relations/download?relation=commit_notes&batched=true&batch_number=1" \
  | gunzip | head

    Each line is a serialized commit note ("noteable_type":"Commit" with a commit_id).

Validate file-based Import/Export

  1. Trigger a project export and inspect the archive:

    curl --request POST --header "PRIVATE-TOKEN: $TOKEN" "$HOST/api/v4/projects/$PROJECT_ID/export"

curl --silent --header "PRIVATE-TOKEN: $TOKEN" \
  "$HOST/api/v4/projects/$PROJECT_ID/export" | jq '.export_status' # wait for "finished"

curl --silent --header "PRIVATE-TOKEN: $TOKEN" \
  "$HOST/api/v4/projects/$PROJECT_ID/export/download" --output export.tar.gz

tar -xzf export.tar.gz
head tree/project/commit_notes.ndjson
wc -l tree/project/commit_notes.ndjson # count lines

    tree/project/commit_notes.ndjson contains the seeded notes.

Validate cross-project isolation (forks)

A fork shares its parent's commit SHAs, so a commit note on the same SHA in a different project must not leak into this project's export.

  1. For the project, create a note on the same SHA in forked project (Rails console):

    
shared_sha = 'ABC' # commit in the original project that has a commit note

other = Project.find_by_full_path('forked-project')
other.notes.create!(
  author: other.first_owner,
  noteable_type: 'Commit',
  commit_id: shared_sha,
  note: 'Belongs to the other project'
)

  2. Re-run the export for project and confirm the exported commit_notes.ndjson does not contain Belongs to the other project — only project's own notes appear. This verifies the project scoping in CommitNotesBatcher (the commit_id IN (...) lookup is intentionally cross-project for index efficiency, then filtered to the exporting project).

Confirm the new path is exercised

With the flag enabled, commit notes are resolved by walking the repository (CommitNotesBatcher) rather than paginating notes by noteable_type='Commit'. Toggle the flag off and re-run to compare; the exported NDJSON is identical.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rodrigo Tomonari

Merge request reports