Skip to content

Add stats API to Direct Transfer

George Koltsov requested to merge georgekoltsov/direct-transfer-stats into master

What does this MR do and why?

This MR adds stats to Direct Transfer API so we can see how many objects were fetched & imported. Similar to GitHub Importer stats we provide.

Migration output

Click to expand
main: == [advisory_lock_connection] object_id: 174460, pg_backend_pid: 62754
main: == 20231016092554 AddObjectCountFieldsToBulkImportTrackers: migrating =========
main: -- add_column(:bulk_import_trackers, :imported_objects_count, :integer, {:null=>false, :default=>0})
main:    -> 0.0014s
main: -- add_column(:bulk_import_trackers, :fetched_objects_count, :integer, {:null=>false, :default=>0})
main:    -> 0.0005s
main: == 20231016092554 AddObjectCountFieldsToBulkImportTrackers: migrated (0.0054s) 

main: == [advisory_lock_connection] object_id: 174460, pg_backend_pid: 62754
ci: == [advisory_lock_connection] object_id: 174760, pg_backend_pid: 62756
ci: == 20231016092554 AddObjectCountFieldsToBulkImportTrackers: migrating =========
ci: -- add_column(:bulk_import_trackers, :imported_objects_count, :integer, {:null=>false, :default=>0})
ci:    -> 0.0011s
ci: -- add_column(:bulk_import_trackers, :fetched_objects_count, :integer, {:null=>false, :default=>0})
ci:    -> 0.0005s
ci: == 20231016092554 AddObjectCountFieldsToBulkImportTrackers: migrated (0.0084s) 

update! query plan & execution

Click to expand
 EXPLAIN UPDATE "bulk_import_trackers" SET "updated_at" = '2023-10-19 08:54:04.889990', "fetched_objects_count" = 1 WHERE "bulk_import_trackers"."id" = 956;
                                                 QUERY PLAN                                                  
-------------------------------------------------------------------------------------------------------------
 Update on bulk_import_trackers  (cost=0.14..2.16 rows=0 width=0)
   ->  Index Scan using bulk_import_trackers_pkey on bulk_import_trackers  (cost=0.14..2.16 rows=1 width=18)
         Index Cond: (id = 956)
(3 rows)

Screenshots or screen recordings

image

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by George Koltsov

Merge request reports