Make GitHub importer jobs idempotent
Zendesk Ticket: https://gitlab.zendesk.com/agent/tickets/111729 (internal link)
A customer is trying to import a large GitHub project using the GitHub integration, which fails because of memory constraints. In addition to the memory problems, however, I'm also seeing numerous occurrences of the following error in the Sidekiq logs during the import:
2019-01-14_11:54:20.82729 2019-01-14T11:54:20.826Z 45265 TID-ovcuzjc6h Gitlab::GithubImport::ImportPullRequestWorker JID-7448d305b1ee7f85b454ec29 INFO: fail: 13.281 sec
2019-01-14_11:54:20.82731 2019-01-14T11:54:20.826Z 45265 TID-ovcuzjc6h WARN: {"context":"Job raised exception","job":{"class":"Gitlab::GithubImport::ImportPullRequestWorker","args":[7366,{"iid":6717,"title":"<REDACTED>","description":"<REDACTED>","source_branch":"<REDACTED>","target_branch":"develop","source_branch_sha":"0337dec89775965760156af6f8fc4bdd6ea88a72","target_branch_sha":"3ed60e99e732c47ce708af61efd277dcddc11503","source_repository_id":302,"target_repository_id":142,"source_repository_owner":"<USERNAME>","state":"closed","milestone_number":null,"author":{"id":76,"login":"<USERNAME>"},"assignee":null,"created_at":"2017-05-08 01:43:02 UTC","updated_at":"2017-05-15 20:47:37 UTC","merged_at":"2017-05-15 20:47:37 UTC"},"gitlab:job_waiter:2ba78725-3179-450d-8f74-21e1c92efbc1"],"retry":5,"queue":"github_importer:github_import_import_pull_request","queue_namespace":"github_importer","dead":false,"jid":"7448d305b1ee7f85b454ec29","created_at":1547466772.4627714,"enqueued_at":1547466772.758825},"jobstr":"{\"class\":\"Gitlab::GithubImport::ImportPullRequestWorker\",\"args\":[7366,{\"iid\":6717,\"title\":\"<REDACTED>\",\"description\":\"<REDACTED>\",\"source_branch\":\"<REDACTED>\",\"target_branch\":\"develop\",\"source_branch_sha\":\"0337dec89775965760156af6f8fc4bdd6ea88a72\",\"target_branch_sha\":\"3ed60e99e732c47ce708af61efd277dcddc11503\",\"source_repository_id\":302,\"target_repository_id\":142,\"source_repository_owner\":\"<USERNAME>\",\"state\":\"closed\",\"milestone_number\":null,\"author\":{\"id\":76,\"login\":\"<REDACTED>\"},\"assignee\":null,\"created_at\":\"2017-05-08 01:43:02 UTC\",\"updated_at\":\"2017-05-15 20:47:37 UTC\",\"merged_at\":\"2017-05-15 20:47:37 UTC\"},\"gitlab:job_waiter:2ba78725-3179-450d-8f74-21e1c92efbc1\"],\"retry\":5,\"queue\":\"github_importer:github_import_import_pull_request\",\"queue_namespace\":\"github_importer\",\"dead\":false,\"jid\":\"7448d305b1ee7f85b454ec29\",\"created_at\":1547466772.4627714,\"enqueued_at\":1547466772.758825}"}
2019-01-14_11:54:20.82746 2019-01-14T11:54:20.826Z 45265 TID-ovcuzjc6h WARN: ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_merge_request_diff_commits_on_mr_diff_id_and_order"
2019-01-14_11:54:20.82751 DETAIL: Key (merge_request_diff_id, relative_order)=(53434, 0) already exists.
2019-01-14_11:54:20.82752 : INSERT INTO merge_request_diff_commits ("message", "authored_date", "author_name", "author_email", "committed_date", "committer_name", "committer_email", "merge_request_diff_id", "relative_order", "sha")
2019-01-14_11:54:20.82753 VALUES ('Window scroll as per code review
2019-01-14_11:54:20.82754 ', '2017-05-15 20:25:31.000000', '<REDACTED_NAME>', '<REDACTED_EMAIL>', '2017-05-15 20:25:31.000000', '<REDACTED_NAME>', '<REDACTED_EMAIL>', 53434, 0, '\x0337dec89775965760156af6f8fc4bdd6ea88a72'), ('merge conflict resolution
2019-01-14_11:54:20.82755 ', '2017-05-13 18:26:57.000000', '<REDACTED_NAME>', '<REDACTED_EMAIL>', '2017-05-13 18:26:57.000000', '<REDACTED_NAME>', '<REDACTED_EMAIL>', 53434, 1, '\x21fd72d727ac060fa6001ce5f541ed8c523005ef'), ('=3316124 <REDACTED>
2019-01-14_11:54:20.82756 ', '2017-05-08 01:36:39.000000', '<REDACTED_NAME>', '<REDACTED_EMAIL>', '2017-05-10 15:31:21.000000', '<REDACTED_NAME>', '<REDACTED_EMAIL>', 53434, 2, '\x6a41876df191edb5d4a00f4cde98858027a2aef1')
2019-01-14_11:54:20.82759
I wasn't able to find any issue reporting the above error. It doesn't look to be related to the memory problems. The customer is running 11.4.5-ee.
Edited by Stan Hu