PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_issues_on_project_id_and_iid"

Summary/Details

• On long running imports in potentially very active projects it is possible to fail to import some issues because of conflicting iids.
• Related sentry exception https://sentry.gitlab.net/gitlab/gitlabcom/issues/1645806/?referrer=gitlab_plugin
• somehow being triggered by http://gitlab.com/gitlab-org/gitlab/blob/b228b1568657c2a5d5724e6c7eca899bd47193e2/lib/gitlab/import_export/importer.rb#L21-21 restore is somehow finding duplicates...

Proposal/Plan

• Maybe we can improve iid allocation in Jira Imports by making use of the InternalId by generating the iid for each issue InternalId.generate_next(project, {project: project}, :issues, project&.issues&.maximum(:iid) ) which will make several queries for every issue:

[43] pry(main)> InternalId.generate_next(project, {project: project}, :issues, project&.issues&.maximum(:iid) )
   (1.0ms)  SELECT MAX("issues"."iid") FROM "issues" WHERE "issues"."project_id" = 22
   (0.2ms)  BEGIN
  InternalId Load (1.3ms)  SELECT "internal_ids".* FROM "internal_ids" WHERE "internal_ids"."project_id" = 22 AND "internal_ids"."usage" = 0 LIMIT 1
  InternalId Load (0.5ms)  SELECT "internal_ids".* FROM "internal_ids" WHERE "internal_ids"."id" = 97 LIMIT 1 FOR UPDATE
  InternalId Update (0.5ms)  UPDATE "internal_ids" SET "last_value" = 1701 WHERE "internal_ids"."id" = 97
   (0.4ms)  COMMIT

• or try to allocate iids to cover the size of the issues to be imported I guess with something along the lines:

current_iid = InternalId.generate_next(project, {project: project}, :issues, project&.issues&.maximum(:iid) )
last_iid = InternalId.track_greatest(project, { project: project }, :issues, current_iid + issues.size, project&.issues&.maximum(:iid) )