PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_issues_on_project_id_and_iid"
Summary/Details
- On long running imports in potentially very active projects it is possible to fail to import some issues because of conflicting iids.
- Related sentry exception https://sentry.gitlab.net/gitlab/gitlabcom/issues/1645806/?referrer=gitlab_plugin
- somehow being triggered by http://gitlab.com/gitlab-org/gitlab/blob/b228b1568657c2a5d5724e6c7eca899bd47193e2/lib/gitlab/import_export/importer.rb#L21-21
restore
is somehow finding duplicates...
Proposal/Plan
- Maybe we can improve iid allocation in Jira Imports by making use of the
InternalId
by generating the iid for each issueInternalId.generate_next(project, {project: project}, :issues, project&.issues&.maximum(:iid) )
which will make several queries for every issue:
[43] pry(main)> InternalId.generate_next(project, {project: project}, :issues, project&.issues&.maximum(:iid) )
(1.0ms) SELECT MAX("issues"."iid") FROM "issues" WHERE "issues"."project_id" = 22
(0.2ms) BEGIN
InternalId Load (1.3ms) SELECT "internal_ids".* FROM "internal_ids" WHERE "internal_ids"."project_id" = 22 AND "internal_ids"."usage" = 0 LIMIT 1
InternalId Load (0.5ms) SELECT "internal_ids".* FROM "internal_ids" WHERE "internal_ids"."id" = 97 LIMIT 1 FOR UPDATE
InternalId Update (0.5ms) UPDATE "internal_ids" SET "last_value" = 1701 WHERE "internal_ids"."id" = 97
(0.4ms) COMMIT
- or try to allocate iids to cover the size of the issues to be imported I guess with something along the lines:
current_iid = InternalId.generate_next(project, {project: project}, :issues, project&.issues&.maximum(:iid) )
last_iid = InternalId.track_greatest(project, { project: project }, :issues, current_iid + issues.size, project&.issues&.maximum(:iid) )
Edited by Doug Stull