ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_merge_requests...

https://sentry.gitlab.net/gitlab/gitlabcom/issues/1960839/?referrer=gitlab_plugin

PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_merge_requests_on_target_project_id_and_iid"
DETAIL:  Key (target_project_id, iid)=(22536282, 2) already exists.

  active_record/connection_adapters/postgresql_adapter.rb:675:in `exec_params'
    @connection.exec_params(sql, type_casted_binds)
  active_record/connection_adapters/postgresql_adapter.rb:675:in `block (2 levels) in exec_no_cache'
    @connection.exec_params(sql, type_casted_binds)
  active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
    yield
  active_support/concurrency/share_lock.rb:187:in `yield_shares'
    yield
  active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
    @lock.yield_shares(compatible: [:load]) do
...
(185 additional frame(s) were not displayed)

ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_merge_requests_on_target_project_id_and_iid"
DETAIL:  Key (target_project_id, iid)=(22536282, 2) already exists.

logs - https://log.gprd.gitlab.net/app/discover#/?_g=h@e011096&_a=h@243de58

Plan

• Started a discussion about possible incorrect shard issues in #332616 (comment 685637339).
• Once that issue is resolved, we plan on observing for a week or so after and see what errors remain.
  • If no further errors, close this issue as resolved.
  • If there are more errors produced, but we are on the right shard, we may need to still solve this issue.
    • Consider the case of workers running more than once perhaps through deployments where sidekiq reliable fetch is invoked
      • somehow track the position of the import as it proceeds and only start where it left off(in case of rerun)
        • position = filename and line number
        • record this position in Redis and only save the position again when the next line starts.
      • consider upsert if not too expensive of an operation instead of tracking the position. See - #285111 (comment 685766705)
      • semi-related #296960 (closed)
    • Consider the case of the import files being modified in terms of the iid (less likely)