Importer: Ensure specs produce no `ImportFailure`s
Problem
Last year we started to track individually failing relations by inserting them as ImportFailure
s instead of failing fast. Unfortunately this has led to several problems going under the radar. On master
, there are several specs that currently pass, but where the imports that are being tested are actually partial failures because ImportFailure
s exist. Since we do not assert that there are no failures currently, these tests produce false negatives since they are green when they should be red instead.
Goal
We need to investigate and fix all existing ImportFailure
s in tests. Once fixed, we should then have any import spec that is a functional test case (i.e. where actual imports execute) assert that ImportFailure.all
is always empty, so that we can detect regressions quickly and easily.
Approach
I suggest we add a test assertion such as expect(ImportFailure.all).to eq([])
to the following specs:
spec/lib/gitlab/import_export/project_tree_restorer_spec.rb
spec/lib/gitlab/import_export/import_export_equivalence_spec.rb
and perhaps others that I might be unaware of.
We need to investigate and fix the following errors so that we arrive at a green build with 0
import failures.
The project_tree_restorer_spec
would currently fail on the following items:
+[#<ImportFailure:0x000055f16e263af8
+ id: 1,
+ relation_index: 0,
+ project_id: 1,
+ created_at: Wed, 19 Feb 2020 08:39:31 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordNotUnique",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_external_pull_requests_on_project_and_branches\"\n" +
+ "DETAIL: Key (project_id, source_branch, target_branch)=(1, feature, master) already exists.\n",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">]
The import_export_equivalence_spec
spec would currently fail on the following items:
+[#<ImportFailure:0x000055f16cd64f80
+ id: 18,
+ relation_index: 0,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:06 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "Gitlab::Git::Repository::NoRepository",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "5:GetRepoPath: not a git repository '/home/git/gitlab/tmp/tests/repositories/@hashed/a2/18/a21855da08cb102d1d217c53dc5824a3a795c1c1a44e971bf01ab9da3a2acbbf.git'",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cd64c10
+ id: 19,
+ relation_index: 1,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:08 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveModel::UnknownAttributeError",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message: "unknown attribute 'diff_head_sha' for MergeRequest.",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cd64a80
+ id: 20,
+ relation_index: 0,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:08 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordNotUnique",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_external_pull_requests_on_project_and_branches\"\n" +
+ "DETAIL: Key (project_id, source_branch, target_branch)=(68, feature, master) already exists.\n",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cd649b8
+ id: 21,
+ relation_index: 1,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:08 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordInvalid",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message: "Validation failed: Owner can't be blank",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cd648f0
+ id: 22,
+ relation_index: 0,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordInvalid",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "Validation failed: Sentry issue is already associated to a GitLab Issue. New issue will not be associated.",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cd64800
+ id: 23,
+ relation_index: 1,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordNotUnique",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_issues_on_project_id_and_iid\"\n" +
+ "DETAIL: Key (project_id, iid)=(68, 9) already exists.\n",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cd64738
+ id: 24,
+ relation_index: 2,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordNotUnique",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_issues_on_project_id_and_iid\"\n" +
+ "DETAIL: Key (project_id, iid)=(68, 8) already exists.\n",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cd64648
+ id: 25,
+ relation_index: 3,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordNotUnique",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_issues_on_project_id_and_iid\"\n" +
+ "DETAIL: Key (project_id, iid)=(68, 7) already exists.\n",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cd64580
+ id: 26,
+ relation_index: 4,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordNotUnique",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_issues_on_project_id_and_iid\"\n" +
+ "DETAIL: Key (project_id, iid)=(68, 6) already exists.\n",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cd644b8
+ id: 27,
+ relation_index: 5,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordNotUnique",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_issues_on_project_id_and_iid\"\n" +
+ "DETAIL: Key (project_id, iid)=(68, 5) already exists.\n",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cd643f0
+ id: 28,
+ relation_index: 6,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordNotUnique",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_issues_on_project_id_and_iid\"\n" +
+ "DETAIL: Key (project_id, iid)=(68, 4) already exists.\n",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cd64328
+ id: 29,
+ relation_index: 7,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordNotUnique",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_issues_on_project_id_and_iid\"\n" +
+ "DETAIL: Key (project_id, iid)=(68, 3) already exists.\n",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cd641e8
+ id: 30,
+ relation_index: 8,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordNotUnique",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_issues_on_project_id_and_iid\"\n" +
+ "DETAIL: Key (project_id, iid)=(68, 2) already exists.\n",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cdabf98
+ id: 31,
+ relation_index: 9,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordNotUnique",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_issues_on_project_id_and_iid\"\n" +
+ "DETAIL: Key (project_id, iid)=(68, 1) already exists.\n",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cdabe58
+ id: 32,
+ relation_index: 0,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "Gitlab::Git::Repository::NoRepository",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "5:GetRepoPath: not a git repository '/home/git/gitlab/tmp/tests/repositories/@hashed/a2/18/a21855da08cb102d1d217c53dc5824a3a795c1c1a44e971bf01ab9da3a2acbbf.git'",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cdabcf0
+ id: 33,
+ relation_index: 1,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordNotUnique",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_merge_requests_on_target_project_id_and_iid\"\n" +
+ "DETAIL: Key (target_project_id, iid)=(68, 7) already exists.\n",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cdabc28
+ id: 34,
+ relation_index: 2,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordNotUnique",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_merge_requests_on_target_project_id_and_iid\"\n" +
+ "DETAIL: Key (target_project_id, iid)=(68, 6) already exists.\n",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cdabae8
+ id: 35,
+ relation_index: 3,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordNotUnique",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_merge_requests_on_target_project_id_and_iid\"\n" +
+ "DETAIL: Key (target_project_id, iid)=(68, 5) already exists.\n",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cdaba20
+ id: 36,
+ relation_index: 4,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordNotUnique",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_merge_requests_on_target_project_id_and_iid\"\n" +
+ "DETAIL: Key (target_project_id, iid)=(68, 4) already exists.\n",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cdab958
+ id: 37,
+ relation_index: 5,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordNotUnique",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_merge_requests_on_target_project_id_and_iid\"\n" +
+ "DETAIL: Key (target_project_id, iid)=(68, 3) already exists.\n",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cdab818
+ id: 38,
+ relation_index: 6,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordNotUnique",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_merge_requests_on_target_project_id_and_iid\"\n" +
+ "DETAIL: Key (target_project_id, iid)=(68, 2) already exists.\n",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cdab728
+ id: 39,
+ relation_index: 7,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:10 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordNotUnique",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_merge_requests_on_target_project_id_and_iid\"\n" +
+ "DETAIL: Key (target_project_id, iid)=(68, 1) already exists.\n",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cdab610
+ id: 40,
+ relation_index: 1,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:10 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordNotUnique",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_ci_pipelines_on_project_id_and_iid\"\n" +
+ "DETAIL: Key (project_id, iid)=(68, 1) already exists.\n",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cdab548
+ id: 41,
+ relation_index: 3,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:10 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordNotUnique",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_external_pull_requests_on_project_and_branches\"\n" +
+ "DETAIL: Key (project_id, source_branch, target_branch)=(68, feature, master) already exists.\n",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cdab458
+ id: 42,
+ relation_index: 0,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:10 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordNotUnique",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message:
+ "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_external_pull_requests_on_project_and_branches\"\n" +
+ "DETAIL: Key (project_id, source_branch, target_branch)=(68, feature, master) already exists.\n",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cdab2f0
+ id: 43,
+ relation_index: 0,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:10 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordInvalid",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message: "Validation failed: Name has already been taken",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cdab200
+ id: 44,
+ relation_index: 0,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:10 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordInvalid",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message: "Validation failed: Key has already been taken",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">,
+ #<ImportFailure:0x000055f16cdaaf80
+ id: 45,
+ relation_index: 1,
+ project_id: 68,
+ created_at: Wed, 19 Feb 2020 08:40:10 UTC +00:00,
+ relation_key: [FILTERED],
+ exception_class: "ActiveRecord::RecordInvalid",
+ correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
+ exception_message: "Validation failed: Key has already been taken",
+ retry_count: 0,
+ group_id: nil,
+ source: "process_relation_item!">]