Add BulkImports::Failure model to capture Bulk Import failures

What does this MR do?

This MR:

• adds a new BulkImports::Failure model to help track import failures during Group Migration process (read about this new feature Import Group is working on here: &4374 (closed))
• adds a new db table bulk_import_failures to store import failures
• adds new failure capturing logic to BulkImports::Pipeline::Runner

There are 2 types of failures that we would like to track: the ones that abort the entire run & the ones that don't.

1. If an error occurred during group creation - there is no need in trying to import epics or anything else into it, since group does not exist. In this case, raise an error, mark entire run to abort.
2. If during import of 300 epics 1 epic failed to import - track the error but keep going. This is similar to existing Project/Group Import's ImportFailure logic.

The overall Group Migration feature is still in development, not rolled out and is behind bulk_import feature flag (that is turned off).

Mentions #268022 (closed)

Migration Output

Up

== 20201112132808 CreateBulkImportFailures: migrating =========================
-- table_exists?(:bulk_import_failures)
   -> 0.0005s
-- create_table(:bulk_import_failures)
   -> 0.0176s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0002s
-- execute("ALTER TABLE bulk_import_failures\nADD CONSTRAINT check_053d65c7a4\nCHECK ( char_length(pipeline_class) <= 255 )\nNOT VALID;\n")
   -> 0.0006s
-- current_schema()
   -> 0.0001s
-- execute("SET statement_timeout TO 0")
   -> 0.0002s
-- execute("ALTER TABLE bulk_import_failures VALIDATE CONSTRAINT check_053d65c7a4;")
   -> 0.0005s
-- execute("RESET ALL")
   -> 0.0002s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0002s
-- execute("ALTER TABLE bulk_import_failures\nADD CONSTRAINT check_c7dba8398e\nCHECK ( char_length(exception_class) <= 255 )\nNOT VALID;\n")
   -> 0.0004s
-- current_schema()
   -> 0.0002s
-- execute("ALTER TABLE bulk_import_failures VALIDATE CONSTRAINT check_c7dba8398e;")
   -> 0.0005s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0002s
-- execute("ALTER TABLE bulk_import_failures\nADD CONSTRAINT check_6eca8f972e\nCHECK ( char_length(exception_message) <= 255 )\nNOT VALID;\n")
   -> 0.0003s
-- current_schema()
   -> 0.0002s
-- execute("ALTER TABLE bulk_import_failures VALIDATE CONSTRAINT check_6eca8f972e;")
   -> 0.0005s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0002s
-- execute("ALTER TABLE bulk_import_failures\nADD CONSTRAINT check_e787285882\nCHECK ( char_length(correlation_id_value) <= 255 )\nNOT VALID;\n")
   -> 0.0003s
-- current_schema()
   -> 0.0002s
-- execute("ALTER TABLE bulk_import_failures VALIDATE CONSTRAINT check_e787285882;")
   -> 0.0004s
== 20201112132808 CreateBulkImportFailures: migrated (0.0384s) ================

Down

== 20201112132808 CreateBulkImportFailures: reverting =========================
-- drop_table(:bulk_import_failures)
   -> 0.0020s
== 20201112132808 CreateBulkImportFailures: reverted (0.0070s) ================

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• [-] Documentation (if required)
• Code review guidelines
• [-] Merge request performance guidelines
• Style guides
• Database guides
• [-] Separation of EE specific content

Availability and Testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• Tested in all supported browsers
• Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

• Label as security and @ mention @gitlab-com/gl-security/appsec
• The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• Security reports checked/validated by a reviewer from the AppSec team

changed milestone to %13.6

added backend devopsmanage featureenhancement + 1 deleted label

added typefeature label

added databasereview pending label

added database label

	1 Warning
	featureaddition and featureenhancement merge requests normally have a documentation change. Consider adding a documentation update or confirming the documentation plan with the Technical Writer counterpart. For more information, see: The Handbook page on throughput implementation. The definition of done documentation.

Reviewer roulette

Changes that require review have been detected! A merge request is normally reviewed by both a reviewer and a maintainer in its primary category (e.g. frontend or backend), and by a maintainer in all other categories.

To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited, or the chosen person is unavailable.

To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.

Once you've decided who will review this merge request, mention them as you normally would! Danger does not automatically notify them for you.

Category	Reviewer	Maintainer
backend	Sean Arnold (@seanarnold) (UTC+13, 13 hours ahead of @georgekoltsov)	Shinya Maeda (@shinya.maeda) (UTC+7, 7 hours ahead of @georgekoltsov)
database	Robert May (@robotmay_gitlab) (UTC+0, same timezone as @georgekoltsov)	Mayra Cabrera (@mayra-cabrera) (UTC-6, 6 hours behind @georgekoltsov)

If needed, you can retry the danger-review job that generated this comment.

Generated by Danger

added 1 commit

• 578f596c - Add BulkImports::Failure model to capture Bulk Import failures

Compare with previous version

added sectiondev label

added 616 commits

• 578f596c...792c1ca1 - 615 commits from branch master
• 0e901b88 - Add BulkImports::Failure model to capture Bulk Import failures

Compare with previous version

added 1 commit

• 4d5ebf52 - Add BulkImports::Failure model to capture Bulk Import failures

Compare with previous version

added 1 commit

• a31bb4e5 - Add BulkImports::Failure model to capture Bulk Import failures

Compare with previous version

added 1 commit

• 2a8bca60 - Add BulkImports::Failure model to capture Bulk Import failures

Compare with previous version

added 1 commit

• b373fc1a - Add BulkImports::Failure model to capture Bulk Import failures

Compare with previous version

added 1 commit

• eb4550c6 - Add BulkImports::Failure model to capture Bulk Import failures

Compare with previous version

added 1 commit

• 54165065 - Add BulkImports::Failure model to capture Bulk Import failures

Compare with previous version

added 1 commit

• 180d3050 - Add BulkImports::Failure model to capture Bulk Import failures

Compare with previous version

added 1 commit

• 149191ae - Add BulkImports::Failure model to capture Bulk Import failures

Compare with previous version

@kassio may I ask you to do an early review for this? still a draft as I polish it up and get green, but for the most part it's ready for review

assigned to @kassio

changed the description

mentioned in merge request !47923 (merged)

added 708 commits

• 149191ae...e09933bf - 707 commits from branch master
• 75db4225 - Add BulkImports::Failure model to capture Bulk Import failures

Compare with previous version

added 1 commit

• 70b0f5f6 - Add BulkImports::Failure model to capture Bulk Import failures

Compare with previous version

marked this merge request as ready

unassigned @kassio

changed milestone to %13.7

added missed:13.6 label

added 162 commits

• 70b0f5f6...ad24346b - 161 commits from branch master
• 5020b2ad - Add BulkImports::Failure model to capture Bulk Import failures

Compare with previous version

added 1 commit

• 6cb7367a - Add BulkImports::Failure model to capture Bulk Import failures

Compare with previous version

resolved all threads

added 1 commit

• ad3f8117 - Add BulkImports::Failure model to capture Bulk Import failures

Compare with previous version

added 1 commit

• c9964b87 - Add BulkImports::Failure model to capture Bulk Import failures

Compare with previous version

@robotmay_gitlab may I ask you to do database review of this change, please?

assigned to @kassio and @robotmay_gitlab

unassigned @kassio

approved this merge request

database looks good to me. Could you add the output of running the migrations both up & down to the description of the MR?

I'll pass onto the database maintainer anyway on the assumption that will be fine

added databasereviewed label and removed databasereview pending label

assigned to @pbair and unassigned @robotmay_gitlab

added 126 commits

• c9964b87...85332ae2 - 125 commits from branch master
• 9f47c44a - Add BulkImports::Failure model to capture Bulk Import failures

Compare with previous version

marked the checklist item Changelog entry as completed

changed the description

assigned to @kassio

approved this merge request

unassigned @kassio

added 13 commits

• 9f47c44a...dd13cc03 - 12 commits from branch master
• 8d79b3ce - Add BulkImports::Failure model to capture Bulk Import failures

Compare with previous version

@engwan may I ask you for backend maintainer review, please?

assigned to @engwan

changed the description

Sorry I wasn't able to get to this today and I'll be OOO until next Monday.

Can you assign to another maintainer please?

unassigned @engwan

added databaseapproved label and removed databasereviewed label

approved this merge request

@georgekoltsov LGTM from database. I left one small question, but if you think this is fine as-is then I see no problems with it.

unassigned @pbair

@jameslopez may I ask you to review this change, please?

assigned to @jameslopez

Thanks @georgekoltsov - looks good, just a couple of questions :)

unassigned @jameslopez

mentioned in issue #271233 (closed)

added 741 commits

• 8d79b3ce...80aa006c - 740 commits from branch master
• e4eae80e - Add BulkImports::Failure model to capture Bulk Import failures

Compare with previous version

assigned to @jameslopez

resolved all threads

approved this merge request

merged

mentioned in commit c6da895b

added releasedcandidate label

added workflowstaging label

added workflowcanary label and removed workflowstaging label

added workflowproduction label and removed workflowcanary label

added releasedpublished label and removed releasedcandidate label

added groupimport and integrate label and removed 1 deleted label