Store entities that were not imported with errors in DB
Problem
Currently we do not provide exact feedback to the user on finished import process. We show how many entities were imported, but we don't provide which entities exactly were not successfully imported. We also do not show errors that prevented entities to get imported.
In this issue, we want to store in DB details about entities that were not imported. This data will be needed later to build a direct link (to GitHub) to the item that was not imported.
Proposal
In each worker responsible for creating a record on the last retry (exhausted) save the entity's identifiers (from GitHub) to the DB. For this purpose we want to extend existing import_failure
table as it is mentioned in the comment (adding external_identifiers
column).
Technical details
We already store error details in the DB using Gitlab::Import::ImportFailureService
. This means that when we can't import some entity during the GitHub project import, we create a new record in the import_failures
table. We do it on every worker retry.
There may be the case, when after some retries the item will be successfully imported, which means we have to collect errors only on the last retry (when retries are exhausted) to make sure we are not collecting false negative results.