GitLab Migration: Track number of partially completed GitLab migrations
Problem
In GitLab Migration: Track number of attempted/fai... (#341506 - closed) we added tracking GitLab migrations that finished, failed or timeouted.
We don't track however how many GitLab migrations completed successfully and how many only partially.
Proposal
Create a "Partial import" state for migration that didn't migrate all resources and/or didn't map all users.
Track this state to be able to show the number on partial migrations on Sisense chart.
Introduce "Fully completed" for migration that migrated all the resources.
Should we have a state where 98% percent of resources were migrated, as Fully completed migration might not happen at all for big groups/projects?
UPDATE Refinement meeting Dec 20 notes Proposal:
1st step in scope of this issue is to start identifying which imports are successful and which are partially successfully.
Partially complete = if there is a failure at any stage
Fully complete = if there are no failures at all
Keep "finished" as a state. But tag imports with "incomplete" if any entities fail to import.
We can expand on the degree of detail (i.e. percentage completed) later. And decide on any percentage of import to equal success later.
Tasks on the backend todo:
-
update state of overall import - once we mark it as finished we flag it as partially complete. This happens at the end of the import. -
add event tracking at the end of the import before it's updated.
Proposed solution
-
Add a has_failures
flag to thebulk_import_entities
table -
Set the flag to true in the BulkImports::Common::Pipelines::EntityFinisher
if errors occurred when importing the entity -
Update the metrics implemented in #341506 (closed) to use the has_failures
flag -
As with finishing this issue, the information shown to the user could still be misleading (import marked as fully completed while actually partially completed, because of relations that failed to be exported), we should document this and link to this documentation from UI.