Pre-import failures due to unknown tags

Context

Found during gitlab#350920 (closed).

Problem

We're seeing a very small number of pre-imports failing due to unknown tags. So far we've seen 3 occurrences:

gitlabhq_registry=> select * from repositories where migration_error like 'pre importing tagged manifests: reading tag%';
-[ RECORD 1 ]----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------
id                     | 440633
top_level_namespace_id | 3307
parent_id              |
created_at             | 2022-04-26 21:18:03.321702+00
updated_at             | 2022-04-26 21:18:38.60843+00
migration_status       | pre_import_failed
deleted_at             |
migration_error        | pre importing tagged manifests: reading tag "develop-e9942248" from filesystem: unknown tag=develop-e9942248
-[ RECORD 2 ]----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------
id                     | 444564
top_level_namespace_id | 7459
parent_id              |
created_at             | 2022-04-27 06:10:58.065238+00
updated_at             | 2022-04-27 06:12:05.061012+00
migration_status       | pre_import_failed
deleted_at             |
migration_error        | pre importing tagged manifests: reading tag "184660772" from filesystem: unknown tag=184660772
-[ RECORD 3 ]----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------
id                     | 430377
top_level_namespace_id | 14032
parent_id              |
created_at             | 2022-04-25 15:05:18.086542+00
updated_at             | 2022-04-25 15:05:21.070477+00
migration_status       | pre_import_failed
deleted_at             |
migration_error        | pre importing tagged manifests: reading tag "07a2b3d9b5b7948b5043e8f0f5c3625f1539fad0" from filesystem: unknown tag=07a2b3d9b5b7948b5043e8f0f5c3625f1539fad0

Sample logs screenshot:

Screen_Shot_2022-04-27_at_10.08.47

All these pre-imports were retried twice and eventually skipped (as expected). It's unclear how can a tag show up on the tag list request done by the importer but then cause this "unknown tag" error when it's time to process it.

Unlike a few other observed occurrences (related to gitlab#360652 (closed)), these tags were not deleted while the pre-import was running, so this is most likely a data corruption on the old code path.

Proposal

Look at the GCS storage backend (old code path) to find out what's the status of these tags. Then identify why they are found on the tag list request but not when we try to process them in the importer. Finally, develop a fix so that these are gracefully handled.

Conclusion

After looking at these samples, we can see that they occurred due to tags being deleted between the tag list operation and the "get tag details" that follows. The solution is to skip tags that no longer exist during the latter.

Edited by João Pereira