Skip to content

GithubImporter: Ensure to fail and log imports on exceptions

What does this MR do?

Currently, the Github Importer might end up stuck even after failing to import the git repository. This might also happen when importing other types of objects from Github.

To provide better feedback to the end user this kind of exception will now mark the import (import_state) as failed. Also, to provide information to developers and testers the exceptions errors will be logged (importer.log) and saved in the import_failures table.


What was done

  • Created Gitlab::Import::ImportFailureService: This way we now have a standard way to log exceptions/errors that happens while importing a project;
    • All the exceptions/errors are logged to the importer.log with a standard format;
    • All the exceptions/errors are tracked in Gitlab::ErrorTracking;
    • All the exceptions/errors are recorded in the import_failures table;
    • When required, the exception/error might mark the whole import as failed;
    • This was created in a more generic way because it can later be used in the other importers (bitbucket, gitlab, etc);
  • Updated Github Importer workers/importers to leverage of the Gitlab::Import::ImportFailureService;
  • Updated Gitlab::GithubImport::Queue, to use Gitlab::Import::ImportFailureService when jobs exhausted retries;
  • Updated Gitlab::Import::StuckImportJob to use Gitlab::Import::ImportFailureService;
    • This is also used in the Jira importer;

Related to: #335660 (closed)

Screenshots or Screencasts (strongly suggested)

How to setup and validate locally (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Kassio Borges

Merge request reports