Skip to content

Prevent Active Bitbucket Server/Cloud imports from being marked as stuck

Sam Word requested to merge 434144-prevent-active-bitbucket-server-import-from-being-marked-as-stuck into master

What does this MR do and why?

This MR adds RefreshImportWorker to Bitbucket Cloud and Server importer stage workers so that their job ids are refreshed when a stage begins. This will prevent StuckImportWroker from mistakenly thinking a particularly import is stuck. Now that RefreshImportWorker is called within the BitbucketImport and BitbucketServerImport namespaces (i.e. outside the GitHub importer), RefreshImportWorker was moved to a generic Import namespace.

The related issues also proposed a second proposed solution to call the refresh_jid_expiration "Every time AdvanceWorkers executes and the number of pending jobs decreases." This seems to already be done: advance_stage.rb#L44-49.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

No relevant UI changes

How to set up and validate locally

It's a little difficult to simulate a successful Bitbucket import that takes longer than 24 hours to complete, so here's what I did to recreate the issue as best as I could locally:

  1. Set StuckImportJob's IMPORT_JOBS_EXPIRATION to something relatively short: IMPORT_JOBS_EXPIRATION = 10.minutes.seconds.to_i.
  2. Make some portion of the Bitbucket server importer sleep for longer than your new IMPORT_JOBS_EXPIRATION. For example, add sleep(20.minutes) to Gitlab::BitbucketServerImport::Stage::ImportRepositoryWorker#import (filepath: app/workers/gitlab/bitbucket_server_import/stage/import_repository_worker.rb)
  3. Start your local server, navigate to the Import New Project path (/projects/new#import_project).
  4. Select Bitbucket Server import, and configure the Bitbucket server import settings. Gitlab has an instance of Bitbucket server with a test repository. See 1Password for login details and access tokens.
  5. Import the Bitbucket server repository so that the import can begin and work longer than IMPORT_JOBS_EXPIRATION, but not complete before the next scheduled StuckProjectImportJobsWorker can run. By default StuckProjectImportJobsWorker runs every hour on the 15 minute mark, but this can be changed in settings for local testing.
  6. Verify that the import completed, despite running longer than the status timeout and StuckProjectImportJobsWorker running.

Related to #434144 (closed) and #434145 (closed)

Edited by Sam Word

Merge request reports