Skip to content

Creating timeout strategies for external imports

Max Fan requested to merge 422979-add-AdvanceStageWorker-unsticking-logic into master

What does this MR do and why?

Currently if the stages on advanceWorker get stuck. The job will re-queue itself for 24 hours until it eventually gets killed by the StuckProjectImportJobsWorker.

This MR changes this so after a 2 hours timeout with no changes in its dependent workers, the AdvanceStageWorker will either go directly to the next stage ("optimistic"). Or fail the import pre-emptively.

This ("pessmisitic") approach is the default, and is used for all imports using the UI/FE. If an import is done via API, the user will have the option to set the timeout_strategy: optimistic

Screenshots or screen recordings

n/a for the imports as they should behave the same

But this will create additional log statements in the shape of

{"feature_category":"importers","severity":"INFO","time":"2023-09-13T22:22:00.347Z","correlation_id":"01HA8A4SJQ5PN1JA2C764B9NQ4","meta.caller_id":"Gitlab::GithubImport::AdvanceStageWorker","meta.remote_ip":"172.16.123.1","meta.feature_category":"importers","meta.user":"MaxUser2","meta.user_id":55,"meta.project":"MaxUser2/publicImagesRepo7","meta.root_namespace":"MaxUser2","meta.client_id":"user/55","meta.root_caller_id":"Import::GithubController#create","message":"Timeout reached, no longer retrying","project_id":141,"jobs_remaining":0,"waiters":{},"timeout_strategy":"optimistic"}

{"feature_category":"importers","severity":"ERROR","time":"2023-09-12T00:53:58.000Z","correlation_id":"c8a5029ced8f729a6edb379f19a18374","import_type":null,"project_id":58,"source":"Gitlab::BitbucketServerImport::AdvanceStageWorker","message":"importer failed","error.message":"Failing advance stage, timeout reached with pessimistic strategy"}

How to set up and validate locally

  1. Since this is a timeout problem, it's not easy to reproduce locally. You'll have to
  2. Clone the branch
    1. In advance_stage.rb under timeout_reached? -> return true ;
    2. In advance_stage.rb change if new_waiters.empty? -> if new_waiters.empty? && false - since we don't want this to trigger
  3. Triggering Jira import/GitHub import/Bit bucket import works, and you can see the import messages in log/importer.log
  4. Triggering Github or BitBucket server imports with the API and using timeout_strategy: "optimistic" should move the import to the next stage

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #422979 (closed)

Edited by Max Fan

Merge request reports