Creating timeout strategies for external imports
What does this MR do and why?
Currently if the stages on advanceWorker get stuck. The job will re-queue itself for 24 hours until it eventually gets killed by the StuckProjectImportJobsWorker.
This MR changes this so after a 2 hours timeout with no changes in its dependent workers, the AdvanceStageWorker will either go directly to the next stage ("optimistic"). Or fail the import pre-emptively.
This ("pessmisitic") approach is the default, and is used for all imports using the UI/FE. If an import is done via API, the user will have the option to set the timeout_strategy: optimistic
Screenshots or screen recordings
n/a for the imports as they should behave the same
But this will create additional log statements in the shape of
{"feature_category":"importers","severity":"INFO","time":"2023-09-13T22:22:00.347Z","correlation_id":"01HA8A4SJQ5PN1JA2C764B9NQ4","meta.caller_id":"Gitlab::GithubImport::AdvanceStageWorker","meta.remote_ip":"172.16.123.1","meta.feature_category":"importers","meta.user":"MaxUser2","meta.user_id":55,"meta.project":"MaxUser2/publicImagesRepo7","meta.root_namespace":"MaxUser2","meta.client_id":"user/55","meta.root_caller_id":"Import::GithubController#create","message":"Timeout reached, no longer retrying","project_id":141,"jobs_remaining":0,"waiters":{},"timeout_strategy":"optimistic"}
{"feature_category":"importers","severity":"ERROR","time":"2023-09-12T00:53:58.000Z","correlation_id":"c8a5029ced8f729a6edb379f19a18374","import_type":null,"project_id":58,"source":"Gitlab::BitbucketServerImport::AdvanceStageWorker","message":"importer failed","error.message":"Failing advance stage, timeout reached with pessimistic strategy"}
How to set up and validate locally
- Since this is a timeout problem, it's not easy to reproduce locally. You'll have to
- Clone the branch
- In
advance_stage.rb
undertimeout_reached? -> return true
; - In
advance_stage.rb
changeif new_waiters.empty? -> if new_waiters.empty? && false
- since we don't want this to trigger
- In
- Triggering Jira import/GitHub import/Bit bucket import works, and you can see the import messages in
log/importer.log
- Triggering Github or BitBucket server imports with the API and using
timeout_strategy: "optimistic"
should move the import to the next stage
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #422979 (closed)