Skip to content

Adjust Bitbucket Cloud issues worker to be resumable

  • Please check this box if this contribution uses AI-generated content (including content generated by GitLab Duo features) as outlined in the GitLab DCO & CLA

What does this MR do and why?

Adjust Bitbucket Cloud issues worker to be resumable. This is a continuation from this MR, but for "Issue Worker".

This MR also change the feature flag from wip to beta. Quoting from the doc.

Once the feature is complete, the feature flag type can be changed to the gitlab_com_derisk or beta type depending on how the feature will be presented/documented to customers.

There are 2 more components to be worked on:

  • pull_requests_notes
  • issues_notes

But both of them need a refactoring to move the API call from "Importer Worker" to "Stage Worker", similar with this issue.

Changelog: performance

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After
Successful imported issues:
Screenshot_2024-07-07_at_21.14.00
Interruption happened after issue title issue 40:
Screenshot_2024-07-07_at_20.59.53
Upon resume, retrying issue title issue 40. Then continuing to MR title issue 41:
Screenshot_2024-07-07_at_21.03.47

How to set up and validate locally

  1. Setup Bitbucket Cloud following this guide

  2. Enable issue tracker by following this guide

  3. In rails console enable the feature flag

    Feature.enable(:bitbucket_import_resumable_worker)
  4. Patch the code to add interruption:

diff --git a/lib/gitlab/bitbucket_import/parallel_scheduling.rb b/lib/gitlab/bitbucket_import/parallel_scheduling.rb
index c4703c71da9a..7a89b2232a80 100644
--- a/lib/gitlab/bitbucket_import/parallel_scheduling.rb
+++ b/lib/gitlab/bitbucket_import/parallel_scheduling.rb
@@ -42,6 +42,16 @@ def each_object_to_import
         options = collection_options.merge(next_url: page_keyset.current)
 
         client.each_page(collection_method, representation_type, repo, options) do |page|
+          if collection_method == :issues
+            log_info(message: page.inspect)
+            Gitlab::Redis::SharedState.with do |redis|
+              temp_key = 'test-bitbucket-pr'
+              temp_counter = redis.incr(temp_key)
+              redis.expire(temp_key, 5.minutes)
+              raise "purposely interrupt" if temp_counter == 2
+            end
+          end
+
           page.items.each do |object|
             job_waiter.jobs_remaining = Gitlab::Cache::Import::Caching.increment(job_waiter_remaining_cache_key)
  1. Visit Bitbucket Cloud import page http://127.0.0.1:3000/import/bitbucket/status
  2. Click the Import button.
  3. Tail the log file: tail -f log/importer.log

Related to #448895

Edited by Ivan Sebastian

Merge request reports