Skip to content

Add feature flag to the lock mechanism added to the UserFinder

Rodrigo Tomonari requested to merge rodrigo/add-feature-flag-to-user-finder-on-github-import into master

What does this MR do and why?

Add feature flag to the lock mechanism added to the GitHub Import's UserFinder on !140051 (merged).

The recent update has improved the lock method by using the ETAG. Previously, even with the lock, user_finder was making multiple requests for users without an email set.

Now, user_finder will still make the initial request, but subsequent requests will utilize the ETAG, which won't count towards the rate limit.

Related to: !140051 (merged) and #435340 (closed)

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

  1. Enable the feature flag github_import_lock_user_finder
  2. Clear the log/importer.log.
echo -n > log/importer.log
  1. Delete UserFinder cache
Gitlab::Redis::SharedState.with do |redis|
  key_prefix = 'cache:gitlab:github-import/user-finder/'

  cursor = '0'
  begin
    cursor, keys = redis.scan(cursor, match: "#{key_prefix}*")

    keys.each do |key|
      redis.del(key)
      puts "Deleted key: #{key}"
    end
  end while cursor != '0'
end
  1. Restart Sidekiq
  2. Trigger a GitHub Import migration
curl --location 'http://gdk.test:3000/api/v4/import/github' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer ACCESS_TOKEN' \
--data '{
    "personal_access_token": "GITHUB_ACCESS_TOKEN",
    "repo_id": "8514", 
    "target_namespace": "root",
    "new_name": "Rails",
    "optional_stages": {
      "single_endpoint_issue_events_import": true,
      "single_endpoint_notes_import": true,
      "attachments_import": false,
      "collaborators_import": false
    },
    "timeout_strategy": "optimistic"
}'
  1. Wait for some records to be migrated
  2. Use the command below to count how many times each user's email was fetched. We should only see the username only once
grep '"Fetching email from GitHub"' log/importer.log | jq .username | sort | uniq -c | sort -g
Edited by Rodrigo Tomonari

Merge request reports