Skip to content

Queue workers only for objects with attachments

What does this MR do and why?

Queue GitHub attachment importer workers only for objects with attachments

During a GitHub import, when we queue a worker we spread them to run 1,000 per minute. The more workers queued, the longer the spread across time.

Because most workers will do nothing, this spread adds unnecessary time to an import. The larger the import, the more time is wasted in this stage.

This change will queue workers only for objects that have attachments.

#436400 (closed)

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

  1. In rails console enable the feature:
    Feature.enable(:github_importer_attachments)
  2. Tail your Sidekiq logs for GitHub import attachment workers: tail -f sidekiq_client.log | grep "GithubImport::Attachments::".
  3. Ensure GitHub is enabled as an import source for your instance (docs).
  4. Create a GitHub project:
    1. Add 1 issue with just a text description
    2. Add 1 more issue with an image attachment in its description. Add comments to this issue:
      1. Add 1 comment with an attachment in its description
      2. Add 1 comment which is just text
  5. Import the GitHub project (docs). Ensure that the Import Markdown attachments (links) UI option is checked.
  6. Expand the Details section to watch the progress of the import. You should only see counts of Issue links, Note links, or PR attachments that match the counts of actual attachments. On master these counts will match the number of issues, notes and PRs regardless of whether they have attachments of note.
  7. Ensure that the project's attachments are imported:
    1. Go to the imported project and right-click on image attachments to verify they are hosted locally and not on github.com.

Related to #436400 (closed)

Edited by Luke Duncalfe

Merge request reports