Skip to content

Add application limits for Jira imports

Bojan Marjanovic requested to merge 244859-add-application-limits-for-jira-imports into master

What does this MR do and why?

When we do a Jira import, we can generate a lot of cache keys. gitlab-com/gl-infra/scalability#419 (comment 381941081) shows that we had ~760,000 keys in the cache at one point while a large import was running.

We want to pause the import worker as per comment.

This MR adds pause import worker and a retry mechanism:

  • if the issue import queue reaches 100K workers, then we pause the import until we reach 50K and resume from the page we left;
  • every 5 seconds we will check the size of the queue;
  • we have 10 retries (to avoid indefinitely pause).

If the number of retries drops to 0, the import with fail, and we will track the RetriesExceededError.

Screenshots or screen recordings

How to set up and validate locally

  1. Checkout the current branch
  2. Enable JIRA integration on one of the projects
  3. Set the local Jira import limits with suitable values for testing
    • JIRA_IMPORT_THRESHOLD
    • JIRA_IMPORT_PAUSE_LIMIT
  4. Visit the issues page
  5. Click on Import from Jira button
  6. Select one of the projects with more issues than the limit you set on the step 3.
  7. Proceed with the import.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #244859 (closed)

Edited by Bojan Marjanovic

Merge request reports