Skip to content
GitLab Next
  • Menu
Projects Groups Snippets
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • GitLab GitLab
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 43,843
    • Issues 43,843
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 1,399
    • Merge requests 1,399
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages & Registries
    • Packages & Registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Metrics
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.org
  • GitLabGitLab
  • Issues
  • #340630
Closed
Open
Created Apr 16, 2021 by Bob Van Landuyt@reprazent🔴Maintainer

Make repository pull mirroring not depend on sidekiq queue sizes

Follow up from: gitlab-com/gl-infra/scalability#995 (closed)

At the time of writing, the UpdateAllMirrorsWorker uses the queue size of the ProjectImportScheduleWorker before possibly rescheduling itself.

I believe this is done to spread the load out a bit more, making sure we don't schedule more ProjectImportScheduleWorkers and the following RepositoryUpdateMirrorWorkers than we can chew.

In gitlab-com/gl-infra&447 (closed) we're moving from having a queue per worker, to jobs for multiple workers being scheduled on a single queue.

This means that checking the queue size would no longer be valid for checking if the UpdateAllMirrorsWorker can continue scheduling jobs.

In the short term, we're keeping the ProjectImportScheduleWorker on its own queue so this mechanism stays correct. But ideally we'd figure out a better way to do this.

Assignee
Assign to
Time tracking