Skip to content
GitLab
Next
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • GitLab GitLab
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 44,761
    • Issues 44,761
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 1,329
    • Merge requests 1,329
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Metrics
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.orgGitLab.org
  • GitLabGitLab
  • Issues
  • #340630
Closed
Open
Issue created Apr 16, 2021 by Bob Van Landuyt@reprazentMaintainer

Make repository pull mirroring not depend on sidekiq queue sizes

Follow up from: gitlab-com/gl-infra/scalability#995 (closed)

At the time of writing, the UpdateAllMirrorsWorker uses the queue size of the ProjectImportScheduleWorker before possibly rescheduling itself.

I believe this is done to spread the load out a bit more, making sure we don't schedule more ProjectImportScheduleWorkers and the following RepositoryUpdateMirrorWorkers than we can chew.

In gitlab-com/gl-infra&447 (closed) we're moving from having a queue per worker, to jobs for multiple workers being scheduled on a single queue.

This means that checking the queue size would no longer be valid for checking if the UpdateAllMirrorsWorker can continue scheduling jobs.

In the short term, we're keeping the ProjectImportScheduleWorker on its own queue so this mechanism stays correct. But ideally we'd figure out a better way to do this.

Assignee
Assign to
Time tracking