9.3.1 introduced a change where changing project.import_status from started to finished results in housekeeping being performed. This in turn seems to have lead to GitLab.com ending up with 20 000 jobs in the git_garbage_collect queue, leading me to believe this is triggered every time we update a mirror since mirroring re-uses this state.
This is really problematic because it means updating a mirror will result in a job being scheduled. I looked at the queue and for many jobs housekeeping isn't even necessary.
Then there's the method Project#perform_housekeeping:
defperform_housekeepingreturnunlessrepo_exists?run_after_commitdobeginProjects::HousekeepingService.new(self).executerescueProjects::HousekeepingService::LeaseTaken=>eRails.logger.info("Could not perform housekeeping for project #{self.path_with_namespace} (#{self.id}): #{e}")endendend
Since mirroring also changes import_status this means that both imports and mirrors will end up calling perform_housekepeing, even when not necessary. The patch I shared above should fix this problem.
If we're calling needed? here, we should try to ensure that returns true when an import of a fresh project is completed, but if that's not possible, let @dbalexandre know.