Provide a better way to monitor/reset the state of mirror scheduling
In the past week, RepositoryUpdateMirror
jobs ran away to 500,000 jobs in the queue. There's no good way to drop this queue and reset the state of the worker.
I don't like the idea of a button to do this, but the current mode of resetting the state of the mirrors is not a good one. This is what we have to do in production on the Rails console:
curl -O /tmp/sq.rb https://gitlab.com/gitlab-com/runbooks/raw/master/troubleshooting/db_scripts/sq.rb
load '/tmp/sq.rb'
PULL_CAPACITY_KEY = 'MIRROR_PULL_CAPACITY'
100.times { kill_jobs_by_worker_name(OpenStruct.new(dry_run: false), 'RepositoryUpdateMirrorWorker'); sleep 0.01 }
Gitlab::Redis.with { |redis| redis.del(PULL_CAPACITY_KEY, projects) }
/cc: @tiagonbotelho, @DouweM, @pcarranza
Edited by Stan Hu