Problems with Git garbage collection on gitlab.com
I'm aware of one problem and one anomaly regarding Git garbage collection on gitlab.com. Maybe they are isolated but I'm going to throw them together in this issue for now.
Problem: "Run housekeeping" button does nothing
Edit 2019-06-14: this is solved, it's an application bug. https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6960#note_181267414
How to reproduce: I go to Project Settings (project/edit), click 'Advanced', then 'Run housekeeping'.
Expected: within a few minutes I see a log entry on log.gitlab.net, in the pubsub-gitaly-inf-gprd index, for the GarbageCollect RPC for the given repository.
Actual: nothing happens. Outside effects suggest nothing indeed happened, and this is not a logging problem. E.g. gitlab-org/gitaly#1728 (closed); that issue should have gone away after housekeeping and it did not. All signs point to these manually triggered housekeeping jobs not running in the first place.
Anomaly: the GarbageCollect Gitaly RPC is running 100x as often as expected
Expected behavior: Git pushes increment counters, which trigger the GarbageCollect RPC once every 200 pushes. The related RepackIncremental and RepackFull RPC's run at slight higher frequencies, in the same order of magnitude. See this test:
(Users can also trigger GarbageCollect manually but that appears to not work; see above.)
Actual behavior: GarbageCollect runs about 10/s, RepackIncremental 1/s, RepackFull 0.1/s. So GarbageCollect runs about 100x as often as expected.
What is going on here?