It appears that `git fetch` no longer works on gitlab-ce project

Summary

At approximately 23:00 UTC - we were notified of issues pulling and fetching gitlab-ce and have been investigating. It looks like the errors are related to work on #1096 (closed) which may have garbage collected objects. The objects that were causing most of the errors were restored and the engineers working on 1096 will continue investigation into the issues.

Current state - operational.

Service(s) affected : git

Team attribution :

Minutes downtime or degradation :

Timeline

2019-08-28

  • 21:59 UTC - moved the refs/dangling directory out of the way for gitlab-org/gitlab-ce
  • 22:05 UTC - drained canary: web-cny-01-sv-gprd, web-cny-02-sv-gprd, git-cny-01-sv-gprd, git-cny-02-sv-gprd, api-cny-01-sv-gprd, api-cny-02-sv-gprd
  • 22:16 UTC - file-08 is really slow right now
  • 22:17 UTC - Gitaly was pegging all cores on file-08 for at least minute or two maybe more; then went quiet for some time
  • 22:29 UTC - reports incoming that git fetch fails silently and git push -f fails loudly
  • 22:44 UTC - Gitaly is pegging all cores on file-08 again

  • 22:46 UTC - Gitaly has quieted down again
  • 23:04 UTC - restoring a backup to see if we can restore some of the error objects
  • 23:33 UTC - packed objects copied from a snapshot gitlab-ee to file-08
  • 23:34 UTC - restoring from a disk snapshot of file-08 has brought back most of the erroring objects. Users should be able to push and pull on Gitlab-ce
  • 23:44 UTC - push, pull and fetch are now all working on GitLab-CE
  • 23:50 UTC - umounted /dev/sdc from /mnt/issue-recovery on console-01-sv-gprd.c.gitlab-production.internal ([2020084.726017] EXT4-fs (sdc): mounted filesystem with ordered data mode.)
  • 23:51 UTC - detatched virtual disk issue-1097 (https://console.cloud.google.com/compute/disksDetail/zones/us-east1-c/disks/issue-1097?project=gitlab-production) from instance console-01-sv-gprd (https://console.cloud.google.com/compute/instancesDetail/zones/us-east1-c/instances/console-01-sv-gprd?project=gitlab-production)
  • 23:54 UTC - deleted virtual disk issue-1097

Possibly related issue: #1096 (closed)

Edited by Stan Hu