Increased incidents of inconsistent branch cache/empty repository state
Over the last 24 hours, we have seen an increased number of reports of two issues that may be similar in nature:
-
"Source branch does not exist" (https://gitlab.com/gitlab-org/gitlab-ce/issues/15392) when it does exist
- This is a issue where the
Repository#branch_names
appears to omit the newly-created branch. All other branches appear to be present.
- This is a issue where the
-
Repository appears empty when it is not
- https://gitlab.com/gitlab-com/support-forum/issues/1073
- https://gitlab.com/gitlab-com/support-forum/issues/1066
When I went to check the state of one of the reported repos, it appears the
Repository #empty?
state was wrong:irb(main):006:0> proj.repository.exists? => true irb(main):007:0> proj.repository.has_visible_content? => false irb(main):014:0> proj.repository.empty? => true irb(main):013:0> proj.repository.branch_names => ["master"]
In both cases, these incidents occurred after a push. Initially, I suspected some issue with CephFS, but it may be occurring because git gc
may be interfering with the state of the repo. We know from https://gitlab.com/gitlab-org/gitlab-ce/issues/15392 there is an issue that git gc
and Rugged do not always play well together. More specifically, if git gc
puts loose refs into the packed-refs file, in some cases Rugged::Repository
may not pick up the change.
I was able to reproduce the second issue this morning:
- Created a blank repo
- Imported the www-gitlab-com repo
- After a while, I saw the "The repository for this project does not exist." message (from the
projects/no_repo
partial). - I clicked on " Create empty bare repository"
- The repository appeared in its full glory
@markglenfletcher also mentioned that he was able to reproduce the second problem.
RC6 was deployed yesterday around 2 am UTC time. The one change that may have affected this, gitlab-org/gitlab-ce!6326, was shipped and deployed in RC3, so it's a bit odd that the problem would suddenly start showing up more now.
To gather more information, today @yorickpeterse disabled git gc
via the patch:
diff --git a/app/services/projects/housekeeping_service.rb b/app/services/projects/housekeeping_service.rb
index c3dfc8c..5be9ad1 100644
--- a/app/services/projects/housekeeping_service.rb
+++ b/app/services/projects/housekeeping_service.rb
@@ -26,6 +26,7 @@ module Projects
end
def needed?
+ return false # DISABLED AS A TEST
@project.pushes_since_gc >= 10
end
We also cleared the cache so that any affected repos should have a consistent state on next load.