Skip to content

Increased incidents of inconsistent branch cache/empty repository state

Over the last 24 hours, we have seen an increased number of reports of two issues that may be similar in nature:

  1. "Source branch does not exist" (https://gitlab.com/gitlab-org/gitlab-ce/issues/15392) when it does exist

    • This is a issue where the Repository#branch_names appears to omit the newly-created branch. All other branches appear to be present.
  2. Repository appears empty when it is not

    When I went to check the state of one of the reported repos, it appears the Repository #empty? state was wrong:

    irb(main):006:0> proj.repository.exists?
    => true
    irb(main):007:0> proj.repository.has_visible_content?
    => false
    irb(main):014:0> proj.repository.empty?
    => true
    irb(main):013:0> proj.repository.branch_names
    => ["master"]

In both cases, these incidents occurred after a push. Initially, I suspected some issue with CephFS, but it may be occurring because git gc may be interfering with the state of the repo. We know from https://gitlab.com/gitlab-org/gitlab-ce/issues/15392 there is an issue that git gc and Rugged do not always play well together. More specifically, if git gc puts loose refs into the packed-refs file, in some cases Rugged::Repository may not pick up the change.

I was able to reproduce the second issue this morning:

  1. Created a blank repo
  2. Imported the www-gitlab-com repo
  3. After a while, I saw the "The repository for this project does not exist." message (from the projects/no_repo partial).
  4. I clicked on " Create empty bare repository"
  5. The repository appeared in its full glory

@markglenfletcher also mentioned that he was able to reproduce the second problem.

RC6 was deployed yesterday around 2 am UTC time. The one change that may have affected this, gitlab-org/gitlab-ce!6326, was shipped and deployed in RC3, so it's a bit odd that the problem would suddenly start showing up more now.

To gather more information, today @yorickpeterse disabled git gc via the patch:

diff --git a/app/services/projects/housekeeping_service.rb b/app/services/projects/housekeeping_service.rb
index c3dfc8c..5be9ad1 100644
--- a/app/services/projects/housekeeping_service.rb
+++ b/app/services/projects/housekeeping_service.rb
@@ -26,6 +26,7 @@ module Projects
     end
 
     def needed?
+      return false # DISABLED AS A TEST
       @project.pushes_since_gc >= 10
     end

We also cleared the cache so that any affected repos should have a consistent state on next load.

/cc: @pcarranza, @northrup, @jacobvosmaer-gitlab, @rdavila