Commit list returns 404 on branches whose names contain forward slashes

Summary

With the upgrade to GitLab 11.9 the Haskell installation (#55039 (moved)) began blowing up in various ways when faced with branches whose names contain forward slashes:

• Merge requests whose source branch name contains a forward slash suggested that the source branch did not exist ("source branch does not exist. Please restore it or use a different source branch") despite the fact that the pages list most certainly did exist
• Viewing the commits list (e.g. https://gitlab.haskell.org/ghc/ghc/commits/wip/lint) of a branch whose name contains a forward slash returned a 404

Steps to reproduce

Push a branch whose name contains a forward slash (e.g. wip/lint).

Example Project

Unfortunately I have been unable to reproduce this on gitlab.com.

What is the current bug behavior?

Various things break, as described above.

What is the expected correct behavior?

Things don't blow up.

Relevant logs and/or screenshots

See "possible fixes" section below.

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:
Proxy:          no
Current User:   root
Using RVM:      no
Ruby Version:   unknown
Gem Version:
Bundler Version:1.17.2
Rake Version:   12.3.2
Redis Version:  unknown
Git Version:    2.19.2
Sidekiq Version:5.2.5
Go Version:     unknown
GitLab information
Version:        11.9.0-ee
Revision:       Unknown
Directory:      /nix/store/wlxwq8g78cmiy0qca6zqa089qjj91940-gitlab-ee-11.9.0/share/gitlab
DB Adapter:     postgresql
DB Version:     9.6.11
URL:            https://gitlab.haskell.org
HTTP Clone URL: https://gitlab.haskell.org/some-group/some-project.git
SSH Clone URL:  git@gitlab.haskell.org:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: github
GitLab Shell
Version:        8.7.1
Repository storage paths:

• default: /var/gitlab/state/repositories GitLab Shell path: /nix/store/yfy2wr01mv6j5g86as6h0jdx1ggjcnqp-gitlab-shell-8.7.1 Git: git

Results of GitLab application Check

Expand for output related to the GitLab application check

    Checking GitLab subtasks ...                      


Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 8.7.1 ? ... OK (8.7.1)
Running /nix/store/yfy2wr01mv6j5g86as6h0jdx1ggjcnqp-gitlab-shell-8.7.1/bin/check
Check GitLab API access: OK

Redis available via internal API: OK
Access to /var/gitlab/state/home/.ssh/authorized_keys: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... no
Try fixing it:
sudo -u git -H RAILS_ENV=production bin/background_jobs start
For more information see:
doc/install/installation.md in section "Install Init Script"
see log/sidekiq.log for possible errors
Please fix the error above and rerun the checks.
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... no
Try fixing it:
sudo chmod 700 /var/gitlab/state/uploads
For more information see:
doc/install/installation.md in section "GitLab"
Please fix the error above and rerun the checks.
Uploads directory tmp has correct permissions? ... no
Try fixing it:
sudo chown -R git /var/gitlab/state/uploads
sudo find /var/gitlab/state/uploads -type f -exec chmod 0644 {} ;
sudo find /var/gitlab/state/uploads -type d -not -path /var/gitlab/state/uploads -exec chmod 0700 {} ;
For more information see:
doc/install/installation.md in section "GitLab"
Please fix the error above and rerun the checks.
Init script exists? ... no
Try fixing it:
Install the init script
For more information see:
doc/install/installation.md in section "Install Init Script"
Please fix the error above and rerun the checks.
Init script up-to-date? ... can't check because of previous errors
Projects have namespace: ...
Glasgow Haskell Compiler / GHC ... yes
Glasgow Haskell Compiler / libffi-tarballs ... yes
Glasgow Haskell Compiler / gmp-tarballs ... yes
Glasgow Haskell Compiler / arcanist-external-json-linter ... yes
Glasgow Haskell Compiler / hsc2hs ... yes
Glasgow Haskell Compiler / haddock ... yes
Glasgow Haskell Compiler / nofib ... yes
Glasgow Haskell Compiler / Packages / array ... yes
Glasgow Haskell Compiler / Packages / binary ... yes
Glasgow Haskell Compiler / Packages / bytestring ... yes
Glasgow Haskell Compiler / Packages / Cabal ... yes
Glasgow Haskell Compiler / Packages / containers ... yes
Glasgow Haskell Compiler / Packages / deepseq ... yes
Glasgow Haskell Compiler / Packages / directory ... yes
Glasgow Haskell Compiler / Packages / filepath ... yes
Glasgow Haskell Compiler / Packages / haskeline ... yes
Glasgow Haskell Compiler / Packages / hpc ... yes
Glasgow Haskell Compiler / Packages / mtl ... yes
Glasgow Haskell Compiler / Packages / parsec ... yes
Glasgow Haskell Compiler / Packages / pretty ... yes
Glasgow Haskell Compiler / Packages / process ... yes
Glasgow Haskell Compiler / Packages / terminfo ... yes
Glasgow Haskell Compiler / Packages / text ... yes
Glasgow Haskell Compiler / Packages / time ... yes
Glasgow Haskell Compiler / Packages / transformers ... yes
Glasgow Haskell Compiler / Packages / unix ... yes
Glasgow Haskell Compiler / Packages / Win32 ... yes
Glasgow Haskell Compiler / Packages / xhtml ... yes
Glasgow Haskell Compiler / Packages / parallel ... yes
Glasgow Haskell Compiler / Packages / stm ... yes
Ben Gamari / Glasgow Haskell Compiler ... yes
Haskell / array ... yes
Mr Kerckhove / Glasgow Haskell Compiler ... yes
Gabor Greif / Glasgow Haskell Compiler ... yes
Peter Trommler / Glasgow Haskell Compiler ... yes
Zejun Wu / Glasgow Haskell Compiler ... yes
Roland Senn / Glasgow Haskell Compiler ... yes
Matthew Pickering / Glasgow Haskell Compiler ... yes
David Eichmann / Glasgow Haskell Compiler ... yes
Carter Schonwald / Glasgow Haskell Compiler ... yes
Alec Theriault / Glasgow Haskell Compiler ... yes
Dmitry Ivanov / Glasgow Haskell Compiler ... yes
Dejan Mijić / Glasgow Haskell Compiler ... yes
Adam Sandberg Eriksson / Glasgow Haskell Compiler ... yes
Alexander Biehl / Glasgow Haskell Compiler ... yes
Eric Seidel / Glasgow Haskell Compiler ... yes
Sergei Trofimovich / Glasgow Haskell Compiler ... yes
Sven Tennie / Glasgow Haskell Compiler ... yes
Thi H / Glasgow Haskell Compiler ... yes
Glasgow Haskell Compiler / Hadrian ... yes
Piyush P Kurur / Glasgow Haskell Compiler ... yes
cnd / Shake.it.off ... yes
mrchurrisky / haskell_projects ... yes
Artem Pelenitsyn / Glasgow Haskell Compiler ... yes
Andreas Klebinger / Glasgow Haskell Compiler ... yes
Geoffrey Huntley / Glasgow Haskell Compiler ... yes
Shayne Fletcher / Glasgow Haskell Compiler ... yes
Francesco Gazzetta / Glasgow Haskell Compiler ... yes
Ara Adkins / Glasgow Haskell Compiler ... yes
Iavor S. Diatchki / Glasgow Haskell Compiler ... yes
Ömer Sinan Ağacan / Glasgow Haskell Compiler ... yes
Alan Mock / Glasgow Haskell Compiler ... yes
Susan Potter / Glasgow Haskell Compiler ... yes
Sebastian Graf / nofib ... yes
Sasa Bogicevic / Glasgow Haskell Compiler ... yes
Julian Leviston / Glasgow Haskell Compiler ... yes
Simon Hafner / Glasgow Haskell Compiler ... yes
Richard Eisenberg / GHC ... yes
Richard Eisenberg / haddock ... yes
Takenobu Tani / Glasgow Haskell Compiler ... yes
Glasgow Haskell Compiler / head.hackage ... yes
Michal Terepeta / Glasgow Haskell Compiler ... yes
Andrew Martin / Glasgow Haskell Compiler ... yes
Sylvain Henry / Glasgow Haskell Compiler ... yes
scottgw / Glasgow Haskell Compiler ... yes
Simon Jakobi / Glasgow Haskell Compiler ... yes
Moritz Angermann / Glasgow Haskell Compiler ... yes
Simon Jakobi / haddock ... yes
Ryan Scott / Glasgow Haskell Compiler ... yes
Ryan Scott / haddock ... yes
Herbert Valerio Riedel / GHC ... yes
howtonotwin / GHC ... yes
chessai / GHC ... yes
Matt Renaud / GHC ... yes
Ben Price / GHC ... yes
Steve Hart / GHC ... yes
Glasgow Haskell Compiler / Packages / old-locale ... yes
Glasgow Haskell Compiler / Packages / dph ... yes
Glasgow Haskell Compiler / Packages / old-time ... yes
Glasgow Haskell Compiler / Packages / haskell2010 ... yes
Glasgow Haskell Compiler / Packages / haskell98 ... yes
Glasgow Haskell Compiler / Packages / base ... yes
Glasgow Haskell Compiler / Packages / integer-gmp ... yes
Glasgow Haskell Compiler / Packages / ghc-prim ... yes
Glasgow Haskell Compiler / Packages / template-haskell ... yes
Glasgow Haskell Compiler / Packages / integer-simple ... yes
Glasgow Haskell Compiler / ghc-tarballs ... yes
Glasgow Haskell Compiler / Packages / utf8-string ... yes
Glasgow Haskell Compiler / Packages / hoopl ... yes
Glasgow Haskell Compiler / Packages / primitive ... yes
Glasgow Haskell Compiler / Packages / random ... yes
Glasgow Haskell Compiler / GHC Testsuite ... yes
Glasgow Haskell Compiler / Packages / vector ... yes
Glasgow Haskell Compiler / Packages / extensible-exceptions ... yes
Tao He / GHC ... yes
kgardas / GHC ... yes
David Eichmann / GHC-Perf-Notes-Test ... yes
Glasgow Haskell Compiler / GHC Performance Notes ... yes
Abhiroop Sarkar / nofib ... yes
Chaitanya Koparkar / hpc ... yes
Chaitanya Koparkar / GHC ... yes
Ben Gamari / ghc-utils ... yes
Alp Mestanogullari / GHC ... yes
Samuel Holland / GHC ... yes
Kazuki Okamoto / GHC ... yes
P.C. Shyamshankar / GHC ... yes
John Ky / GHC ... yes
Gabor Greif / nofib ... yes
toon nolten / GHC ... yes
Alex Dudarenko / GHC ... yes
Tamar Christina / GHC ... yes
Edward Z. Yang / GHC ... yes
tweag / GHC ... yes
Langston Barrett / GHC ... yes
Neil Mitchell / GHC ... yes
Alexandre R. Baldé / GHC ... yes
Artem Pyanykh / GHC ... yes
mniip / GHC ... yes
My Nguyen / GHC ... yes
My Nguyen / haddock ... yes
Alexandre Esteves / GHC ... yes
crockeea / GHC ... yes
Niklas Hambüchen / GHC ... yes
Jan van Brügge / GHC ... yes
Vaibhav Sagar / GHC ... yes
Oleg Grenrus / nofib ... yes
Oleg Grenrus / GHC ... yes
Glasgow Haskell Compiler / ci-images ... yes
Phil de Joux / GHC ... yes
Ben Gamari / test ... yes
Rinat Striungis / GHC ... yes
Pluralia / GHC ... yes
Trịnh Tuấn Phương / GHC ... yes
Sophie Taylor / GHC ... yes
Ben Gamari / gitlab-migration ... yes
Glasgow Haskell Compiler / homepage ... yes
Ben Gamari / trac-to-remarkup ... yes
Glasgow Haskell Compiler / marge-bot ... yes
Michael Peyton Jones / GHC ... yes
mgsloan / GHC ... yes
Henry Cohen / GHC ... yes
Travis Whitaker / GHC ... yes
Radosław Rowicki / GHC ... yes
Simon Marlow / GHC ... yes
dxmgnwma / Testo Drive 365 ... yes
Jens Petersen / homepage ... yes
Andreas Klebinger / nofib ... yes
Takenobu Tani / GHC wiki tracking ... yes
dero hereri / health ... yes
Alka Tone Keto Reviews / Alka Tone Keto Reviews  ... yes
Alka Tone Keto Reviews / Alka Tone Keto Shark Tank ... yes
Redis version >= 2.8.0? ... no
Try fixing it:
Update your redis server to a version >= 2.8.0
For more information see:
gitlab-public-wiki/wiki/Trouble-Shooting-Guide in section sidekiq
Please fix the error above and rerun the checks.
Ruby version >= 2.3.5 ? ... yes (2.5.3)
Git version >= 2.18.0 ? ... yes (2.19.2)
Git user has default SSH configuration? ... yes
Active users: ... 3555
Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes

I spent quite some time tracing this down. I managed to come up with a workaround but still can't claim to understand the root cause of the issue. I'll try to summarize my findings here.

I started by looking at the issue of the commit list page of the branch wip/lint returning a 404. It turned out that this was due to ExtractsPath#extract_ref incorrectly splitting the query into a ref of wip and a path of lint. Inspecting the implementation of this function suggested that this was due to the wip/lint ref not being present in the list returned by ExtractPath#ref_names.

I then turned my attention to working out why the ref was not present. I inspected gitaly's branch ref name query implementation (listRefNames in gitaly/internal/service/ref/refnames.go) but found it to be correct. However, I then noticed that this code wasn't being hit when servicing the failing request.

In trying to work out why this was the case I stumbled upon some caching logic for Repository model's branch_names list. It just so happens that disabling this caching logic avoided the issue:

diff --git a/app/models/repository.rb b/app/models/repository.rb
index ff355295862..bba51925c5f 100644
--- a/app/models/repository.rb
+++ b/app/models/repository.rb
@@ -37,7 +37,7 @@ class Repository
   # stores its data in the `commit_count` cache key.
   CACHED_METHODS = %i(size commit_count rendered_readme readme_path contribution_guide
                       changelog license_blob license_key gitignore
-                      gitlab_ci_yml branch_names tag_names branch_count
+                      gitlab_ci_yml tag_names branch_count
                       tag_count avatar exists? root_ref has_visible_content?
                       issue_template_names merge_request_template_names xcode_project?).freeze
 
@@ -571,7 +571,7 @@ class Repository
   end
 
   delegate :branch_names, to: :raw_repository
-  cache_method :branch_names, fallback: []
+  #cache_method :branch_names, fallback: []
 
   delegate :tag_names, to: :raw_repository
   cache_method :tag_names, fallback: []

I have no explanation for why this help but reverting this patch causes the issue to recur so there is certainly something fishy in this caching logic.