Skip to content

Random error 500 when pulling/pushing to repository using HTTPS

EDIT: It also happens with pulls.

Summary

Just upgraded to 12.2 and getting random 500 errors when using git and pulling/pushing to a project using HTTPS.

Steps to reproduce

Try to pull any repository using HTTPS on http://lab.shelter.moe (self-hosted gitlab) or push to one you created there.

What is the current bug behavior?

Git commands fail with a 500 error. These errors are random as, when you try pushing again, and again, it sometimes fails during the push, and sometimes the push succeeds.

See this terminal log for an example :

$ git push -u origin master
fatal: unable to access 'https://lab.shelter.moe/axelterizaki/ultrastar2ass.git/': The requested URL returned error: 500

$ git push -u origin master
fatal: unable to access 'https://lab.shelter.moe/axelterizaki/ultrastar2ass.git/': The requested URL returned error: 500

$ git push -u origin master
Counting objects: 24, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (19/19), done.
Writing objects: 100% (24/24), 25.73 KiB | 0 bytes/s, done.
Total 24 (delta 1), reused 0 (delta 0)
error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500 Internal Server Error
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
Everything up-to-date

$ git push -u origin master
fatal: unable to access 'https://lab.shelter.moe/axelterizaki/ultrastar2ass.git/': The requested URL returned error: 500

$ git push -u origin master
fatal: unable to access 'https://lab.shelter.moe/axelterizaki/ultrastar2ass.git/': The requested URL returned error: 500

$ git push -u origin master
Counting objects: 24, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (19/19), done.
Writing objects: 100% (24/24), 25.73 KiB | 0 bytes/s, done.
Total 24 (delta 1), reused 0 (delta 0)
To https://lab.shelter.moe/axelterizaki/ultrastar2ass.git
 * [new branch]      master -> master
Branch master set up to track remote branch master from origin.

What is the expected correct behavior?

It should be working all right as in 12.1.x

This happened ever since I upgraded Omnibus gitlab to 12.2 this morning.

Relevant logs and/or screenshots

I examined the production.log and here's what I found when trying a git push :

Started GET "/axelterizaki/ultrastar2ass.git/info/refs?service=git-receive-pack" for 192.168.122.1 at 2019-08-23 12:55:18 +0200
Processing by Gitlab::RequestForgeryProtection::Controller#index as */*
  Parameters: {"service"=>"git-receive-pack"}
Completed 200 OK in 0ms (ActiveRecord: 0.0ms)

Redis::CommandError (ERR value is not an integer or out of range):

lib/peek/views/redis_detailed.rb:10:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

Here it says it returned a 200, but my Apache frontend server returns a 500 instead :

xx.xxx.xx.xxx - - [23/Aug/2019:13:03:05 +0200] "GET /axelterizaki/ultrastar2ass.git/info/refs?service=git-receive-pack HTTP/1.1" 500 3228 "-" "git/2.19.0.windows.1"

I'm not sure what's wrong but I guess the Redis error in production.log has something to do with it.

Output of checks

See below

Results of GitLab environment info

Expand for output related to GitLab environment info

System information System: Ubuntu 16.04 Current User: git Using RVM: no Ruby Version: 2.6.3p62 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.22.0 Sidekiq Version:5.2.7 Go Version: unknown

GitLab information Version: 12.2.0 Revision: 1c1d47c5974 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.9 URL: https://lab.shelter.moe HTTP Clone URL: https://lab.shelter.moe/some-group/some-project.git SSH Clone URL: git@shelter.mahoro-net.org:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: gitlab, google_oauth2, twitter, github

GitLab Shell Version: 9.3.0 Repository storage paths:

  • default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 9.3.0 ? ... OK (9.3.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK

Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Checking Reply by email ...

IMAP server credentials are correct? ... yes Init.d configured correctly? ... skipped MailRoom running? ... skipped

Checking Reply by email ... Finished

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 5/2 ... yes 2/4 ... yes 1/6 ... yes 1/7 ... yes 1/8 ... yes 7/9 ... yes 19/10 ... yes 4/11 ... yes 22/12 ... yes 156/13 ... yes 15/14 ... yes 8/15 ... yes 5/18 ... yes 45/22 ... yes 156/23 ... yes 52/24 ... yes 52/25 ... yes 59/26 ... yes 155/28 ... yes 62/32 ... yes 62/33 ... yes 52/34 ... yes 155/35 ... yes 5/36 ... yes 54/40 ... yes 67/41 ... yes 67/43 ... yes 67/44 ... yes 156/45 ... yes 65/46 ... yes 66/47 ... yes 82/52 ... yes 96/55 ... yes 82/59 ... yes 8/65 ... yes 7/66 ... yes 5/67 ... yes 5/68 ... yes 82/69 ... yes 20/70 ... yes 45/71 ... yes 110/72 ... yes 82/73 ... yes 8/74 ... yes 8/75 ... yes 8/76 ... yes 156/77 ... yes 9/80 ... yes 66/81 ... yes 78/83 ... yes 1/87 ... yes 135/89 ... yes 126/90 ... yes 135/91 ... yes 135/92 ... yes 135/93 ... yes 1/98 ... yes 153/99 ... yes 155/100 ... yes 155/101 ... yes 5/102 ... yes 20/104 ... yes 8/106 ... yes 45/107 ... yes 102/109 ... yes 129/110 ... yes 5/111 ... yes 9/112 ... yes 198/113 ... yes 102/114 ... yes 198/115 ... yes 193/116 ... yes 202/117 ... yes 45/118 ... yes 135/119 ... yes 107/120 ... yes 107/121 ... yes 20/123 ... yes 5/129 ... yes 135/130 ... yes 5/131 ... yes 52/132 ... yes 1/133 ... yes 1/134 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.3) Git version >= 2.22.0 ? ... yes (2.22.0) Git user has default SSH configuration? ... yes Active users: ... 196

Checking GitLab App ... Finished

Checking GitLab subtasks ... Finished

Possible fixes

For now my only possible workaround is to migrate back to an older version (12.1) where this didn't happen.

Edited by Guillaume Lebigot