Push to a remote repository using proxy fails since 11.x
Summary
After updating my GitLab from 10.x to 11.x running inside Docker, push to an external remote repository fails. Note: I'm running GitLab behind a corporate proxy.
My proxy config in the docker-compose.yml file:
gitlab_rails['env'] = {
'http_proxy' => 'http://myproxy:3128',
'https_proxy' => 'http://myproxy:3128',
'no_proxy' => '127.0.0.1,localhost'
}
gitlab_workhorse['env'] = {
'http_proxy' => 'http://myproxy:3128',
'https_proxy' => 'http://myproxy:3128',
'no_proxy' => '127.0.0.1,localhost'
}
Steps to reproduce
Setup GitLab 10.x with correct proxy settings and configure Push to a remote repository for any repository. Works fine. Now update to 11.x and the Push to a remote repository fails with a connection timeout.
Interesting fact: If I'm providing wrong credentials, an "authentication failure" is raised. So in this case, the connection to the remote repository seems working.
With curl, I'm able to connect to the external git.
What is the current bug behavior?
RepositoryUpdateRemoteMirrorWorker fails with Connection timed out
What is the expected correct behavior?
RepositoryUpdateRemoteMirrorWorker should update the remote repository after any changes are made to the original repository.
Relevant logs and/or screenshots
2018-08-07_06:45:48.72171 2018-08-07T06:45:48.721Z 4430 TID-ovfs1qjpi RepositoryUpdateRemoteMirrorWorker JID-5c861ae547a87f0eb0acb737 INFO: fail: 127.664 sec
2018-08-07_06:45:48.72173 2018-08-07T06:45:48.721Z 4430 TID-ovfs1qjpi WARN: {"context":"Job raised exception","job":{"class":"RepositoryUpdateRemoteMirrorWorker","args":[7,"2018-08-07 06:43:41 +0000"],"retry":3,"queue":"repository_update_remote_mirror","dead":false,"jid":"5c861ae547a87f0eb0acb737","created_at":1533624221.05623,"enqueued_at":1533624221.0570886},"jobstr":"{\"class\":\"RepositoryUpdateRemoteMirrorWorker\",\"args\":[7,\"2018-08-07 06:43:41 +0000\"],\"retry\":3,\"queue\":\"repository_update_remote_mirror\",\"dead\":false,\"jid\":\"5c861ae547a87f0eb0acb737\",\"created_at\":1533624221.05623,\"enqueued_at\":1533624221.0570886}"}
2018-08-07_06:45:48.72174 2018-08-07T06:45:48.721Z 4430 TID-ovfs1qjpi WARN: RepositoryUpdateRemoteMirrorWorker::UpdateError: 2:fatal: unable to access 'https://[FILTERED]@my-external-git.com/group/project.git/': Failed to connect to my-external-git.com port 443: Connection timed out
2018-08-07_06:45:48.72175 2018-08-07T06:45:48.721Z 4430 TID-ovfs1qjpi WARN: /opt/gitlab/embedded/service/gitlab-rails/app/workers/repository_update_remote_mirror_worker.rb:29:in `perform'
2018-08-07_06:45:48.72175 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:187:in `execute_job'
2018-08-07_06:45:48.72176 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:169:in `block (2 levels) in process'
2018-08-07_06:45:48.72176 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
2018-08-07_06:45:48.72176 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_status/server_middleware.rb:5:in `call'
2018-08-07_06:45:48.72177 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-08-07_06:45:48.72177 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:6:in `call'
2018-08-07_06:45:48.72177 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-08-07_06:45:48.72178 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/shutdown.rb:52:in `call'
2018-08-07_06:45:48.72179 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-08-07_06:45:48.72179 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/server/active_record.rb:16:in `call'
2018-08-07_06:45:48.72179 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-08-07_06:45:48.72180 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sentry-raven-2.7.2/lib/raven/integrations/sidekiq.rb:9:in `call'
2018-08-07_06:45:48.72181 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-08-07_06:45:48.72182 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:133:in `invoke'
2018-08-07_06:45:48.72182 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:168:in `block in process'
2018-08-07_06:45:48.72182 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:139:in `block (6 levels) in dispatch'
2018-08-07_06:45:48.72183 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/job_retry.rb:98:in `local'
2018-08-07_06:45:48.72184 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:138:in `block (5 levels) in dispatch'
2018-08-07_06:45:48.72184 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq.rb:36:in `block in <module:Sidekiq>'
2018-08-07_06:45:48.72184 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:134:in `block (4 levels) in dispatch'
2018-08-07_06:45:48.72185 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:199:in `stats'
2018-08-07_06:45:48.72185 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:129:in `block (3 levels) in dispatch'
2018-08-07_06:45:48.72185 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/job_logger.rb:8:in `call'
2018-08-07_06:45:48.72186 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:128:in `block (2 levels) in dispatch'
2018-08-07_06:45:48.72186 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/job_retry.rb:73:in `global'
2018-08-07_06:45:48.72186 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:127:in `block in dispatch'
2018-08-07_06:45:48.72187 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/logging.rb:48:in `with_context'
2018-08-07_06:45:48.72188 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
2018-08-07_06:45:48.72188 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:126:in `dispatch'
2018-08-07_06:45:48.72188 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:167:in `process'
2018-08-07_06:45:48.72189 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:85:in `process_one'
2018-08-07_06:45:48.72189 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:73:in `run'
2018-08-07_06:45:48.72189 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:16:in `watchdog'
2018-08-07_06:45:48.72190 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:25:in `block in safe_thread'
Results of GitLab environment info
root@63304148535e:/# gitlab-rake gitlab:env:info
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.4.4p296
Gem Version: 2.7.6
Bundler Version:1.16.2
Rake Version: 12.3.1
Redis Version: 3.2.11
Git Version: 2.17.1
Sidekiq Version:5.1.3
Go Version: unknown
GitLab information
Version: 11.1.4
Revision: 63daf37
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: https://my-internal-git.com
HTTP Clone URL: https://my-internal-git.com/some-group/some-project.git
SSH Clone URL: git@my-internal-git.com:some-group/some-project.git
Using LDAP: yes
Using Omniauth: no
GitLab Shell
Version: 7.1.4
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks
Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
root@63304148535e:/# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab Shell ...
GitLab Shell version >= 7.1.4 ? ... OK (7.1.4)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
3/1 ... ok
3/2 ... ok
3/3 ... ok
3/4 ... ok
3/5 ... ok
3/6 ... ok
3/7 ... ok
3/8 ... ok
3/9 ... ok
3/10 ... ok
3/11 ... ok
7/13 ... ok
7/14 ... ok
7/15 ... ok
7/16 ... ok
9/17 ... ok
9/18 ... ok
11/19 ... ok
11/20 ... ok
11/21 ... ok
11/22 ... ok
11/23 ... ok
11/24 ... ok
20/25 ... ok
18/27 ... ok
2/30 ... ok
20/31 ... ok
30/32 ... ok
28/34 ... ok
30/35 ... ok
29/36 ... repository is empty
30/37 ... ok
28/38 ... ok
28/39 ... ok
28/40 ... ok
28/41 ... ok
28/42 ... ok
28/44 ... ok
28/45 ... ok
28/46 ... ok
28/47 ... ok
28/48 ... ok
31/49 ... ok
20/50 ... ok
3/51 ... ok
33/52 ... ok
34/53 ... ok
34/54 ... ok
34/55 ... ok
34/56 ... ok
34/58 ... ok
35/59 ... ok
35/60 ... ok
34/61 ... ok
3/62 ... ok
3/63 ... ok
34/64 ... ok
34/65 ... ok
3/67 ... ok
34/68 ... ok
34/69 ... ok
30/72 ... ok
35/73 ... ok
36/74 ... ok
3/97 ... ok
12/104 ... ok
44/105 ... ok
7/109 ... ok
7/110 ... ok
7/111 ... ok
55/112 ... ok
55/113 ... ok
55/114 ... ok
55/115 ... ok
55/116 ... ok
55/118 ... ok
55/119 ... ok
55/120 ... ok
55/121 ... ok
2/122 ... ok
46/123 ... ok
55/124 ... ok
55/125 ... ok
55/126 ... ok
55/127 ... ok
55/128 ... ok
57/130 ... ok
55/131 ... ok
58/132 ... ok
55/133 ... ok
55/134 ... ok
57/135 ... ok
20/136 ... ok
46/137 ... ok
55/138 ... ok
3/139 ... ok
60/141 ... ok
60/142 ... ok
60/143 ... ok
55/144 ... ok
55/145 ... ok
2/152 ... ok
2/153 ... ok
2/154 ... ok
2/155 ... ok
2/156 ... ok
2/157 ... ok
44/158 ... ok
55/159 ... ok
55/160 ... ok
55/161 ... ok
55/162 ... ok
55/163 ... ok
55/164 ... ok
55/165 ... ok
67/166 ... ok
55/167 ... ok
14/168 ... ok
2/169 ... ok
12/171 ... ok
3/172 ... ok
12/173 ... ok
55/174 ... ok
12/176 ... ok
12/177 ... ok
14/178 ... ok
36/184 ... ok
67/185 ... ok
44/186 ... ok
20/187 ... ok
67/188 ... ok
20/189 ... ok
3/190 ... ok
67/191 ... ok
3/192 ... ok
12/193 ... ok
20/194 ... ok
67/195 ... ok
3/196 ... ok
3/197 ... ok
7/198 ... ok
67/199 ... ok
67/200 ... ok
67/201 ... ok
20/203 ... ok
20/205 ... ok
58/206 ... ok
67/207 ... ok
55/208 ... ok
67/209 ... ok
55/210 ... ok
58/211 ... ok
44/212 ... ok
44/213 ... ok
44/215 ... ok
60/216 ... ok
70/217 ... ok
70/218 ... ok
58/219 ... ok
72/220 ... ok
72/221 ... ok
72/222 ... ok
67/223 ... ok
14/224 ... ok
73/228 ... ok
67/229 ... ok
73/230 ... ok
74/232 ... ok
53/233 ... repository is empty
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml
Checking LDAP ...
Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
--> removed <--
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
3/1 ... yes
3/2 ... yes
3/3 ... yes
3/4 ... yes
3/5 ... yes
3/6 ... yes
3/7 ... yes
3/8 ... yes
3/9 ... yes
3/10 ... yes
3/11 ... yes
7/13 ... yes
7/14 ... yes
7/15 ... yes
7/16 ... yes
9/17 ... yes
9/18 ... yes
11/19 ... yes
11/20 ... yes
11/21 ... yes
11/22 ... yes
11/23 ... yes
11/24 ... yes
20/25 ... yes
18/27 ... yes
2/30 ... yes
20/31 ... yes
30/32 ... yes
28/34 ... yes
30/35 ... yes
29/36 ... yes
30/37 ... yes
28/38 ... yes
28/39 ... yes
28/40 ... yes
28/41 ... yes
28/42 ... yes
28/44 ... yes
28/45 ... yes
28/46 ... yes
28/47 ... yes
28/48 ... yes
31/49 ... yes
20/50 ... yes
3/51 ... yes
33/52 ... yes
34/53 ... yes
34/54 ... yes
34/55 ... yes
34/56 ... yes
34/58 ... yes
35/59 ... yes
35/60 ... yes
34/61 ... yes
3/62 ... yes
3/63 ... yes
34/64 ... yes
34/65 ... yes
3/67 ... yes
34/68 ... yes
34/69 ... yes
30/72 ... yes
35/73 ... yes
36/74 ... yes
3/97 ... yes
12/104 ... yes
44/105 ... yes
7/109 ... yes
7/110 ... yes
7/111 ... yes
55/112 ... yes
55/113 ... yes
55/114 ... yes
55/115 ... yes
55/116 ... yes
55/118 ... yes
55/119 ... yes
55/120 ... yes
55/121 ... yes
2/122 ... yes
46/123 ... yes
55/124 ... yes
55/125 ... yes
55/126 ... yes
55/127 ... yes
55/128 ... yes
57/130 ... yes
55/131 ... yes
58/132 ... yes
55/133 ... yes
55/134 ... yes
57/135 ... yes
20/136 ... yes
46/137 ... yes
55/138 ... yes
3/139 ... yes
60/141 ... yes
60/142 ... yes
60/143 ... yes
55/144 ... yes
55/145 ... yes
2/152 ... yes
2/153 ... yes
2/154 ... yes
2/155 ... yes
2/156 ... yes
2/157 ... yes
44/158 ... yes
55/159 ... yes
55/160 ... yes
55/161 ... yes
55/162 ... yes
55/163 ... yes
55/164 ... yes
55/165 ... yes
67/166 ... yes
55/167 ... yes
14/168 ... yes
2/169 ... yes
12/171 ... yes
3/172 ... yes
12/173 ... yes
55/174 ... yes
12/176 ... yes
12/177 ... yes
14/178 ... yes
36/184 ... yes
67/185 ... yes
44/186 ... yes
20/187 ... yes
67/188 ... yes
20/189 ... yes
3/190 ... yes
67/191 ... yes
3/192 ... yes
12/193 ... yes
20/194 ... yes
67/195 ... yes
3/196 ... yes
3/197 ... yes
7/198 ... yes
67/199 ... yes
67/200 ... yes
67/201 ... yes
20/203 ... yes
20/205 ... yes
58/206 ... yes
67/207 ... yes
55/208 ... yes
67/209 ... yes
55/210 ... yes
58/211 ... yes
44/212 ... yes
44/213 ... yes
44/215 ... yes
60/216 ... yes
70/217 ... yes
70/218 ... yes
58/219 ... yes
72/220 ... yes
72/221 ... yes
72/222 ... yes
67/223 ... yes
14/224 ... yes
73/228 ... yes
67/229 ... yes
73/230 ... yes
74/232 ... yes
53/233 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.4.4)
Git version >= 2.9.5 ? ... yes (2.17.1)
Git user has default SSH configuration? ... yes
Active users: ... 47
Checking GitLab ... Finished