Cluster integration does not honor HTTP proxy settings
Summary
When installing Helm Tiller via the Cluster integration page, it does not respect HTTP proxy settings defined in gitlab_rails['env']
.
Steps to reproduce
NOTE: For step 3 to display the output shown below, Gitlab must be running behind an Internet proxy (a common scenario in many companies).
-
Add an existing Kubernetes cluster in your project > CI/CD > Clusters. Make sure the serviceaccount token+cert you enter here has enough privileges to create a new k8s namespace (
gitlab-managed-apps
) and to deploy resources into that namespace. -
Click the "Install" button.
-
Check back a few minutes later. If you're running behind a corporate proxy, you should see the following error: By the way, the error overflows the red box. The full message is:
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.6/main: operation timed out WARNING: Ignoring APKINDEX.84815163.tar.gz: No such file or directory ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.6/community: operation timed out WARNING: Ignoring APKINDEX.24d64ab1.tar.gz: No such file or directory ERROR: unsatisfiable constraints:
-
Add http_proxy settings to
/etc/gitlab/gitlab.rb
like so (replaceproxy.example.com
of course):##! Docs: https://docs.gitlab.com/omnibus/settings/environment-variables.html gitlab_rails['env'] = { 'http_proxy' => 'http://proxy.example.com', 'https_proxy' => 'http://proxy.example.com', 'HTTP_PROXY' => 'http://proxy.example.com', 'HTTPS_PROXY' => 'http://proxy.example.com', 'no_proxy' => 'localhost', 'NO_PROXY' => 'localhost' }
-
Run
gitlab-ctl reconfigure
and wait for it to finish -
Test that your proxy settings work, through an action such as importing an external repo from outside your network.
-
Back on the cluster page, click the "Install" button next to Helm Tiller.
-
Refresh the page until you see the following rather unhelpful error (it took upto a minute to appear, in my case):
What is the current bug behavior?
- Gitlab does not honor my proxy settings in the process of setting up Tiller
- When I do add proxy settings, the error displayed does not indicate what the real problem is (I assume the 502/503 response code was not propagated to the UI)
What is the expected correct behavior?
- Gitlab should use the given proxy env variables when starting the
install-helm
pod that ultimately deploys Tiller
Results of GitLab environment info
Note: some sensitive output has been edited.
Expand for output related to GitLab environment info
System information System: Proxy: NO_PROXY: localhost HTTP_PROXY: http://proxy.example.com HTTPS_PROXY: http://proxy.example.com https_proxy: http://proxy.example.com http_proxy: http://proxy.example.com no_proxy: localhost Current User: git Using RVM: no Ruby Version: 2.3.6p384 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.3.0 Redis Version: 3.2.11 Git Version: 2.14.3 Sidekiq Version:5.0.5 Go Version: unknown
GitLab information Version: 10.4.0-ee Revision: 00592f0 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql DB Version: 9.6.6 URL: https://gitlab.example.com HTTP Clone URL: https://gitlab.example.com/some-group/some-project.git SSH Clone URL: git@gitlab.example.com:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: yes Using Omniauth: no
GitLab Shell Version: 5.11.0 Repository storage paths:
- default: /gitlab-data/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...
GitLab Shell version >= 5.11.0 ? ... OK (5.11.0) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 7/1 ... ok 10/2 ... repository is empty 17/3 ... ok 3/4 ... ok 17/5 ... ok 17/6 ... ok 17/7 ... ok 17/8 ... ok 17/9 ... ok 17/10 ... ok 17/11 ... ok 18/12 ... ok 16/13 ... ok 19/14 ... ok 18/18 ... ok 7/24 ... ok 7/25 ... ok 16/26 ... ok 126/28 ... ok 126/29 ... ok 126/30 ... ok 7/31 ... ok 29/32 ... ok 29/33 ... ok 7/34 ... ok 21/35 ... ok 7/36 ... ok 25/37 ... ok 7/38 ... ok 42/39 ... ok 59/40 ... ok 7/44 ... ok 7/47 ... ok 59/48 ... ok 42/49 ... ok 42/50 ... ok 41/51 ... ok 44/52 ... ok 17/53 ... ok 41/54 ... ok 11/56 ... ok 29/58 ... repository is empty 59/59 ... ok 51/60 ... ok 62/61 ... repository is empty 62/62 ... ok 7/63 ... ok 68/64 ... repository is empty 20/65 ... ok 47/66 ... ok 47/68 ... ok 7/70 ... ok 17/71 ... ok 38/72 ... ok 17/73 ... ok 17/75 ... ok 38/76 ... ok 18/77 ... ok 19/78 ... ok 84/79 ... ok 85/80 ... ok 92/81 ... ok 17/82 ... ok 62/83 ... repository is empty 80/85 ... ok 59/86 ... ok 75/87 ... repository is empty 37/88 ... ok 23/89 ... ok 7/90 ... ok 19/92 ... ok 106/93 ... ok 106/94 ... ok 44/95 ... ok 84/96 ... ok 116/97 ... ok 66/98 ... repository is empty 6/99 ... ok 116/101 ... ok 116/102 ... ok 7/111 ... ok 35/112 ... ok 18/113 ... ok 37/114 ... ok 116/116 ... ok 116/117 ... ok 116/118 ... ok 116/119 ... ok 4/121 ... ok 64/122 ... ok 64/123 ... ok 17/124 ... ok 17/125 ... ok 4/126 ... ok 6/127 ... ok 15/128 ... ok 127/129 ... repository is empty 51/130 ... ok 130/131 ... ok 41/132 ... ok 131/133 ... repository is empty 126/146 ... repository is empty 138/147 ... ok 7/148 ... ok 116/149 ... ok 7/151 ... repository is empty 7/152 ... repository is empty 139/153 ... ok 116/163 ... ok 116/164 ... ok 116/165 ... ok 116/166 ... ok 116/167 ... ok 116/168 ... ok 116/169 ... ok 116/170 ... ok 116/171 ... ok 116/172 ... ok 116/173 ... ok 116/174 ... ok 116/175 ... ok 116/176 ... ok 116/178 ... ok 116/179 ... ok 116/180 ... repository is empty 116/181 ... repository is empty 116/182 ... ok 116/183 ... ok 116/184 ... ok 116/185 ... ok 116/186 ... ok 139/187 ... ok 7/188 ... ok 47/190 ... ok 59/191 ... ok 139/192 ... ok 4/194 ... ok 145/196 ... ok 145/197 ... ok 145/205 ... ok 145/206 ... ok 145/207 ... ok 145/208 ... ok 145/209 ... ok 145/210 ... ok 145/211 ... ok 145/212 ... ok 145/213 ... ok 145/214 ... ok 145/215 ... ok 145/216 ... ok 145/217 ... ok 145/223 ... ok 145/224 ... ok 145/225 ... ok 145/226 ... ok 145/227 ... ok 145/228 ... ok 145/229 ... ok 145/230 ... ok 145/231 ... ok 145/232 ... ok 145/233 ... ok 145/234 ... ok 145/235 ... ok 145/237 ... ok 145/239 ... ok 7/240 ... ok 145/241 ... ok 145/242 ... ok 141/243 ... ok 67/244 ... ok 3/245 ... repository is empty 4/246 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /gitlab-data/ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) [MILDLY SENSITIVE INFO; IRRELEVANT FOR THIS ISSUE; ELIDED]
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 7/1 ... yes 10/2 ... yes 17/3 ... yes 3/4 ... yes 17/5 ... yes 17/6 ... yes 17/7 ... yes 17/8 ... yes 17/9 ... yes 17/10 ... yes 17/11 ... yes 18/12 ... yes 16/13 ... yes 19/14 ... yes 18/18 ... yes 7/24 ... yes 7/25 ... yes 16/26 ... yes 126/28 ... yes 126/29 ... yes 126/30 ... yes 7/31 ... yes 29/32 ... yes 29/33 ... yes 7/34 ... yes 21/35 ... yes 7/36 ... yes 25/37 ... yes 7/38 ... yes 42/39 ... yes 59/40 ... yes 7/44 ... yes 7/47 ... yes 59/48 ... yes 42/49 ... yes 42/50 ... yes 41/51 ... yes 44/52 ... yes 17/53 ... yes 41/54 ... yes 11/56 ... yes 29/58 ... yes 59/59 ... yes 51/60 ... yes 62/61 ... yes 62/62 ... yes 7/63 ... yes 68/64 ... yes 20/65 ... yes 47/66 ... yes 47/68 ... yes 7/70 ... yes 17/71 ... yes 38/72 ... yes 17/73 ... yes 17/75 ... yes 38/76 ... yes 18/77 ... yes 19/78 ... yes 84/79 ... yes 85/80 ... yes 92/81 ... yes 17/82 ... yes 62/83 ... yes 80/85 ... yes 59/86 ... yes 75/87 ... yes 37/88 ... yes 23/89 ... yes 7/90 ... yes 19/92 ... yes 106/93 ... yes 106/94 ... yes 44/95 ... yes 84/96 ... yes 116/97 ... yes 66/98 ... yes 6/99 ... yes 116/101 ... yes 116/102 ... yes 7/111 ... yes 35/112 ... yes 18/113 ... yes 37/114 ... yes 116/116 ... yes 116/117 ... yes 116/118 ... yes 116/119 ... yes 4/121 ... yes 64/122 ... yes 64/123 ... yes 17/124 ... yes 17/125 ... yes 4/126 ... yes 6/127 ... yes 15/128 ... yes 127/129 ... yes 51/130 ... yes 130/131 ... yes 41/132 ... yes 131/133 ... yes 126/146 ... yes 138/147 ... yes 7/148 ... yes 116/149 ... yes 7/151 ... yes 7/152 ... yes 139/153 ... yes 116/163 ... yes 116/164 ... yes 116/165 ... yes 116/166 ... yes 116/167 ... yes 116/168 ... yes 116/169 ... yes 116/170 ... yes 116/171 ... yes 116/172 ... yes 116/173 ... yes 116/174 ... yes 116/175 ... yes 116/176 ... yes 116/178 ... yes 116/179 ... yes 116/180 ... yes 116/181 ... yes 116/182 ... yes 116/183 ... yes 116/184 ... yes 116/185 ... yes 116/186 ... yes 139/187 ... yes 7/188 ... yes 47/190 ... yes 59/191 ... yes 139/192 ... yes 4/194 ... yes 145/196 ... yes 145/197 ... yes 145/205 ... yes 145/206 ... yes 145/207 ... yes 145/208 ... yes 145/209 ... yes 145/210 ... yes 145/211 ... yes 145/212 ... yes 145/213 ... yes 145/214 ... yes 145/215 ... yes 145/216 ... yes 145/217 ... yes 145/223 ... yes 145/224 ... yes 145/225 ... yes 145/226 ... yes 145/227 ... yes 145/228 ... yes 145/229 ... yes 145/230 ... yes 145/231 ... yes 145/232 ... yes 145/233 ... yes 145/234 ... yes 145/235 ... yes 145/237 ... yes 145/239 ... yes 7/240 ... yes 145/241 ... yes 145/242 ... yes 141/243 ... yes 67/244 ... yes 3/245 ... yes 4/246 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.3.6) Git version >= 2.7.3 ? ... yes (2.14.3) Git user has default SSH configuration? ... yes Active users: ... 117 Elasticsearch version 5.1 - 5.5? ... skipped (elasticsearch is disabled)
Checking GitLab ... Finished
Possible fixes
I think the issue lies here: https://gitlab.com/gitlab-org/gitlab-ce/blob/89854139a4bcdd0cc16b67e7c49ec781aee70036/lib/gitlab/kubernetes/helm/pod.rb#L74-80