http(s)_proxy environment not honored when artifacts on object storage are enabled
Summary
GitLab does not honor the http_proxy
/ https_proxy
environment variables on some routes with artifacts and object storage enabled.
Steps to reproduce
- Setup a GitLab instance (Omnibus in my case)
- Enable artifacts object storage :
gitlab_rails['artifacts_enabled'] = true gitlab_rails['artifacts_object_store_enabled'] = true gitlab_rails['artifacts_object_store_remote_directory'] = 'a-random-bucket' gitlab_rails['artifacts_object_store_connection'] = { 'provider' => 'AWS', 'region' => 'eu-west-1', 'aws_access_key_id' => 'xxx', 'aws_secret_access_key' => 'xxx' }
0. Enable http_proxy:
gitlab_rails['env'] = { "http_proxy" => "http://url-to-my-proxy:12345", "https_proxy" => "http://url-to-my-proxy:12345" }
0. Run a job that succeeds (I'm not sure if it's important that the job creates an artifact)
0. Once the job has succeeded, try to open the logs of the job.
0. The call to `/foo/bar/-/jobs/12345.json` will bypass the `http_proxy` when calling AWS.
0. Try to run a job in a following stage; GitLab would again try to bypass the proxy when retrieving the artifacts from the previous stage from AWS.
### What is the current *bug* behavior?
`http(s)_proxy` environment gets ignored when:
- Opening the job logs page in the UI (raw job logs works properly)
- A runner tries to download an artifact.
- Downloading an artifact from API. (downloading artifact from UI works properly)
### What is the expected *correct* behavior?
Never bypass `http(s)_proxy` environment.
### Relevant logs and/or screenshots
Since port 443 was blocked on my instance, I'm able to provide this stacktrace:
Completed 500 Internal Server Error in 60018ms (ActiveRecord: 2.4ms)
Net::OpenTimeout (execution expired):
lib/gitlab/ci/trace/http_io.rb:141:in get_chunk' lib/gitlab/ci/trace/http_io.rb:98:in
readline'
lib/gitlab/ci/trace/http_io.rb:71:in each_line' lib/gitlab/ci/ansi2html.rb:156:in
convert'
lib/gitlab/ci/ansi2html.rb:28:in convert' lib/gitlab/ci/trace/stream.rb:63:in
html_with_state'
app/controllers/projects/jobs_controller.rb:77:in block (3 levels) in trace' app/controllers/projects/jobs_controller.rb:68:in
block in trace'
lib/gitlab/ci/trace.rb:71:in read' app/controllers/projects/jobs_controller.rb:67:in
trace'
lib/gitlab/i18n.rb:50:in with_locale' lib/gitlab/i18n.rb:56:in
with_user_locale'
app/controllers/application_controller.rb:326:in set_locale' lib/gitlab/middleware/multipart.rb:95:in
call'
lib/gitlab/request_profiler/middleware.rb:14:in call' lib/gitlab/middleware/go.rb:17:in
call'
lib/gitlab/etag_caching/middleware.rb:11:in call' lib/gitlab/middleware/read_only/controller.rb:28:in
call'
lib/gitlab/middleware/read_only.rb:16:in call' lib/gitlab/request_context.rb:18:in
call'
lib/gitlab/metrics/requests_rack_middleware.rb:27:in call' lib/gitlab/middleware/release_env.rb:10:in
call'
### Output of checks
Not relevant?
#### Results of GitLab environment info
<details>
<summary>Expand for output related to GitLab environment info</summary>
<pre>
System information
System: Ubuntu 16.04
Current User: git
Using RVM: no
Ruby Version: 2.3.6p384
Gem Version: 2.6.13
Bundler Version:1.13.7
Rake Version: 12.3.0
Redis Version: 3.2.11
Git Version: 2.14.3
Sidekiq Version:5.0.5
Go Version: unknown
GitLab information
Version: 10.7.0
Revision: 2011c9b
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: https://<redacted>.com
HTTP Clone URL: https://<redacted>/some-group/some-project.git
SSH Clone URL: git@<redacted>:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: bitbucket
GitLab Shell
Version: 7.1.2
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks
Git: /opt/gitlab/embedded/bin/git
</pre>
</details>
#### Results of GitLab application Check
<details>
<summary>Expand for output related to the GitLab application check</summary>
<pre>
Checking GitLab Shell ...
GitLab Shell version >= 7.1.2 ? ... OK (7.1.2) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 15/5 ... ok 16/6 ... ok 7/7 ... ok 16/8 ... ok 7/11 ... ok 7/12 ... ok 30/13 ... ok 30/14 ... ok 15/16 ... ok 7/17 ... ok 7/18 ... ok 7/19 ... ok 7/20 ... ok 9/22 ... ok 30/23 ... ok 30/24 ... ok 16/25 ... ok 7/26 ... ok 30/27 ... ok 7/28 ... ok 7/29 ... ok 11/30 ... ok 11/31 ... ok 7/32 ... ok 7/33 ... ok 7/34 ... ok 2/36 ... ok 15/37 ... ok 13/38 ... ok 7/39 ... ok 7/40 ... ok 7/41 ... ok 21/42 ... ok 7/43 ... ok 16/45 ... ok 16/46 ... ok 7/47 ... ok 16/49 ... ok 16/50 ... ok 9/51 ... ok 15/52 ... ok 9/53 ... ok 20/54 ... ok 7/55 ... ok 20/56 ... ok 20/57 ... ok 20/59 ... ok 20/61 ... ok 16/62 ... ok 9/64 ... ok 21/66 ... ok 20/67 ... ok 21/69 ... ok 20/72 ... ok 20/73 ... ok 7/74 ... ok 7/75 ... ok 20/76 ... ok 9/78 ... ok 7/82 ... ok 8/83 ... ok 7/84 ... ok 9/85 ... ok 7/86 ... ok 15/87 ... ok 7/88 ... ok 7/89 ... ok 7/90 ... ok 7/91 ... ok 13/93 ... ok 20/96 ... ok 20/97 ... ok 16/98 ... ok 7/100 ... ok 21/101 ... ok 16/102 ... ok 11/105 ... ok 20/106 ... ok 20/107 ... ok 16/108 ... ok 16/109 ... ok 7/110 ... ok 19/111 ... repository is empty 16/112 ... ok 7/114 ... repository is empty 20/115 ... ok 16/116 ... ok 7/117 ... ok 7/118 ... ok 11/119 ... ok 9/120 ... ok 7/121 ... ok 11/122 ... repository is empty 7/123 ... repository is empty 16/124 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 15/5 ... yes 16/6 ... yes 7/7 ... yes 16/8 ... yes 7/11 ... yes 7/12 ... yes 30/13 ... yes 30/14 ... yes 15/16 ... yes 7/17 ... yes 7/18 ... yes 7/19 ... yes 7/20 ... yes 9/22 ... yes 30/23 ... yes 30/24 ... yes 16/25 ... yes 7/26 ... yes 30/27 ... yes 7/28 ... yes 7/29 ... yes 11/30 ... yes 11/31 ... yes 7/32 ... yes 7/33 ... yes 7/34 ... yes 2/36 ... yes 15/37 ... yes 13/38 ... yes 7/39 ... yes 7/40 ... yes 7/41 ... yes 21/42 ... yes 7/43 ... yes 16/45 ... yes 16/46 ... yes 7/47 ... yes 16/49 ... yes 16/50 ... yes 9/51 ... yes 15/52 ... yes 9/53 ... yes 20/54 ... yes 7/55 ... yes 20/56 ... yes 20/57 ... yes 20/59 ... yes 20/61 ... yes 16/62 ... yes 9/64 ... yes 21/66 ... yes 20/67 ... yes 21/69 ... yes 20/72 ... yes 20/73 ... yes 7/74 ... yes 7/75 ... yes 20/76 ... yes 9/78 ... yes 7/82 ... yes 8/83 ... yes 7/84 ... yes 9/85 ... yes 7/86 ... yes 15/87 ... yes 7/88 ... yes 7/89 ... yes 7/90 ... yes 7/91 ... yes 13/93 ... yes 20/96 ... yes 20/97 ... yes 16/98 ... yes 7/100 ... yes 21/101 ... yes 16/102 ... yes 11/105 ... yes 20/106 ... yes 20/107 ... yes 16/108 ... yes 16/109 ... yes 7/110 ... yes 19/111 ... yes 16/112 ... yes 7/114 ... yes 20/115 ... yes 16/116 ... yes 7/117 ... yes 7/118 ... yes 11/119 ... yes 9/120 ... yes 7/121 ... yes 11/122 ... yes 7/123 ... yes 16/124 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.3.6) Git version >= 2.9.5 ? ... yes (2.14.3) Git user has default SSH configuration? ... yes Active users: ... 15
Checking GitLab ... Finished
</pre>
</details>
### Possible fixes
Really unsure, but may be related to `ReleaseEnv` middleware, as seen in the stacktrace. It seems to explicitely clear the env, which might be why Excon doesn't honor it when calling AWS. See: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/middleware/release_env.rb#L10
app.call(env).tap { env.clear }
If somebody can give me some hint on the root cause of the bug and the proper way fix it, I can submit a PR.
Maybe related to https://gitlab.com/gitlab-org/gitlab-runner/issues/2181