GraphQL ContainerRegistry::BaseClient ignores http_proxy
Summary
Gitlab in docker is configured with http_proxy and blob storage for container registry. Container registry works as expected, images are stored in blob storage and are accessible via podman or docker from within gitlab-runner jobs or workstations.
Steps to reproduce
- setup docker behind http proxy
- run gitlab in such docker daemon using docker-compose
- configure http_proxy in
gitlab.rb# grep -B2 -A3 http_proxy gitlab-config/gitlab.rb|grep -v -e '^--$' -e '^# *}$'|sort |uniq -c|sort -n 1 alertmanager['env'] = { 1 gitaly['env'] = { 1 gitlab_rails['env'] = { 1 gitlab_workhorse['env'] = { 1 mattermost['env'] = { 1 prometheus['env'] = { 1 registry['env'] = { 1 "REGISTRY_HTTP_RELATIVEURLS" => true, 7 } 7 "http_proxy" => "http://<redacted>:3128", 7 "https_proxy" => "http://<redacted>:3128", 7 "no_proxy" => ".<redacted>,localhost,127.0.0.1" - enable
object_storage# grep "^[^#].*object_store" gitlab-config/gitlab.rb gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['proxy_download'] = false gitlab_rails['object_store']['connection'] = { gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'ci-artifacts' gitlab_rails['object_store']['objects']['external_diffs']['enabled'] = false gitlab_rails['object_store']['objects']['lfs']['enabled'] = false gitlab_rails['object_store']['objects']['uploads']['enabled'] = false gitlab_rails['object_store']['objects']['packages']['bucket'] = 'container-registry' gitlab_rails['object_store']['objects']['dependency_proxy']['enabled'] = false gitlab_rails['object_store']['objects']['terraform_state']['enabled'] = false gitlab_rails['object_store']['objects']['pages']['enabled'] = false gitlab_rails['external_diffs_object_store_enabled'] = false gitlab_rails['lfs_object_store_enabled'] = false# grep -A6 "^[^#]*registry\['storage'\]" gitlab-config/gitlab.rb registry['storage'] = { 'azure' => { 'accountname' => 'redacted', 'accountkey' => 'redactoed', 'container' => 'container-registry' } }gitlab_rails['object_store']['proxy_download'] = truedoes not make difference
What is the current bug behavior?
Gitlab UI shows all container repositories and lists tag counts but the tag listing timeouts.
What is the expected correct behavior?
Gitlab UI shows all container repositories and lists tag counts and the tag listing shows all tags for repository
Relevant logs and/or screenshots
The gitlab-rails/exceptions_json.log shows
{
"severity": "ERROR",
"time": "2022-08-10T13:27:34.539Z",
"correlation_id": "01GA3Z0DZZ52P001YH824TQV3J",
"exception.class": "Faraday::ConnectionFailed",
"exception.message": "Failed to open TCP connection to <redacted>.blob.core.windows.net:443 (Connection refused - connect(2) for \"<redacted>.blob.core.windows.net\" port 443)",
"exception.backtrace": [
"lib/gitlab/faraday/error_callback.rb:30:in `call'",
"lib/container_registry/client.rb:133:in `blob'",
"lib/container_registry/blob.rb:47:in `data'",
"lib/container_registry/tag.rb:69:in `config'",
"lib/container_registry/tag.rb:78:in `created_at'",
"lib/gitlab/graphql/present/field_extension.rb:18:in `resolve'",
"lib/gitlab/graphql/tracers/timer_tracer.rb:20:in `trace'",
"lib/gitlab/graphql/tracers/logger_tracer.rb:14:in `trace'",
"lib/gitlab/graphql/tracers/metrics_tracer.rb:13:in `trace'",
"lib/gitlab/graphql/tracers/application_context_tracer.rb:23:in `trace'",
"lib/gitlab/graphql/tracers/timer_tracer.rb:20:in `trace'",
"lib/gitlab/graphql/generic_tracing.rb:48:in `with_labkit_tracing'",
"lib/gitlab/graphql/generic_tracing.rb:38:in `platform_trace'",
"lib/gitlab/graphql/tracers/logger_tracer.rb:14:in `trace'",
"lib/gitlab/graphql/tracers/metrics_tracer.rb:13:in `trace'",
"lib/gitlab/graphql/tracers/application_context_tracer.rb:23:in `trace'",
"lib/gitlab/graphql/tracers/timer_tracer.rb:20:in `trace'",
"lib/gitlab/graphql/generic_tracing.rb:48:in `with_labkit_tracing'",
"lib/gitlab/graphql/generic_tracing.rb:38:in `platform_trace'",
"lib/gitlab/graphql/tracers/logger_tracer.rb:14:in `trace'",
"lib/gitlab/graphql/tracers/metrics_tracer.rb:13:in `trace'",
"lib/gitlab/graphql/tracers/application_context_tracer.rb:23:in `trace'",
"app/graphql/gitlab_schema.rb:51:in `multiplex'",
"app/controllers/graphql_controller.rb:160:in `execute_multiplex'",
"app/controllers/graphql_controller.rb:57:in `execute'",
"app/controllers/application_controller.rb:531:in `set_current_admin'",
"lib/gitlab/session.rb:11:in `with_session'",
"app/controllers/application_controller.rb:522:in `set_session_storage'",
"lib/gitlab/i18n.rb:107:in `with_locale'",
"lib/gitlab/i18n.rb:113:in `with_user_locale'",
"app/controllers/application_controller.rb:516:in `set_locale'",
"app/controllers/application_controller.rb:510:in `set_current_context'",
"lib/gitlab/middleware/memory_report.rb:13:in `call'",
"lib/gitlab/middleware/speedscope.rb:13:in `call'",
"lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
"lib/gitlab/jira/middleware.rb:19:in `call'",
"lib/gitlab/middleware/go.rb:20:in `call'",
"lib/gitlab/etag_caching/middleware.rb:21:in `call'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'",
"lib/gitlab/database/query_analyzer.rb:37:in `within'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `call'",
"lib/gitlab/middleware/multipart.rb:173:in `call'",
"lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
"lib/gitlab/middleware/read_only.rb:18:in `call'",
"lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
"lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
"lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
"lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
"lib/gitlab/middleware/request_context.rb:21:in `call'",
"lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
"config/initializers/fix_local_cache_middleware.rb:11:in `call'",
"lib/gitlab/middleware/compressed_json.rb:26:in `call'",
"lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
"lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
"lib/gitlab/metrics/requests_rack_middleware.rb:77:in `call'",
"lib/gitlab/middleware/release_env.rb:13:in `call'"
],
"user.username": "<redacted>",
"tags.program": "web",
"tags.locale": "en",
"tags.feature_category": "container_registry",
"tags.correlation_id": "01GA3Z0DZZ52P001YH824TQV3J",
"extra.class": "ContainerRegistry::BaseClient",
"extra.url": "http://127.0.0.1:5000/v2/<redacted>"
}Results of GitLab environment info
When Farraday invoked from gitlab-rails console the connectivity is OK.
irb(main):005:0> f = Faraday::Connection.new("https://etngitlab.blob.core.windows.net"); f.get().status
=> 400Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`) System information System: Current User: git Using RVM: no Ruby Version: 2.7.5p203 Gem Version: 3.1.4 Bundler Version:2.3.15 Rake Version: 13.0.6 Redis Version: 6.2.7 Sidekiq Version:6.4.0 Go Version: unknown GitLab information Version: 15.2.2 Revision: 4ecb014a935 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 13.6 URL: https:// HTTP Clone URL: https://>redacted>/some-group/some-project.git SSH Clone URL: git@:some-group/some-project.git Using LDAP: no Using Omniauth: no GitLab Shell Version: 14.9.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true) root@6580a21e27f7:/# gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.9.0 ? ... OK (14.9.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... no Try fixing it: sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} ; sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads - exec chmod 0700 {} ; For more information see: doc/install/installation.md in section "GitLab" Please fix the error above and rerun the checks. Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd u nits) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor syst emd units) Projects have namespace: ... 5/1 ... yes (...) 554/2505 ... yes Redis version >= 6.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.5) Git user has default SSH configuration? ... yes Active users: ... 226 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
(we will only investigate if the tests are passing)
Possible fixes
Graphql maybe does not respect the http_proxy settings.