GitLab Next

Summary

GitLab tries to resolve the hostname of hosts when importing a github project (namely github.com and api.github.com), which requires access to a DNS even when it's not needed ( thanks to a HTTP/HTTPS proxy)

Steps to reproduce

• setup an instance with an outgoing HTTP proxy and no external DNS resolution

gitlab_rails['env'] = {
   'http_proxy' => "http://foo.bar:8080",'https_proxy' => "http://foo.bar:8080",'no_proxy' => "*.blah"
}

gitaly['env'] = {
   'http_proxy' => "http://foo.bar:8080",'https_proxy' => "http://foo.bar:8080",'no_proxy' => "*.blah"
}

gitlab_workhorse['env'] = {
   'http_proxy' => "http://foo.bar:8080",'https_proxy' => "http://foo.bar:8080",'no_proxy' => "*.blah"
}

registry['env'] = {
   'http_proxy' => "http://foo.bar:8080",'https_proxy' => "http://foo.bar:8080",'no_proxy' => "*.blah"
}

• import any project from Github

What is the current bug behavior?

We're getting a 500 error ( log below)

After a bit of digging in the source code, I added github.com and api.github.com to the local hosts file and voila, imports work (through the proxy).

What is the expected correct behavior?

Hostname resolution should not be required to import a project.

Relevant logs and/or screenshots

production.log when the 500 error occurs:

Completed 500 Internal Server Error in 20028ms (ActiveRecord: 2.4ms | Elasticsearch: 0.0ms)
  
Gitlab::UrlBlocker::BlockedUrlError (Host cannot be resolved or invalid):
  
lib/gitlab/url_blocker.rb:127:in `rescue in get_address_info'
lib/gitlab/url_blocker.rb:110:in `get_address_info'
lib/gitlab/url_blocker.rb:48:in `validate!'
lib/gitlab/octokit/middleware.rb:11:in `call'
lib/gitlab/legacy_github_import/client.rb:100:in `rate_limit'
lib/gitlab/legacy_github_import/client.rb:111:in `has_rate_limit?'
lib/gitlab/legacy_github_import/client.rb:115:in `rate_limit_exceed?'
lib/gitlab/legacy_github_import/client.rb:123:in `request'
lib/gitlab/legacy_github_import/client.rb:60:in `method_missing'
app/controllers/import/github_controller.rb:107:in `client_repos'
app/controllers/import/github_controller.rb:33:in `status'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:450:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:444:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:57:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/multipart.rb:117:in `call'
lib/gitlab/middleware/read_only/controller.rb:42:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

Plan

As seen through #37941 (comment 713539071), using https://docs.gitlab.com/ee/security/webhooks.html#allowlist-for-local-requests as a guide to add in the hostnames that the importer reaches out to will enable importer to work as expected.

Based on that we should either update the generic importer documentation and/or the https://docs.gitlab.com/ee/security/webhooks.html#allowlist-for-local-requests to better describe the use here.