Importing a github project requires DNS resolution even when behind a proxy
Summary
GitLab tries to resolve the hostname of hosts when importing a github project (namely github.com and api.github.com), which requires access to a DNS even when it's not needed ( thanks to a HTTP/HTTPS proxy)
Steps to reproduce
- setup an instance with an outgoing HTTP proxy and no external DNS resolution
gitlab_rails['env'] = {
'http_proxy' => "http://foo.bar:8080",'https_proxy' => "http://foo.bar:8080",'no_proxy' => "*.blah"
}
gitaly['env'] = {
'http_proxy' => "http://foo.bar:8080",'https_proxy' => "http://foo.bar:8080",'no_proxy' => "*.blah"
}
gitlab_workhorse['env'] = {
'http_proxy' => "http://foo.bar:8080",'https_proxy' => "http://foo.bar:8080",'no_proxy' => "*.blah"
}
registry['env'] = {
'http_proxy' => "http://foo.bar:8080",'https_proxy' => "http://foo.bar:8080",'no_proxy' => "*.blah"
}
- import any project from Github
What is the current bug behavior?
We're getting a 500 error ( log below)
After a bit of digging in the source code, I added github.com and api.github.com to the local hosts file and voila, imports work (through the proxy).
What is the expected correct behavior?
Hostname resolution should not be required to import a project.
Relevant logs and/or screenshots
production.log when the 500 error occurs:
Completed 500 Internal Server Error in 20028ms (ActiveRecord: 2.4ms | Elasticsearch: 0.0ms)
Gitlab::UrlBlocker::BlockedUrlError (Host cannot be resolved or invalid):
lib/gitlab/url_blocker.rb:127:in `rescue in get_address_info'
lib/gitlab/url_blocker.rb:110:in `get_address_info'
lib/gitlab/url_blocker.rb:48:in `validate!'
lib/gitlab/octokit/middleware.rb:11:in `call'
lib/gitlab/legacy_github_import/client.rb:100:in `rate_limit'
lib/gitlab/legacy_github_import/client.rb:111:in `has_rate_limit?'
lib/gitlab/legacy_github_import/client.rb:115:in `rate_limit_exceed?'
lib/gitlab/legacy_github_import/client.rb:123:in `request'
lib/gitlab/legacy_github_import/client.rb:60:in `method_missing'
app/controllers/import/github_controller.rb:107:in `client_repos'
app/controllers/import/github_controller.rb:33:in `status'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:450:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:444:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:57:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/multipart.rb:117:in `call'
lib/gitlab/middleware/read_only/controller.rb:42:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'
Plan
As seen through #37941 (comment 713539071), using https://docs.gitlab.com/ee/security/webhooks.html#allowlist-for-local-requests as a guide to add in the hostnames that the importer reaches out to will enable importer to work as expected.
Based on that we should either update the generic importer documentation and/or the https://docs.gitlab.com/ee/security/webhooks.html#allowlist-for-local-requests to better describe the use here.
%15.9
Update inAfter getting completed Skip DNS rebinding protection when HTTP_PROXY environment is set. it's no longer necessary to add GitHub host to the allowlist.