no_proxy setting is ignored in Webhooks
Summary
Related support ticket - internal.
no_proxy
configuration is ignored in Gitlab 15.11.2. Due to this, webhook fails with connection error.
{
"severity": "ERROR",
"time": "2023-05-08T16:45:42.450Z",
"correlation_id": "01GZY4262WVNHVEQSPFBZB3EJ5",
"exception.class": "Errno::ECONNRESET",
"exception.message": "Connection reset by peer - SSL_connect",
"exception.backtrace": [
"lib/gitlab/net_http_adapter.rb:21:in `connect'",
"lib/gitlab/http.rb:55:in `perform_request'",
"app/services/web_hook_service.rb:123:in `make_request'",
"app/services/web_hook_service.rb:72:in `execute'",
"app/models/hooks/web_hook.rb:57:in `execute'",
"app/controllers/concerns/web_hooks/hook_log_actions.rb:36:in `execute_hook'",
"app/controllers/concerns/web_hooks/hook_log_actions.rb:23:in `retry'",
"ee/lib/gitlab/ip_address_state.rb:10:in `with'",
"ee/app/controllers/ee/application_controller.rb:45:in `set_current_ip_address'",
"app/controllers/application_controller.rb:524:in `set_current_admin'",
"lib/gitlab/session.rb:11:in `with_session'",
"app/controllers/application_controller.rb:515:in `set_session_storage'",
"lib/gitlab/i18n.rb:107:in `with_locale'",
"lib/gitlab/i18n.rb:113:in `with_user_locale'",
"app/controllers/application_controller.rb:506:in `set_locale'",
"app/controllers/application_controller.rb:499:in `set_current_context'",
"lib/gitlab/middleware/memory_report.rb:13:in `call'",
"lib/gitlab/middleware/speedscope.rb:13:in `call'",
"lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
"lib/gitlab/jira/middleware.rb:19:in `call'",
"lib/gitlab/middleware/go.rb:20:in `call'",
"lib/gitlab/etag_caching/middleware.rb:21:in `call'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'",
"lib/gitlab/database/query_analyzer.rb:37:in `within'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `call'",
"lib/gitlab/middleware/multipart.rb:173:in `call'",
"lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
"lib/gitlab/middleware/read_only.rb:18:in `call'",
"lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
"lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
"lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
"lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
"lib/gitlab/middleware/request_context.rb:21:in `call'",
"lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
"config/initializers/fix_local_cache_middleware.rb:11:in `call'",
"lib/gitlab/middleware/compressed_json.rb:37:in `call'",
"lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
"lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
"lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'",
"lib/gitlab/middleware/release_env.rb:13:in `call'"
],
"user.username": "david.coley",
"tags.program": "web",
"tags.locale": "en",
"tags.feature_category": "integrations",
"tags.correlation_id": "01GZY4262WVNHVEQSPFBZB3EJ5"
}
What is the current bug behavior?
Webhook connection does not recognize no_proxy
configuration.
What is the expected correct behavior?
Webhooks execution should take the no_proxy settings into account.
Relevant logs and/or screenshots
{
"severity": "ERROR",
"time": "2023-05-08T16:45:42.450Z",
"correlation_id": "01GZY4262WVNHVEQSPFBZB3EJ5",
"exception.class": "Errno::ECONNRESET",
"exception.message": "Connection reset by peer - SSL_connect",
"exception.backtrace": [
"lib/gitlab/net_http_adapter.rb:21:in `connect'",
"lib/gitlab/http.rb:55:in `perform_request'",
"app/services/web_hook_service.rb:123:in `make_request'",
"app/services/web_hook_service.rb:72:in `execute'",
"app/models/hooks/web_hook.rb:57:in `execute'",
"app/controllers/concerns/web_hooks/hook_log_actions.rb:36:in `execute_hook'",
"app/controllers/concerns/web_hooks/hook_log_actions.rb:23:in `retry'",
"ee/lib/gitlab/ip_address_state.rb:10:in `with'",
"ee/app/controllers/ee/application_controller.rb:45:in `set_current_ip_address'",
"app/controllers/application_controller.rb:524:in `set_current_admin'",
"lib/gitlab/session.rb:11:in `with_session'",
"app/controllers/application_controller.rb:515:in `set_session_storage'",
"lib/gitlab/i18n.rb:107:in `with_locale'",
"lib/gitlab/i18n.rb:113:in `with_user_locale'",
"app/controllers/application_controller.rb:506:in `set_locale'",
"app/controllers/application_controller.rb:499:in `set_current_context'",
"lib/gitlab/middleware/memory_report.rb:13:in `call'",
"lib/gitlab/middleware/speedscope.rb:13:in `call'",
"lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
"lib/gitlab/jira/middleware.rb:19:in `call'",
"lib/gitlab/middleware/go.rb:20:in `call'",
"lib/gitlab/etag_caching/middleware.rb:21:in `call'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'",
"lib/gitlab/database/query_analyzer.rb:37:in `within'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `call'",
"lib/gitlab/middleware/multipart.rb:173:in `call'",
"lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
"lib/gitlab/middleware/read_only.rb:18:in `call'",
"lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
"lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
"lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
"lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
"lib/gitlab/middleware/request_context.rb:21:in `call'",
"lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
"config/initializers/fix_local_cache_middleware.rb:11:in `call'",
"lib/gitlab/middleware/compressed_json.rb:37:in `call'",
"lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
"lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
"lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'",
"lib/gitlab/middleware/release_env.rb:13:in `call'"
],
"user.username": "david.coley",
"tags.program": "web",
"tags.locale": "en",
"tags.feature_category": "integrations",
"tags.correlation_id": "01GZY4262WVNHVEQSPFBZB3EJ5"
}
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System: Ubuntu 20.04 Proxy: https_proxy: http://10.0.54.24:8080 no_proxy: localhost,127.0.0.1,10.,172.,.ethoca.com,.ethocaweb.com,.ethoca.io http_proxy: http://10.0.54.24:8080 Current User: git Using RVM: no Ruby Version: 3.0.6p216 Gem Version: 3.2.33 Bundler Version:2.3.15 Rake Version: 13.0.6 Redis Version: 6.2.11 Sidekiq Version:6.5.7 Go Version: unknown GitLab information Version: 15.11.2-ee Revision: 916d24d1e48 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 13.8 URL: https://gitlab.ethoca.com HTTP Clone URL: https://gitlab.ethoca.com/some-group/some-project.git SSH Clone URL: git@gitlab.ethoca.com:some-group/some-project.git Elasticsearch: yes Geo: no Using LDAP: yes Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 14.18.0 Repository storages: - default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Results of GitLab application Check
Expand for output related to the GitLab application check
Redis version >= 6.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (3.0.6) Git user has default SSH configuration? ... yes Active users: ... 241 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... yes (elasticsearch 7.17.7) All migrations must be finished before doing a major upgrade ... yes
Possible fixes
Disable the "Enforce DNS rebinding attack protection" per this comment on the related issue.