Authentication with Azure AD in GCC high is failing
Summary
A customer has enabled authentication with Azure AD in GCC high. When they attempt to authenticate, they login, then when redirected back to GitLab receive a 500 error. When reviewing the logs a stack trace was observed:
Started GET "/users/auth/azure_activedirectory_v2/callback[SANITIZED]
NoMethodError (undefined method `include?' for nil:NilClass):
config/initializers/http_hostname_override.rb:42:in `addr_port'
config/initializers_before_autoloader/100_patch_omniauth_oauth2.rb:11:in `callback_phase'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/transaction.rb:56:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:172:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:21:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'
This was brought up in an internal slack thread. It was determined based off the stack trace that the method include?
was coming from an external library. From reviewing this, it is believed that address
is nil
, leading to the object having no methods.
Since this error is coming from an external library, and external library errors are silenced, it was requested that the customer comment out the silencer on their instance, restart then trigger the error.
This lead to a much more detailed stack trace, which is shown below:
NoMethodError (undefined method `include?' for nil:NilClass):
/opt/gitlab/embedded/lib/ruby/2.7.0/net/http.rb:1650:in `addr_port'
config/initializers/http_hostname_override.rb:42:in `addr_port'
/opt/gitlab/embedded/lib/ruby/2.7.0/net/http.rb:1585:in `begin_transport'
/opt/gitlab/embedded/lib/ruby/2.7.0/net/http.rb:1518:in `transport_request'
/opt/gitlab/embedded/lib/ruby/2.7.0/net/http.rb:1492:in `request'
gitlab-labkit (0.16.2) lib/labkit/net_http_publisher.rb:47:in `block in request'
activesupport (6.0.3.6) lib/active_support/notifications.rb:180:in `block in instrument'
activesupport (6.0.3.6) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
activesupport (6.0.3.6) lib/active_support/notifications.rb:180:in `instrument'
gitlab-labkit (0.16.2) lib/labkit/net_http_publisher.rb:44:in `request'
/opt/gitlab/embedded/lib/ruby/2.7.0/net/http.rb:1241:in `get'
faraday (1.0.1) lib/faraday/adapter/net_http.rb:145:in `block in request_via_get_method'
/opt/gitlab/embedded/lib/ruby/2.7.0/net/http.rb:933:in `start'
faraday (1.0.1) lib/faraday/adapter/net_http.rb:144:in `request_via_get_method'
faraday (1.0.1) lib/faraday/adapter/net_http.rb:135:in `request_with_wrapped_block'
faraday (1.0.1) lib/faraday/adapter/net_http.rb:128:in `perform_request'
faraday (1.0.1) lib/faraday/adapter/net_http.rb:70:in `block in call'
faraday (1.0.1) lib/faraday/adapter.rb:60:in `connection'
faraday (1.0.1) lib/faraday/adapter/net_http.rb:68:in `call'
faraday (1.0.1) lib/faraday/request/url_encoded.rb:25:in `call'
faraday (1.0.1) lib/faraday/rack_builder.rb:153:in `build_response'
faraday (1.0.1) lib/faraday/connection.rb:492:in `run_request'
oauth2 (1.4.4) lib/oauth2/client.rb:99:in `request'
oauth2 (1.4.4) lib/oauth2/client.rb:114:in `request'
oauth2 (1.4.4) lib/oauth2/access_token.rb:107:in `request'
oauth2 (1.4.4) lib/oauth2/access_token.rb:114:in `get'
omniauth-azure-activedirectory-v2 (0.1.1) lib/omniauth/strategies/azure_activedirectory_v2.rb:61:in `raw_info'
omniauth-azure-activedirectory-v2 (0.1.1) lib/omniauth/strategies/azure_activedirectory_v2.rb:43:in `block in <class:AzureActivedirectoryV2>'
omniauth (1.9.0) lib/omniauth/strategy.rb:109:in `instance_eval'
omniauth (1.9.0) lib/omniauth/strategy.rb:109:in `block in compile_stack'
omniauth (1.9.0) lib/omniauth/strategy.rb:108:in `each'
omniauth (1.9.0) lib/omniauth/strategy.rb:108:in `inject'
omniauth (1.9.0) lib/omniauth/strategy.rb:108:in `compile_stack'
omniauth (1.9.0) lib/omniauth/strategy.rb:102:in `uid_stack'
omniauth (1.9.0) lib/omniauth/strategy.rb:332:in `uid'
omniauth (1.9.0) lib/omniauth/strategy.rb:348:in `auth_hash'
omniauth (1.9.0) lib/omniauth/strategy.rb:372:in `callback_phase'
omniauth-oauth2 (1.6.0) lib/omniauth/strategies/oauth2.rb:75:in `callback_phase'
config/initializers_before_autoloader/100_patch_omniauth_oauth2.rb:11:in `callback_phase'
omniauth (1.9.0) lib/omniauth/strategy.rb:238:in `callback_call'
omniauth (1.9.0) lib/omniauth/strategy.rb:189:in `call!'
omniauth (1.9.0) lib/omniauth/strategy.rb:169:in `call'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/transaction.rb:56:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
batch-loader (2.0.1) lib/batch_loader/middleware.rb:11:in `call'
rack-attack (6.3.0) lib/rack/attack.rb:97:in `call'
apollo_upload_server (2.0.2) lib/apollo_upload_server/middleware.rb:20:in `call'
lib/gitlab/middleware/multipart.rb:172:in `call'
rack-attack (6.3.0) lib/rack/attack.rb:111:in `call'
warden (1.2.8) lib/warden/manager.rb:36:in `block in call'
warden (1.2.8) lib/warden/manager.rb:34:in `catch'
warden (1.2.8) lib/warden/manager.rb:34:in `call'
rack-cors (1.0.6) lib/rack/cors.rb:98:in `call'
rack (2.2.3) lib/rack/tempfile_reaper.rb:15:in `call'
rack (2.2.3) lib/rack/etag.rb:27:in `call'
rack (2.2.3) lib/rack/conditional_get.rb:27:in `call'
rack (2.2.3) lib/rack/head.rb:12:in `call'
actionpack (6.0.3.6) lib/action_dispatch/http/content_security_policy.rb:18:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
rack (2.2.3) lib/rack/session/abstract/id.rb:266:in `context'
rack (2.2.3) lib/rack/session/abstract/id.rb:260:in `call'
actionpack (6.0.3.6) lib/action_dispatch/middleware/cookies.rb:648:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
actionpack (6.0.3.6) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
activesupport (6.0.3.6) lib/active_support/callbacks.rb:101:in `run_callbacks'
actionpack (6.0.3.6) lib/action_dispatch/middleware/callbacks.rb:26:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
actionpack (6.0.3.6) lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
actionpack (6.0.3.6) lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
actionpack (6.0.3.6) lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
railties (6.0.3.6) lib/rails/rack/logger.rb:37:in `call_app'
railties (6.0.3.6) lib/rails/rack/logger.rb:26:in `block in call'
activesupport (6.0.3.6) lib/active_support/tagged_logging.rb:80:in `block in tagged'
activesupport (6.0.3.6) lib/active_support/tagged_logging.rb:28:in `tagged'
activesupport (6.0.3.6) lib/active_support/tagged_logging.rb:80:in `tagged'
railties (6.0.3.6) lib/rails/rack/logger.rb:26:in `call'
actionpack (6.0.3.6) lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
request_store (1.5.0) lib/request_store/middleware.rb:19:in `call'
rack (2.2.3) lib/rack/method_override.rb:24:in `call'
rack (2.2.3) lib/rack/runtime.rb:22:in `call'
rack-timeout (0.5.2) lib/rack/timeout/core.rb:123:in `block in call'
rack-timeout (0.5.2) lib/rack/timeout/support/timeout.rb:19:in `timeout'
rack-timeout (0.5.2) lib/rack/timeout/core.rb:122:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
actionpack (6.0.3.6) lib/action_dispatch/middleware/executor.rb:14:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:21:in `call'
rack (2.2.3) lib/rack/sendfile.rb:110:in `call'
actionpack (6.0.3.6) lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'
gitlab-labkit (0.16.2) lib/labkit/middleware/rack.rb:19:in `block in call'
gitlab-labkit (0.16.2) lib/labkit/context.rb:33:in `with_context'
gitlab-labkit (0.16.2) lib/labkit/middleware/rack.rb:18:in `call'
actionpack (6.0.3.6) lib/action_dispatch/middleware/request_id.rb:27:in `call'
sentry-raven (3.0.4) lib/raven/integrations/rack.rb:51:in `call'
railties (6.0.3.6) lib/rails/engine.rb:527:in `call'
railties (6.0.3.6) lib/rails/railtie.rb:190:in `public_send'
railties (6.0.3.6) lib/rails/railtie.rb:190:in `method_missing'
lib/gitlab/middleware/release_env.rb:12:in `call'
rack (2.2.3) lib/rack/urlmap.rb:74:in `block in call'
rack (2.2.3) lib/rack/urlmap.rb:58:in `each'
rack (2.2.3) lib/rack/urlmap.rb:58:in `call'
puma (5.1.1) lib/puma/configuration.rb:246:in `call'
puma (5.1.1) lib/puma/request.rb:76:in `block in handle_request'
puma (5.1.1) lib/puma/thread_pool.rb:337:in `with_force_shutdown'
puma (5.1.1) lib/puma/request.rb:75:in `handle_request'
puma (5.1.1) lib/puma/server.rb:431:in `process_client'
puma (5.1.1) lib/puma/thread_pool.rb:145:in `block in spawn_thread'
The customer is using a single node installation running version 13.11.3 on a single server behind an AWS classic load balancer. Port 443 goes through the load balancer, which forwards to port 80 of the GitLab instance.
Steps to reproduce
Example Project
What is the current bug behavior?
What is the expected correct behavior?
Relevant logs and/or screenshots
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)