Click to expand or collapse the Incident Review section.
Incident Review
Summary
Service(s) affected:
Team attribution:
Minutes downtime or degradation:
Metrics
Customer Impact
Who was impacted by this incident? (i.e. external customers, internal customers)
What was the customer experience during the incident? (i.e. preventing them from doing X, incorrect display of Y, ...)
How many customers were affected?
If a precise customer impact number is unknown, what is the estimated potential impact?
Incident Response Analysis
How was the event detected?
How could detection time be improved?
How did we reach the point where we knew how to mitigate the impact?
How could time to mitigation be improved?
Post Incident Analysis
How was the root cause diagnosed?
How could time to diagnosis be improved?
Do we have an existing backlog item that would've prevented or greatly reduced the impact of this incident?
Was this incident triggered by a change (deployment of code or change to infrastructure. If yes, have you linked the issue which represents the change?)?
Not sure if we need to take action on it or not, but here it is WIP from my shift gitlab-org/gitlab!35898 (diffs) it adds a test to check the error on pagination and a possible solution
We don't see any of the errors from the projects endpoint in Sentry, and we don't see them in the logs. This could be a middleware as those have this issue: gitlab-org/gitlab#220768 (closed)
It looks like this problem happens on revoked tokens (Gitlab::Auth::RevokedError):
class is NilClass, error is Gitlab::Auth::RevokedError, /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/middleware/error.rb:49:in `rescue in call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/middleware/error.rb:37:in `call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/middleware/base.rb:28:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:60:in `block in call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:58:in `catch'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:58:in `call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/middleware/base.rb:28:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/head.rb:12:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/endpoint.rb:229:in `call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/endpoint.rb:223:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/router/route.rb:58:in `exec'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/router.rb:127:in `process_route'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/router.rb:80:in `block in identity'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/router.rb:99:in `transaction'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/router.rb:78:in `identity'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/router.rb:63:in `block in call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/router.rb:143:in `with_optimization'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/router.rb:62:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/api/instance.rb:167:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/api/instance.rb:71:in `call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/api/instance.rb:66:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:49:in `block in serve'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in `each'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in `serve'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:834:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:420:in `call_app!'/opt/gitlab/embedded/service/gitlab-rails/ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:190:in `call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_profiler/middleware.rb:17:in `call'/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/database/load_balancing/rack_middleware.rb:39:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.12.0/lib/labkit/middleware/rack.rb:19:in `block in call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.12.0/lib/labkit/context.rb:32:in `with_context'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.12.0/lib/labkit/middleware/rack.rb:18:in `call'/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/jira/middleware.rb:19:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/go.rb:20:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/etag_caching/middleware.rb:13:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/batch-loader-1.4.0/lib/batch_loader/middleware.rb:11:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-attack-6.3.0/lib/rack/attack.rb:97:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/apollo_upload_server-2.0.0.beta.3/lib/apollo_upload_server/middleware.rb:20:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/multipart.rb:125:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-attack-6.3.0/lib/rack/attack.rb:111:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.8/lib/warden/manager.rb:36:in `block in call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.8/lib/warden/manager.rb:34:in `catch'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.8/lib/warden/manager.rb:34:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-cors-1.0.6/lib/rack/cors.rb:98:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/tempfile_reaper.rb:15:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/etag.rb:25:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/conditional_get.rb:25:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/head.rb:12:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only/controller.rb:51:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only.rb:18:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/session/abstract/id.rb:259:in `context'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/session/abstract/id.rb:253:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/cookies.rb:648:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/same_site_cookies.rb:27:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:101:in `run_callbacks'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/actionable_exceptions.rb:17:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/basic_health_check.rb:25:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:37:in `call_app'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:26:in `block in call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/tagged_logging.rb:80:in `block in tagged'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/tagged_logging.rb:28:in `tagged'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/tagged_logging.rb:80:in `tagged'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:26:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/request_context.rb:23:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/request_store-1.5.0/lib/request_store/middleware.rb:19:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/method_override.rb:22:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/runtime.rb:22:in `call'/opt/gitlab/embedded/service/gitlab-rails/config/initializers/fix_local_cache_middleware.rb:9:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/executor.rb:14:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/lock.rb:17:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/sendfile.rb:111:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/host_authorization.rb:76:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:60:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sentry-raven-2.9.0/lib/raven/integrations/rack.rb:51:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/engine.rb:527:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-test-1.1.0/lib/rack/mock_session.rb:29:in `request'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-test-1.1.0/lib/rack/test.rb:266:in `process_request'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-test-1.1.0/lib/rack/test.rb:119:in `request'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/testing/integration.rb:270:in `process'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/testing/integration.rb:18:in `get'(irb):12:in `irb_binding'/opt/gitlab/embedded/lib/ruby/2.6.0/irb/workspace.rb:85:in `eval'/opt/gitlab/embedded/lib/ruby/2.6.0/irb/workspace.rb:85:in `evaluate'/opt/gitlab/embedded/lib/ruby/2.6.0/irb/context.rb:385:in `evaluate'/opt/gitlab/embedded/lib/ruby/2.6.0/irb.rb:493:in `block (2 levels) in eval_input'/opt/gitlab/embedded/lib/ruby/2.6.0/irb.rb:647:in `signal_status'/opt/gitlab/embedded/lib/ruby/2.6.0/irb.rb:490:in `block in eval_input'/opt/gitlab/embedded/lib/ruby/2.6.0/irb/ruby-lex.rb:246:in `block (2 levels) in each_top_level_statement'/opt/gitlab/embedded/lib/ruby/2.6.0/irb/ruby-lex.rb:232:in `loop'/opt/gitlab/embedded/lib/ruby/2.6.0/irb/ruby-lex.rb:232:in `block in each_top_level_statement'/opt/gitlab/embedded/lib/ruby/2.6.0/irb/ruby-lex.rb:231:in `catch'/opt/gitlab/embedded/lib/ruby/2.6.0/irb/ruby-lex.rb:231:in `each_top_level_statement'/opt/gitlab/embedded/lib/ruby/2.6.0/irb.rb:489:in `eval_input'/opt/gitlab/embedded/lib/ruby/2.6.0/irb.rb:428:in `block in run'/opt/gitlab/embedded/lib/ruby/2.6.0/irb.rb:427:in `catch'/opt/gitlab/embedded/lib/ruby/2.6.0/irb.rb:427:in `run'/opt/gitlab/embedded/lib/ruby/2.6.0/irb.rb:383:in `start'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/commands/console/console_command.rb:70:in `start'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/commands/console/console_command.rb:19:in `start'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/commands/console/console_command.rb:102:in `perform'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/command/base.rb:69:in `perform'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/command.rb:46:in `invoke'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/commands.rb:18:in `<top (required)>'bin/rails:4:in `require'bin/rails:4:in `<main>'error is here!class is Rack::Response, error is Grape::Exceptions::InvalidResponse, /tmp/test.rb:21:in `run_rescue_handler'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/middleware/error.rb:49:in `rescue in call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/middleware/error.rb:37:in `call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/middleware/base.rb:28:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:60:in `block in call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:58:in `catch'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:58:in `call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/middleware/base.rb:28:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/head.rb:12:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/endpoint.rb:229:in `call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/endpoint.rb:223:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/router/route.rb:58:in `exec'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/router.rb:127:in `process_route'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/router.rb:80:in `block in identity'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/router.rb:99:in `transaction'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/router.rb:78:in `identity'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/router.rb:63:in `block in call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/router.rb:143:in `with_optimization'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/router.rb:62:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/api/instance.rb:167:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/api/instance.rb:71:in `call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.3.3/lib/grape/api/instance.rb:66:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:49:in `block in serve'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in `each'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in `serve'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:834:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:420:in `call_app!'/opt/gitlab/embedded/service/gitlab-rails/ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:190:in `call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_profiler/middleware.rb:17:in `call'/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/database/load_balancing/rack_middleware.rb:39:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.12.0/lib/labkit/middleware/rack.rb:19:in `block in call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.12.0/lib/labkit/context.rb:32:in `with_context'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.12.0/lib/labkit/middleware/rack.rb:18:in `call'/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/jira/middleware.rb:19:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/go.rb:20:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/etag_caching/middleware.rb:13:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/batch-loader-1.4.0/lib/batch_loader/middleware.rb:11:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-attack-6.3.0/lib/rack/attack.rb:97:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/apollo_upload_server-2.0.0.beta.3/lib/apollo_upload_server/middleware.rb:20:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/multipart.rb:125:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-attack-6.3.0/lib/rack/attack.rb:111:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.8/lib/warden/manager.rb:36:in `block in call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.8/lib/warden/manager.rb:34:in `catch'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.8/lib/warden/manager.rb:34:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-cors-1.0.6/lib/rack/cors.rb:98:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/tempfile_reaper.rb:15:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/etag.rb:25:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/conditional_get.rb:25:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/head.rb:12:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only/controller.rb:51:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only.rb:18:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/session/abstract/id.rb:259:in `context'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/session/abstract/id.rb:253:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/cookies.rb:648:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/same_site_cookies.rb:27:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:101:in `run_callbacks'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/actionable_exceptions.rb:17:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/basic_health_check.rb:25:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:37:in `call_app'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:26:in `block in call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/tagged_logging.rb:80:in `block in tagged'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/tagged_logging.rb:28:in `tagged'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/tagged_logging.rb:80:in `tagged'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:26:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/request_context.rb:23:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/request_store-1.5.0/lib/request_store/middleware.rb:19:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/method_override.rb:22:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/runtime.rb:22:in `call'/opt/gitlab/embedded/service/gitlab-rails/config/initializers/fix_local_cache_middleware.rb:9:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/executor.rb:14:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/lock.rb:17:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.9/lib/rack/sendfile.rb:111:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/host_authorization.rb:76:in `call'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:60:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sentry-raven-2.9.0/lib/raven/integrations/rack.rb:51:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/engine.rb:527:in `call'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-test-1.1.0/lib/rack/mock_session.rb:29:in `request'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-test-1.1.0/lib/rack/test.rb:266:in `process_request'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-test-1.1.0/lib/rack/test.rb:119:in `request'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/testing/integration.rb:270:in `process'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.1/lib/action_dispatch/testing/integration.rb:18:in `get'(irb):12:in `irb_binding'/opt/gitlab/embedded/lib/ruby/2.6.0/irb/workspace.rb:85:in `eval'/opt/gitlab/embedded/lib/ruby/2.6.0/irb/workspace.rb:85:in `evaluate'/opt/gitlab/embedded/lib/ruby/2.6.0/irb/context.rb:385:in `evaluate'/opt/gitlab/embedded/lib/ruby/2.6.0/irb.rb:493:in `block (2 levels) in eval_input'/opt/gitlab/embedded/lib/ruby/2.6.0/irb.rb:647:in `signal_status'/opt/gitlab/embedded/lib/ruby/2.6.0/irb.rb:490:in `block in eval_input'/opt/gitlab/embedded/lib/ruby/2.6.0/irb/ruby-lex.rb:246:in `block (2 levels) in each_top_level_statement'/opt/gitlab/embedded/lib/ruby/2.6.0/irb/ruby-lex.rb:232:in `loop'/opt/gitlab/embedded/lib/ruby/2.6.0/irb/ruby-lex.rb:232:in `block in each_top_level_statement'/opt/gitlab/embedded/lib/ruby/2.6.0/irb/ruby-lex.rb:231:in `catch'/opt/gitlab/embedded/lib/ruby/2.6.0/irb/ruby-lex.rb:231:in `each_top_level_statement'/opt/gitlab/embedded/lib/ruby/2.6.0/irb.rb:489:in `eval_input'/opt/gitlab/embedded/lib/ruby/2.6.0/irb.rb:428:in `block in run'/opt/gitlab/embedded/lib/ruby/2.6.0/irb.rb:427:in `catch'/opt/gitlab/embedded/lib/ruby/2.6.0/irb.rb:427:in `run'/opt/gitlab/embedded/lib/ruby/2.6.0/irb.rb:383:in `start'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/commands/console/console_command.rb:70:in `start'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/commands/console/console_command.rb:19:in `start'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/commands/console/console_command.rb:102:in `perform'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/command/base.rb:69:in `perform'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/command.rb:46:in `invoke'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/commands.rb:18:in `<top (required)>'bin/rails:4:in `require'bin/rails:4:in `<main>'500{"Content-Type"=>"application/json", "Content-Length"=>"28"}#<Rack::BodyProxy:0x00007f3a9a3b9f58>=> 500
I redefined:
moduleGrapemoduleMiddlewareclassError<Basedefrun_rescue_handler(handler,error)ifhandler.instance_of?(Symbol)raiseNoMethodError,"undefined method `#{handler}'"unlessrespond_to?(handler)handler=public_method(handler)endresponse=handler.arity.zero??instance_exec(&handler):instance_exec(error,&handler)puts"class is #{response.class}, error is #{error}, #{caller.join("\n")}"putsresponseifresponse.is_a?(Rack::Response)responseelseputs"error is here!"run_rescue_handler(:default_rescue_handler,Grape::Exceptions::InvalidResponse.new)endendendendend
Ok, I reproduced it in my GDK and debugger and looking at it. Good news: This just means 401s are turning into 500s. This shouldn't affect our users, but it does affect our observability.
I believe this is being caused by a recent change to the VSCode GitLab Extension. https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/issues/211#note_372679165.
This thread gives an account on how authentication errors are being transformed to 500s so it does fit. Though we would have a problem anyways without the VSCode issue, it's just a nasty coincidence.
This incident was closed before the IncidentReview-Completed label
was applied. The issue is being reopened so that it will appear on the
Production Incidents
board—and can be moved through the entire incident management workflow.
Please review the Incident Workflow
section on the Incident Management handbook page for more information.
This incident was closed before the IncidentReview-Completed label
was applied. The issue is being reopened so that it will appear on the
Production Incidents
board—and can be moved through the entire incident management workflow.
Please review the Incident Workflow
section on the Incident Management handbook page for more information.