Uncaught errors from the kubernetes internal API are lost
Problem
Rails authenticates REST requests coming from agentk
using the Authorization
header (Auhtorization: Bearer <agent token>
). Aegntk uses a AgentToken
for authorization. The API plumbing expects that the token in this header is an OauthAccessToken
. This means that any error raised by the kubernetes API will not be reported. Instead, the attempt to report it will trigger a new error when the api_helpers
tries to call OauthAccessToken.by_token(token)
on the AgentToken
.
Alternative title: Authenticate agenk (GA4K) with a custom header, don't reuse the Authorization header
We found this error during Category:Remote Development implementation
Example of an error.
NoMethodError at /api/v4/internal/kubernetes/modules/remote_development
=======================================================================
undefined method `preferred_language' for #<Rack::Response:0x000000010af604b8 @status=401, @headers={"Content-Type"=>"application/json"}, @writer=#<Method: Rack::Response(Rack::Response::Helpers)#append(chunk) /Users/tomas/.asdf/installs/ruby/3.0.5/lib/ruby/gems/3.0.0/gems/rack-2.2.6.4/lib/rack/response.rb:287>, @block=nil, @body=["{\"message\":\"401 Unauthorized\"}"], @buffered=false, @length=0>
> To access an interactive console with this error, point your browser to: /__better_errors
lib/api/helpers.rb, line 79
74 def current_user
75 return @current_user if defined?(@current_user)
76
77 @current_user = initial_current_user
78
> 79 Gitlab::I18n.locale = @current_user&.preferred_language
80
81 sudo!
82
83 validate_access_token!(scopes: scopes_registered_for_endpoint) unless sudo?
84
App backtrace
-------------
- lib/api/helpers.rb:79:in `current_user'
- lib/api/helpers.rb:542:in `handle_api_exception'
- lib/api/api.rb:138:in `block in <class:API>'
- lib/gitlab/middleware/memory_report.rb:13:in `call'
- lib/gitlab/middleware/speedscope.rb:13:in `call'
- lib/gitlab/query_limiting/middleware.rb:17:in `block in call'
- lib/gitlab/query_limiting/transaction.rb:45:in `run'
- lib/gitlab/query_limiting/middleware.rb:16:in `call'
- lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
- lib/gitlab/jira/middleware.rb:19:in `call'
- lib/gitlab/middleware/go.rb:20:in `call'
- lib/gitlab/etag_caching/middleware.rb:21:in `call'
- lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
- lib/gitlab/database/query_analyzer.rb:37:in `within'
- lib/gitlab/middleware/query_analyzer.rb:11:in `call'
- lib/gitlab/middleware/multipart.rb:173:in `call'
- lib/gitlab/middleware/read_only/controller.rb:50:in `call'
- lib/gitlab/middleware/read_only.rb:18:in `call'
- lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
- lib/gitlab/middleware/basic_health_check.rb:25:in `call'
- lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
- lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
- lib/gitlab/middleware/request_context.rb:21:in `call'
- lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
- config/initializers/fix_local_cache_middleware.rb:11:in `call'
- lib/gitlab/middleware/compressed_json.rb:37:in `call'
- lib/gitlab/middleware/static.rb:11:in `call'
- lib/gitlab/webpack/dev_server_middleware.rb:34:in `perform_request'
- lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
- lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
- lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'
- lib/gitlab/middleware/release_env.rb:13:in `call'
Full backtrace
--------------
- lib/api/helpers.rb:79:in `current_user'
- lib/api/helpers.rb:542:in `handle_api_exception'
- lib/api/api.rb:138:in `block in <class:API>'
- grape (1.5.2) lib/grape/middleware/error.rb:129:in `run_rescue_handler'
- grape (1.5.2) lib/grape/middleware/error.rb:49:in `rescue in call!'
- grape (1.5.2) lib/grape/middleware/error.rb:37:in `call!'
- grape (1.5.2) lib/grape/middleware/base.rb:29:in `call'
- grape_logging (1.8.4) lib/grape_logging/middleware/request_logger.rb:60:in `block in call!'
- grape_logging (1.8.4) lib/grape_logging/middleware/request_logger.rb:58:in `call!'
- grape (1.5.2) lib/grape/middleware/base.rb:29:in `call'
- rack (2.2.6.4) lib/rack/head.rb:12:in `call'
- grape (1.5.2) lib/grape/endpoint.rb:231:in `call!'
- grape (1.5.2) lib/grape/endpoint.rb:225:in `call'
- grape (1.5.2) lib/grape/router/route.rb:58:in `exec'
- grape (1.5.2) lib/grape/router.rb:116:in `process_route'
- grape (1.5.2) lib/grape/router.rb:72:in `block in identity'
- grape (1.5.2) lib/grape/router.rb:91:in `transaction'
- grape (1.5.2) lib/grape/router.rb:70:in `identity'
- grape (1.5.2) lib/grape/router.rb:55:in `block in call'
- grape (1.5.2) lib/grape/router.rb:132:in `with_optimization'
- grape (1.5.2) lib/grape/router.rb:54:in `call'
- grape (1.5.2) lib/grape/api/instance.rb:167:in `call'
- grape (1.5.2) lib/grape/api/instance.rb:71:in `call!'
- grape (1.5.2) lib/grape/api/instance.rb:66:in `call'
- actionpack (6.1.7.2) lib/action_dispatch/routing/mapper.rb:20:in `block in <class:Constraints>'
- actionpack (6.1.7.2) lib/action_dispatch/routing/mapper.rb:49:in `serve'
- actionpack (6.1.7.2) lib/action_dispatch/journey/router.rb:50:in `block in serve'
- actionpack (6.1.7.2) lib/action_dispatch/journey/router.rb:32:in `serve'
- actionpack (6.1.7.2) lib/action_dispatch/routing/route_set.rb:842:in `call'
- gitlab-experiment (0.7.1) lib/gitlab/experiment/middleware.rb:19:in `call'
- omniauth (2.1.0) lib/omniauth/strategy.rb:202:in `call!'
- omniauth (2.1.0) lib/omniauth/strategy.rb:169:in `call'
- actionpack (6.1.7.2) lib/action_dispatch/middleware/static.rb:24:in `call'
- flipper (0.25.0) lib/flipper/middleware/memoizer.rb:72:in `memoized_call'
- flipper (0.25.0) lib/flipper/middleware/memoizer.rb:37:in `call'
- lib/gitlab/middleware/memory_report.rb:13:in `call'
- lib/gitlab/middleware/speedscope.rb:13:in `call'
- lib/gitlab/query_limiting/middleware.rb:17:in `block in call'
- lib/gitlab/query_limiting/transaction.rb:45:in `run'
- lib/gitlab/query_limiting/middleware.rb:16:in `call'
- lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
- lib/gitlab/jira/middleware.rb:19:in `call'
- lib/gitlab/middleware/go.rb:20:in `call'
- lib/gitlab/etag_caching/middleware.rb:21:in `call'
- lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
- lib/gitlab/database/query_analyzer.rb:37:in `within'
- lib/gitlab/middleware/query_analyzer.rb:11:in `call'
- batch-loader (2.0.1) lib/batch_loader/middleware.rb:11:in `call'
- bullet (7.0.2) lib/bullet/rack.rb:15:in `call'
- rack-attack (6.6.1) lib/rack/attack.rb:103:in `call'
- apollo_upload_server (2.1.0) lib/apollo_upload_server/middleware.rb:19:in `call'
- lib/gitlab/middleware/multipart.rb:173:in `call'
- rack (2.2.6.4) lib/rack/static.rb:161:in `call'
- rack-attack (6.6.1) lib/rack/attack.rb:127:in `call'
- warden (1.2.9) lib/warden/manager.rb:36:in `block in call'
- warden (1.2.9) lib/warden/manager.rb:34:in `call'
- rack-cors (1.1.1) lib/rack/cors.rb:100:in `call'
- rack (2.2.6.4) lib/rack/tempfile_reaper.rb:15:in `call'
- rack (2.2.6.4) lib/rack/etag.rb:27:in `call'
- rack (2.2.6.4) lib/rack/conditional_get.rb:40:in `call'
- rack (2.2.6.4) lib/rack/head.rb:12:in `call'
- actionpack (6.1.7.2) lib/action_dispatch/http/permissions_policy.rb:22:in `call'
- actionpack (6.1.7.2) lib/action_dispatch/http/content_security_policy.rb:19:in `call'
- lib/gitlab/middleware/read_only/controller.rb:50:in `call'
- lib/gitlab/middleware/read_only.rb:18:in `call'
- rack (2.2.6.4) lib/rack/session/abstract/id.rb:266:in `context'
- rack (2.2.6.4) lib/rack/session/abstract/id.rb:260:in `call'
- actionpack (6.1.7.2) lib/action_dispatch/middleware/cookies.rb:697:in `call'
- lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
- activerecord (6.1.7.2) lib/active_record/migration.rb:601:in `call'
- actionpack (6.1.7.2) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
- activesupport (6.1.7.2) lib/active_support/callbacks.rb:98:in `run_callbacks'
- actionpack (6.1.7.2) lib/action_dispatch/middleware/callbacks.rb:26:in `call'
- actionpack (6.1.7.2) lib/action_dispatch/middleware/executor.rb:14:in `call'
- actionpack (6.1.7.2) lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
- sentry-rails (5.8.0) lib/sentry/rails/rescued_exception_interceptor.rb:12:in `call'
- better_errors (2.9.1) lib/better_errors/middleware.rb:87:in `protected_app_call'
- better_errors (2.9.1) lib/better_errors/middleware.rb:82:in `better_errors_call'
- better_errors (2.9.1) lib/better_errors/middleware.rb:60:in `call'
- actionpack (6.1.7.2) lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
- sentry-ruby (5.8.0) lib/sentry/rack/capture_exceptions.rb:28:in `block (2 levels) in call'
- sentry-ruby (5.8.0) lib/sentry/hub.rb:220:in `with_session_tracking'
- sentry-ruby (5.8.0) lib/sentry-ruby.rb:375:in `with_session_tracking'
- sentry-ruby (5.8.0) lib/sentry/rack/capture_exceptions.rb:19:in `block in call'
- sentry-ruby (5.8.0) lib/sentry/hub.rb:59:in `with_scope'
- sentry-ruby (5.8.0) lib/sentry-ruby.rb:355:in `with_scope'
- sentry-ruby (5.8.0) lib/sentry/rack/capture_exceptions.rb:18:in `call'
- actionpack (6.1.7.2) lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
- lib/gitlab/middleware/basic_health_check.rb:25:in `call'
- lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
- railties (6.1.7.2) lib/rails/rack/logger.rb:37:in `call_app'
- railties (6.1.7.2) lib/rails/rack/logger.rb:26:in `block in call'
- activesupport (6.1.7.2) lib/active_support/tagged_logging.rb:99:in `block in tagged'
- activesupport (6.1.7.2) lib/active_support/tagged_logging.rb:37:in `tagged'
- activesupport (6.1.7.2) lib/active_support/tagged_logging.rb:99:in `tagged'
- railties (6.1.7.2) lib/rails/rack/logger.rb:26:in `call'
- sprockets-rails (3.4.2) lib/sprockets/rails/quiet_assets.rb:13:in `call'
- actionpack (6.1.7.2) lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
- lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
- lib/gitlab/middleware/request_context.rb:21:in `call'
- lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
- request_store (1.5.1) lib/request_store/middleware.rb:19:in `call'
- rack-timeout (0.6.3) lib/rack/timeout/core.rb:148:in `block in call'
- rack-timeout (0.6.3) lib/rack/timeout/support/timeout.rb:19:in `timeout'
- rack-timeout (0.6.3) lib/rack/timeout/core.rb:147:in `call'
- rack (2.2.6.4) lib/rack/method_override.rb:24:in `call'
- rack (2.2.6.4) lib/rack/runtime.rb:22:in `call'
- rack-timeout (0.6.3) lib/rack/timeout/core.rb:148:in `block in call'
- rack-timeout (0.6.3) lib/rack/timeout/support/timeout.rb:19:in `timeout'
- rack-timeout (0.6.3) lib/rack/timeout/core.rb:147:in `call'
- config/initializers/fix_local_cache_middleware.rb:11:in `call'
- lib/gitlab/middleware/compressed_json.rb:37:in `call'
- actionpack (6.1.7.2) lib/action_dispatch/middleware/executor.rb:14:in `call'
- actionpack (6.1.7.2) lib/action_dispatch/middleware/static.rb:24:in `call'
- lib/gitlab/middleware/static.rb:11:in `call'
- lib/gitlab/webpack/dev_server_middleware.rb:34:in `perform_request'
- rack-proxy (0.7.6) lib/rack/proxy.rb:87:in `call'
- lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
- rack (2.2.6.4) lib/rack/sendfile.rb:110:in `call'
- lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
- actionpack (6.1.7.2) lib/action_dispatch/middleware/host_authorization.rb:148:in `call'
- lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'
- gitlab-labkit (0.31.1) lib/labkit/middleware/rack.rb:19:in `block in call'
- gitlab-labkit (0.31.1) lib/labkit/context.rb:36:in `with_context'
- gitlab-labkit (0.31.1) lib/labkit/middleware/rack.rb:18:in `call'
- actionpack (6.1.7.2) lib/action_dispatch/middleware/request_id.rb:26:in `call'
- sentry-raven (3.1.2) lib/raven/integrations/rack.rb:51:in `call'
- railties (6.1.7.2) lib/rails/engine.rb:539:in `call'
- railties (6.1.7.2) lib/rails/railtie.rb:207:in `method_missing'
- lib/gitlab/middleware/release_env.rb:13:in `call'
- rack (2.2.6.4) lib/rack/urlmap.rb:74:in `block in call'
- rack (2.2.6.4) lib/rack/urlmap.rb:58:in `call'
- puma (5.6.5) lib/puma/configuration.rb:252:in `call'
- puma (5.6.5) lib/puma/request.rb:77:in `block in handle_request'
- puma (5.6.5) lib/puma/thread_pool.rb:340:in `with_force_shutdown'
- puma (5.6.5) lib/puma/request.rb:76:in `handle_request'
- puma (5.6.5) lib/puma/server.rb:443:in `process_client'
- puma (5.6.5) lib/puma/thread_pool.rb:147:in `block in spawn_thread'
instrument.js:108:32
Temporary fix
We early-exit from the helper method that crashes for all requests to the Kubernetes internal API. There is already a precedent for the PAT tokens.
Solution
The best solution would be not reusing the Authorization
header and using a custom agent token header (e.g. Gitlab-Agentk-Api-Request
). We already do that with the kas
, Workhorse, Pages, and few other services (full-text search for Api-Request
). The CI Runners use header called token
.
This would be implemented in 2 steps:
- Support both
Gitlab-Agentk-Api-Request
andAuthorization
header for authentication - Once a sufficient number of agents is migrated to the new header, we can stop supporting the authorization header and remove the Temporary fix.