Skip to content

Fix kubernetes internal API error handling

Tomas Vik requested to merge 406582-fix-kubernetes-api-error-handling into master

What does this MR do and why?

TL;DR: AgentToken is used in the Authorization header. The API plumbing only expects OauthAccessToken or PersonalAccessToken there and blows up with Unauthorized. This effectively disables all error logging for the internal kubernetes endpoint (#406582). Instead of logging the real error we log Unauthorized error that occurred during the error handling.

Any uncaught error in any API endpoint will trigger the lib/api/helpers.rb.handle_error method. This method tries to get the current_user from the OAuth header (Authorization: Bearer <token>).

This is the path for error handling that any kubernetes exception hits:

  1. handle_api_exception gets called
  2. We want to obtain current_user for better context
  3. current_user method tries to get initial_current_user
  4. initial_current_user calls find_current_user!
  5. find_current_user calls find_user_from_srouces we are getting closer 🔥
  6. find_user_from_srouces calls find_user_from_bearer_token now we are really close 🔥 🔥
  7. find_user_from_bearer_token class find_user_from_access_token almost 🔥 🔥 🔥
  8. find_user_from_access_token calls access_token and that's the method changed in this MR 🚀

How to set up and validate locally

  1. Install Rancher (or any other local Kubernetes cluster) on your local machine
  2. Add GitLab Agent Server (kas) to your GDK
  3. Install agent to your cluster
  4. Enable container scanning
  5. Add raise "test error" above this line, this will cause an error in the api/v4/internal/kubernetes/policies_configuration endpoint
  6. Now every time you start the agent, it asks for the configuration and you'll see the "test error" being thrown.
RuntimeError (test error):
  /Users/tomas/workspace/gl/gdk/gitlab/ee/app/services/security/security_orchestration_policies/operational_vulnerabilities_configuration_service.rb:13:in `execute'
  /Users/tomas/workspace/gl/gdk/gitlab/ee/lib/ee/api/internal/kubernetes.rb:114:in `block (5 levels) in <module:Kubernetes>'
  /Users/tomas/.asdf/installs/ruby/3.0.5/lib/ruby/gems/3.0.0/gems/grape-1.5.2/lib/grape/endpoint.rb:59:in `call

... rest of the stack
  1. If you test this setup on master without (this MR), you'll see the following error:
NoMethodError - undefined method `preferred_language' for #<Rack::Response:0x00000002d291e310 @status=401, @headers={"Content-Type"=>"application/json"}, @writer=#<Method: Rack::Response(Rack::Response::Helpers)#append(chunk) /Users/tomas/.asdf/installs/ruby/3.0.5/lib/ruby/gems/3.0.0/gems/rack-2.2.6.4/lib/rack/response.rb:287>, @block=nil, @body=["{\"message\":\"401 Unauthorized\"}"], @buffered=false, @length=0>:
  lib/api/helpers.rb:79:in `current_user'
  lib/api/helpers.rb:546:in `handle_api_exception'
  lib/api/api.rb:138:in `block in <class:API>'
  
... rest of the stack

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #406582

Edited by Tomas Vik

Merge request reports