GitLab Agent Server: Unauthorized error on staging

Summary

The client cluster connection is failing with an error GitLab Agent Server: Unauthorized on staging. The same setup works as expected on production. The request to the proxy is sent (for example, https://kas.staging.gitlab.com/k8s-proxy/api/v1/services), the KAS cookie and CRSF token are added to the request header together with the related agent id. The response:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "GitLab Agent Server: Unauthorized. Trace ID: b354eac73f93f27c676c52d760a433b1",
  "reason": "Unauthorized",
  "code": 401
}

Steps to reproduce

Create an agent configuration using the user_access.
Connect the agent to a cluster.
Create an environment and select the agent.
The namespace selector on the environment settings form will fail.
Go to the environments list and select the Kubernetes overview section.
All the cluster requests will fail.

Example Project

See the example project on staging.

What is the current bug behavior?

All the client - cluster requests are failing with an Unauthorized error.

What is the expected correct behavior?

The client should be able to request information from the cluster.

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info


(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)