Return metav1.Status on Kubernetes proxy errors (!657) · Merge requests · GitLab.org / cluster-integration / GitLab Agent for Kubernetes

Mikhail Mazurskiy requested to merge ash2k/status-response into master May 24, 2022

Closes gitlab-org/gitlab#338305 (closed).

Current behavior:

➜  ~ k get pods
error: You must be logged in to the server (the server has asked for the client to provide credentials)

➜  ~ k -v 9 get pods
I0524 20:40:24.836334   29828 loader.go:372] Config loaded from file:  ~/.kube/config
I0524 20:40:24.836798   29828 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.24.0 (darwin/arm64) kubernetes/4ce5a89" -H "Authorization: Bearer <masked>" 'https://127.0.0.1:8154/api?timeout=32s'
I0524 20:40:24.837225   29828 round_trippers.go:510] HTTP Trace: Dial to tcp:127.0.0.1:8154 succeed
I0524 20:40:24.863423   29828 round_trippers.go:553] GET https://127.0.0.1:8154/api?timeout=32s 401 Unauthorized in 26 milliseconds
I0524 20:40:24.863439   29828 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 25 ms ServerProcessing 0 ms Duration 26 ms
I0524 20:40:24.863445   29828 round_trippers.go:577] Response Headers:
I0524 20:40:24.863450   29828 round_trippers.go:580]     Server: gitlab-kas/v0.0.0/00000000
I0524 20:40:24.863454   29828 round_trippers.go:580]     X-Content-Type-Options: nosniff
I0524 20:40:24.863459   29828 round_trippers.go:580]     X-Request-Id: 01G3TTEM8ZZWQ8EKTWM2CSHPJ3
I0524 20:40:24.863463   29828 round_trippers.go:580]     Date: Tue, 24 May 2022 10:40:24 GMT
I0524 20:40:24.863467   29828 round_trippers.go:580]     Content-Length: 36
I0524 20:40:24.863472   29828 round_trippers.go:580]     Content-Type: text/plain; charset=utf-8
I0524 20:40:24.883270   29828 request.go:1073] Response Body: Authorization header: invalid value
I0524 20:40:24.902200   29828 request.go:1264] body was not decodable (unable to check for Status): Object 'Kind' is missing in 'Authorization header: invalid value
'
I0524 20:40:24.902220   29828 cached_discovery.go:119] skipped caching discovery info due to the server has asked for the client to provide credentials
I0524 20:40:24.902301   29828 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.24.0 (darwin/arm64) kubernetes/4ce5a89" -H "Authorization: Bearer <masked>" 'https://127.0.0.1:8154/api?timeout=32s'
I0524 20:40:24.903091   29828 round_trippers.go:553] GET https://127.0.0.1:8154/api?timeout=32s 401 Unauthorized in 0 milliseconds
I0524 20:40:24.903103   29828 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 0 ms
I0524 20:40:24.903109   29828 round_trippers.go:577] Response Headers:
I0524 20:40:24.903115   29828 round_trippers.go:580]     X-Content-Type-Options: nosniff
I0524 20:40:24.903120   29828 round_trippers.go:580]     X-Request-Id: 01G3TTEMA60E4ZXVA2C0QNYE1B
I0524 20:40:24.903125   29828 round_trippers.go:580]     Date: Tue, 24 May 2022 10:40:24 GMT
I0524 20:40:24.903130   29828 round_trippers.go:580]     Content-Length: 36
I0524 20:40:24.903135   29828 round_trippers.go:580]     Content-Type: text/plain; charset=utf-8
I0524 20:40:24.903140   29828 round_trippers.go:580]     Server: gitlab-kas/v0.0.0/00000000
I0524 20:40:24.922101   29828 request.go:1073] Response Body: Authorization header: invalid value
I0524 20:40:24.941260   29828 request.go:1264] body was not decodable (unable to check for Status): Object 'Kind' is missing in 'Authorization header: invalid value
'
I0524 20:40:24.941274   29828 cached_discovery.go:119] skipped caching discovery info due to the server has asked for the client to provide credentials
I0524 20:40:24.941287   29828 shortcut.go:89] Error loading discovery information: the server has asked for the client to provide credentials
I0524 20:40:24.941366   29828 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.24.0 (darwin/arm64) kubernetes/4ce5a89" -H "Authorization: Bearer <masked>" 'https://127.0.0.1:8154/api?timeout=32s'
I0524 20:40:24.941989   29828 round_trippers.go:553] GET https://127.0.0.1:8154/api?timeout=32s 401 Unauthorized in 0 milliseconds
I0524 20:40:24.941999   29828 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 0 ms
I0524 20:40:24.942008   29828 round_trippers.go:577] Response Headers:
I0524 20:40:24.942014   29828 round_trippers.go:580]     X-Request-Id: 01G3TTEMBDSC2H86E15NJKMD6C
I0524 20:40:24.942019   29828 round_trippers.go:580]     Date: Tue, 24 May 2022 10:40:24 GMT
I0524 20:40:24.942025   29828 round_trippers.go:580]     Content-Length: 36
I0524 20:40:24.942031   29828 round_trippers.go:580]     Content-Type: text/plain; charset=utf-8
I0524 20:40:24.942036   29828 round_trippers.go:580]     Server: gitlab-kas/v0.0.0/00000000
I0524 20:40:24.942041   29828 round_trippers.go:580]     X-Content-Type-Options: nosniff
I0524 20:40:24.960181   29828 request.go:1073] Response Body: Authorization header: invalid value
I0524 20:40:24.980410   29828 request.go:1264] body was not decodable (unable to check for Status): Object 'Kind' is missing in 'Authorization header: invalid value
'
I0524 20:40:24.980436   29828 cached_discovery.go:119] skipped caching discovery info due to the server has asked for the client to provide credentials
I0524 20:40:24.980550   29828 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.24.0 (darwin/arm64) kubernetes/4ce5a89" -H "Authorization: Bearer <masked>" 'https://127.0.0.1:8154/api?timeout=32s'
I0524 20:40:24.981403   29828 round_trippers.go:553] GET https://127.0.0.1:8154/api?timeout=32s 401 Unauthorized in 0 milliseconds
I0524 20:40:24.981420   29828 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 0 ms
I0524 20:40:24.981426   29828 round_trippers.go:577] Response Headers:
I0524 20:40:24.981432   29828 round_trippers.go:580]     Content-Length: 36
I0524 20:40:24.981438   29828 round_trippers.go:580]     Content-Type: text/plain; charset=utf-8
I0524 20:40:24.981445   29828 round_trippers.go:580]     Server: gitlab-kas/v0.0.0/00000000
I0524 20:40:24.981450   29828 round_trippers.go:580]     X-Content-Type-Options: nosniff
I0524 20:40:24.981456   29828 round_trippers.go:580]     X-Request-Id: 01G3TTEMCNTD9H79EFH22RXY63
I0524 20:40:24.981461   29828 round_trippers.go:580]     Date: Tue, 24 May 2022 10:40:24 GMT
I0524 20:40:25.001150   29828 request.go:1073] Response Body: Authorization header: invalid value
I0524 20:40:25.021818   29828 request.go:1264] body was not decodable (unable to check for Status): Object 'Kind' is missing in 'Authorization header: invalid value
'
I0524 20:40:25.021857   29828 cached_discovery.go:119] skipped caching discovery info due to the server has asked for the client to provide credentials
I0524 20:40:25.022029   29828 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.24.0 (darwin/arm64) kubernetes/4ce5a89" -H "Authorization: Bearer <masked>" 'https://127.0.0.1:8154/api?timeout=32s'
I0524 20:40:25.023248   29828 round_trippers.go:553] GET https://127.0.0.1:8154/api?timeout=32s 401 Unauthorized in 1 milliseconds
I0524 20:40:25.023262   29828 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 1 ms
I0524 20:40:25.023270   29828 round_trippers.go:577] Response Headers:
I0524 20:40:25.023279   29828 round_trippers.go:580]     Date: Tue, 24 May 2022 10:40:25 GMT
I0524 20:40:25.023286   29828 round_trippers.go:580]     Content-Length: 36
I0524 20:40:25.023292   29828 round_trippers.go:580]     Content-Type: text/plain; charset=utf-8
I0524 20:40:25.023299   29828 round_trippers.go:580]     Server: gitlab-kas/v0.0.0/00000000
I0524 20:40:25.023305   29828 round_trippers.go:580]     X-Content-Type-Options: nosniff
I0524 20:40:25.023311   29828 round_trippers.go:580]     X-Request-Id: 01G3TTEMDYZBPGX37MD6YNW28Z
I0524 20:40:25.043278   29828 request.go:1073] Response Body: Authorization header: invalid value
I0524 20:40:25.063848   29828 request.go:1264] body was not decodable (unable to check for Status): Object 'Kind' is missing in 'Authorization header: invalid value
'
I0524 20:40:25.063913   29828 cached_discovery.go:119] skipped caching discovery info due to the server has asked for the client to provide credentials
I0524 20:40:25.064161   29828 helpers.go:222] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server has asked for the client to provide credentials",
  "reason": "Unauthorized",
  "details": {
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "Authorization header: invalid value"
      }
    ]
  },
  "code": 401
}]
error: You must be logged in to the server (the server has asked for the client to provide credentials)

As you can see, kubectl is trying and failing to parse kas' response - I0524 20:40:24.902200 29828 request.go:1264] body was not decodable (unable to check for Status): Object 'Kind' is missing in 'Authorization header: invalid value'. User also gets a generic error: You must be logged in to the server (the server has asked for the client to provide credentials) rather than what kas actually sends in the response body Authorization header: invalid value.

New behavior:

➜  ~ k get pods
error: You must be logged in to the server (GitLab Agent Server: Unauthorized: Authorization header: invalid value. Correlation ID: 01G3TTMA86JFW3SBS31WAVDHG0)

➜  ~ k -v 9 get pods
I0524 20:43:43.985391   29895 loader.go:372] Config loaded from file:  ~/.kube/config
I0524 20:43:43.986389   29895 round_trippers.go:466] curl -v -XGET  -H "User-Agent: kubectl/v1.24.0 (darwin/arm64) kubernetes/4ce5a89" -H "Authorization: Bearer <masked>" -H "Accept: application/json, */*" 'https://127.0.0.1:8154/api?timeout=32s'
I0524 20:43:43.987981   29895 round_trippers.go:510] HTTP Trace: Dial to tcp:127.0.0.1:8154 succeed
I0524 20:43:44.017253   29895 round_trippers.go:553] GET https://127.0.0.1:8154/api?timeout=32s 401 Unauthorized in 30 milliseconds
I0524 20:43:44.017264   29895 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 28 ms ServerProcessing 0 ms Duration 30 ms
I0524 20:43:44.017271   29895 round_trippers.go:577] Response Headers:
I0524 20:43:44.017275   29895 round_trippers.go:580]     Content-Type: application/json
I0524 20:43:44.017279   29895 round_trippers.go:580]     Server: gitlab-kas/v0.0.0/00000000
I0524 20:43:44.017282   29895 round_trippers.go:580]     X-Request-Id: 01G3TTMPRGC4Z7GJ33W6ZB5JXZ
I0524 20:43:44.017285   29895 round_trippers.go:580]     Date: Tue, 24 May 2022 10:43:44 GMT
I0524 20:43:44.017288   29895 round_trippers.go:580]     Content-Length: 231
I0524 20:43:44.038388   29895 request.go:1073] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"GitLab Agent Server: Unauthorized: Authorization header: invalid value. Correlation ID: 01G3TTMPRGC4Z7GJ33W6ZB5JXZ","reason":"Unauthorized","code":401}
I0524 20:43:44.056407   29895 cached_discovery.go:119] skipped caching discovery info due to GitLab Agent Server: Unauthorized: Authorization header: invalid value. Correlation ID: 01G3TTMPRGC4Z7GJ33W6ZB5JXZ
I0524 20:43:44.056498   29895 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "Authorization: Bearer <masked>" -H "User-Agent: kubectl/v1.24.0 (darwin/arm64) kubernetes/4ce5a89" 'https://127.0.0.1:8154/api?timeout=32s'
I0524 20:43:44.059073   29895 round_trippers.go:553] GET https://127.0.0.1:8154/api?timeout=32s 401 Unauthorized in 2 milliseconds
I0524 20:43:44.059084   29895 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 2 ms
I0524 20:43:44.059088   29895 round_trippers.go:577] Response Headers:
I0524 20:43:44.059092   29895 round_trippers.go:580]     Content-Type: application/json
I0524 20:43:44.059097   29895 round_trippers.go:580]     Server: gitlab-kas/v0.0.0/00000000
I0524 20:43:44.059100   29895 round_trippers.go:580]     X-Request-Id: 01G3TTMPSRDKN4DKB28MVTMCV7
I0524 20:43:44.059104   29895 round_trippers.go:580]     Date: Tue, 24 May 2022 10:43:44 GMT
I0524 20:43:44.059109   29895 round_trippers.go:580]     Content-Length: 231
I0524 20:43:44.082077   29895 request.go:1073] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"GitLab Agent Server: Unauthorized: Authorization header: invalid value. Correlation ID: 01G3TTMPSRDKN4DKB28MVTMCV7","reason":"Unauthorized","code":401}
I0524 20:43:44.101247   29895 cached_discovery.go:119] skipped caching discovery info due to GitLab Agent Server: Unauthorized: Authorization header: invalid value. Correlation ID: 01G3TTMPSRDKN4DKB28MVTMCV7
I0524 20:43:44.101500   29895 shortcut.go:89] Error loading discovery information: GitLab Agent Server: Unauthorized: Authorization header: invalid value. Correlation ID: 01G3TTMPSRDKN4DKB28MVTMCV7
I0524 20:43:44.101556   29895 round_trippers.go:466] curl -v -XGET  -H "User-Agent: kubectl/v1.24.0 (darwin/arm64) kubernetes/4ce5a89" -H "Authorization: Bearer <masked>" -H "Accept: application/json, */*" 'https://127.0.0.1:8154/api?timeout=32s'
I0524 20:43:44.102167   29895 round_trippers.go:553] GET https://127.0.0.1:8154/api?timeout=32s 401 Unauthorized in 0 milliseconds
I0524 20:43:44.102176   29895 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 0 ms
I0524 20:43:44.102180   29895 round_trippers.go:577] Response Headers:
I0524 20:43:44.102185   29895 round_trippers.go:580]     Content-Length: 231
I0524 20:43:44.102189   29895 round_trippers.go:580]     Content-Type: application/json
I0524 20:43:44.102194   29895 round_trippers.go:580]     Server: gitlab-kas/v0.0.0/00000000
I0524 20:43:44.102198   29895 round_trippers.go:580]     X-Request-Id: 01G3TTMPV5C52VB87VPS081815
I0524 20:43:44.102202   29895 round_trippers.go:580]     Date: Tue, 24 May 2022 10:43:44 GMT
I0524 20:43:44.104077   29895 request.go:1073] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"GitLab Agent Server: Unauthorized: Authorization header: invalid value. Correlation ID: 01G3TTMPV5C52VB87VPS081815","reason":"Unauthorized","code":401}
I0524 20:43:44.123329   29895 cached_discovery.go:119] skipped caching discovery info due to GitLab Agent Server: Unauthorized: Authorization header: invalid value. Correlation ID: 01G3TTMPV5C52VB87VPS081815
I0524 20:43:44.123443   29895 round_trippers.go:466] curl -v -XGET  -H "User-Agent: kubectl/v1.24.0 (darwin/arm64) kubernetes/4ce5a89" -H "Accept: application/json, */*" -H "Authorization: Bearer <masked>" 'https://127.0.0.1:8154/api?timeout=32s'
I0524 20:43:44.124380   29895 round_trippers.go:553] GET https://127.0.0.1:8154/api?timeout=32s 401 Unauthorized in 0 milliseconds
I0524 20:43:44.124394   29895 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 0 ms
I0524 20:43:44.124399   29895 round_trippers.go:577] Response Headers:
I0524 20:43:44.124405   29895 round_trippers.go:580]     Content-Length: 231
I0524 20:43:44.124409   29895 round_trippers.go:580]     Content-Type: application/json
I0524 20:43:44.124414   29895 round_trippers.go:580]     Server: gitlab-kas/v0.0.0/00000000
I0524 20:43:44.124419   29895 round_trippers.go:580]     X-Request-Id: 01G3TTMPVV7627JZMX0D9VQT3N
I0524 20:43:44.124424   29895 round_trippers.go:580]     Date: Tue, 24 May 2022 10:43:44 GMT
I0524 20:43:44.143208   29895 request.go:1073] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"GitLab Agent Server: Unauthorized: Authorization header: invalid value. Correlation ID: 01G3TTMPVV7627JZMX0D9VQT3N","reason":"Unauthorized","code":401}
I0524 20:43:44.162461   29895 cached_discovery.go:119] skipped caching discovery info due to GitLab Agent Server: Unauthorized: Authorization header: invalid value. Correlation ID: 01G3TTMPVV7627JZMX0D9VQT3N
I0524 20:43:44.162572   29895 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.24.0 (darwin/arm64) kubernetes/4ce5a89" -H "Authorization: Bearer <masked>" 'https://127.0.0.1:8154/api?timeout=32s'
I0524 20:43:44.163716   29895 round_trippers.go:553] GET https://127.0.0.1:8154/api?timeout=32s 401 Unauthorized in 1 milliseconds
I0524 20:43:44.163726   29895 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 1 ms
I0524 20:43:44.163731   29895 round_trippers.go:577] Response Headers:
I0524 20:43:44.163737   29895 round_trippers.go:580]     Content-Length: 231
I0524 20:43:44.163741   29895 round_trippers.go:580]     Content-Type: application/json
I0524 20:43:44.163746   29895 round_trippers.go:580]     Server: gitlab-kas/v0.0.0/00000000
I0524 20:43:44.163750   29895 round_trippers.go:580]     X-Request-Id: 01G3TTMPX3MB77B6FY24SF9V6E
I0524 20:43:44.163755   29895 round_trippers.go:580]     Date: Tue, 24 May 2022 10:43:44 GMT
I0524 20:43:44.183170   29895 request.go:1073] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"GitLab Agent Server: Unauthorized: Authorization header: invalid value. Correlation ID: 01G3TTMPX3MB77B6FY24SF9V6E","reason":"Unauthorized","code":401}
I0524 20:43:44.203349   29895 cached_discovery.go:119] skipped caching discovery info due to GitLab Agent Server: Unauthorized: Authorization header: invalid value. Correlation ID: 01G3TTMPX3MB77B6FY24SF9V6E
I0524 20:43:44.203556   29895 helpers.go:222] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "GitLab Agent Server: Unauthorized: Authorization header: invalid value. Correlation ID: 01G3TTMPX3MB77B6FY24SF9V6E",
  "reason": "Unauthorized",
  "code": 401
}]
error: You must be logged in to the server (GitLab Agent Server: Unauthorized: Authorization header: invalid value. Correlation ID: 01G3TTMPX3MB77B6FY24SF9V6E)

Improvements:

GitLab Agent Server in the message allows to distinguish kas-generated errors vs upstream-generated errors.
Actual error message is displayed to the user.
No parsing errors.
Correlation ID is displayed.

Return metav1.Status on Kubernetes proxy errors

Merge request reports