OAuth error after upgrading 16.9.5 -> 16.11.0
Summary
We have a self-hosted Gitlab instance (Omnibus, as a Docker image) which uses Gitlab to authorize access to Pages. Our gitlab.rb looks like:
pages_external_url "http://pages.internal.domain/"
gitlab_pages['enable'] = true
gitlab_pages['external_http'] = ["0.0.0.0:5500"]
gitlab_pages['access_control'] = true
gitlab_pages['auth_redirect_uri'] = "https://docs.internal.domain/auth"
gitlab_pages['internal_gitlab_server'] = "http://127.0.0.1:8081"
After updating Gitlab to 16.11.0, gitlab-pages started to fail to parse internal authentication callbacks and throws an HTTP 500. The message from the logs is "token is malformed: token contains an invalid number of segments". See logs below.
gitlab-rails has no problem fulfilling the oauth request. I hit the pages domain, which forwards to /auth, which then forwards on to our-internal-gitlab.com/oauth/authorize, which then forwards back to pages.domain.com/auth?code=...
, which is where gitlab-pages then breaks trying to interpret the token.
This has worked splendidly for years, until we upgraded to 16.11.0 yesterday.
Steps to reproduce
- Configure gitlab-pages for internal authentication
- Attempt to access an internal-auth protect pages endpoint
- Observe the reported 500 error.
What is the current bug behavior?
gitlab-pages fails to parse the oauth token returned by gitlab-rails, resulting in an inability to access pages protected by gitlab authentication.
What is the expected correct behavior?
gitlab-pages should properly parse the returned oauth token, and should accept the authentication if valid
Relevant logs and/or screenshots
{"Namespace in path":"","Request host":"docs.internal.domain","Session host":null,"correlation_id":"01HVVF2GDJPQ9T45CEK6ZZTJSH","host":"docs.internal.domain","level":"info","msg":"Resetting session values","path":"/","state":"","time":"2024-04-19T15:19:50Z"}
{"Namespace in path":"","Request host":"docs.internal.domain","Session host":null,"correlation_id":"01HVVF2GDJPQ9T45CEK6ZZTJSH","host":"docs.internal.domain","level":"info","msg":"Resetting session values","path":"/","state":"","time":"2024-04-19T15:19:50Z"}
{"Namespace in path":"","Request host":"docs.internal.domain","Session host":null,"correlation_id":"01HVVF2GDJPQ9T45CEK6ZZTJSH","host":"docs.internal.domain","level":"info","msg":"Resetting session values","path":"/","state":"","time":"2024-04-19T15:19:50Z"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01HVVF2GDJPQ9T45CEK6ZZTJSH","duration_ms":665,"host":"docs.internal.domain","level":"info","method":"GET","msg":"access","pages_https":false,"proto":"HTTP/1.1","referrer":"","remote_addr":"172.17.0.1:52616","remote_ip":"172.17.0.1","status":302,"system":"http","time":"2024-04-19T15:19:50Z","ttfb_ms":666,"uri":"/","user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","written_bytes":116}
{"correlation_id":"01HVVF2H388EFCJVMDCM25770D","host":"docs.internal.domain","level":"info","msg":"Receive OAuth authentication callback","path":"/auth","state":"jcUP_t0TzfHsqWF12PU-Sg==","time":"2024-04-19T15:19:50Z"}
{"correlation_id":"01HVVF2H388EFCJVMDCM25770D","domain_query":"http://docs.internal.domain","host":"docs.internal.domain","level":"info","msg":"User is authenticating via domain","path":"/auth","state":"jcUP_t0TzfHsqWF12PU-Sg==","time":"2024-04-19T15:19:50Z"}
{"correlation_id":"01HVVF2H388EFCJVMDCM25770D","domain_query":"http://docs.internal.domain","host":"docs.internal.domain","level":"info","msg":"Redirecting user to gitlab for oauth","path":"/auth","public_gitlab_server":"https://gitlab.our.domain","state":"jcUP_t0TzfHsqWF12PU-Sg==","time":"2024-04-19T15:19:50Z"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01HVVF2H388EFCJVMDCM25770D","duration_ms":0,"host":"docs.internal.domain","level":"info","method":"GET","msg":"access","pages_https":false,"proto":"HTTP/1.1","referrer":"","remote_addr":"172.17.0.1:52640","remote_ip":"172.17.0.1","status":302,"system":"http","time":"2024-04-19T15:19:50Z","ttfb_ms":0,"uri":"/auth?domain=http://docs.internal.domain\u0026state=jcUP_t0TzfHsqWF12PU-Sg==","user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","written_bytes":257}
{"correlation_id":"01HVVF2HHCRFZG76Y6S7Z4EHXA","host":"docs.internal.domain","level":"info","msg":"Receive OAuth authentication callback","path":"/auth","state":"jcUP_t0TzfHsqWF12PU-Sg==","time":"2024-04-19T15:19:51Z"}
{"correlation_id":"01HVVF2HHCRFZG76Y6S7Z4EHXA","error":"token is malformed: token contains an invalid number of segments","host":"docs.internal.domain","level":"error","msg":"failed to decrypt secure code","path":"/auth","state":"jcUP_t0TzfHsqWF12PU-Sg==","time":"2024-04-19T15:19:51Z"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01HVVF2HHCRFZG76Y6S7Z4EHXA","duration_ms":0,"host":"docs.internal.domain","level":"info","method":"GET","msg":"access","pages_https":false,"proto":"HTTP/1.1","referrer":"https://gitlab.our.domain/","remote_addr":"172.17.0.1:52662","remote_ip":"172.17.0.1","status":500,"system":"http","time":"2024-04-19T15:19:51Z","ttfb_ms":0,"uri":"/auth?code=5819e8ad980c728a124bacc7862ea88b(...sanitized)\u0026state=jcUP_t0TzfHsqWF12PU-Sg%3D%3D","user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","written_bytes":2905}
{"content_type":"text/html; charset=utf-8","correlation_id":"01HVVF2HQCXC8KX5KNZGCRYJVG","duration_ms":0,"host":"docs.internal.domain","level":"info","method":"GET","msg":"access","pages_https":false,"proto":"HTTP/1.1","referrer":"https://docs.internal.domain/auth?code=5819e8ad980c728a124bacc7862ea88b(...sanitized)\u0026state=jcUP_t0TzfHsqWF12PU-Sg%3D%3D","remote_addr":"172.17.0.1:52666","remote_ip":"172.17.0.1","status":302,"system":"http","time":"2024-04-19T15:19:51Z","ttfb_ms":0,"uri":"/favicon.ico","user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","written_bytes":116}
{"correlation_id":"01HVVF2HRAG424SNZR4ZA3VB2V","host":"docs.internal.domain","level":"info","msg":"Receive OAuth authentication callback","path":"/auth","state":"jcUP_t0TzfHsqWF12PU-Sg==","time":"2024-04-19T15:19:51Z"}
{"correlation_id":"01HVVF2HRAG424SNZR4ZA3VB2V","domain_query":"http://docs.internal.domain","host":"docs.internal.domain","level":"info","msg":"User is authenticating via domain","path":"/auth","state":"jcUP_t0TzfHsqWF12PU-Sg==","time":"2024-04-19T15:19:51Z"}
{"correlation_id":"01HVVF2HRAG424SNZR4ZA3VB2V","domain_query":"http://docs.internal.domain","host":"docs.internal.domain","level":"info","msg":"Redirecting user to gitlab for oauth","path":"/auth","public_gitlab_server":"https://gitlab.our.domain","state":"jcUP_t0TzfHsqWF12PU-Sg==","time":"2024-04-19T15:19:51Z"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01HVVF2HRAG424SNZR4ZA3VB2V","duration_ms":0,"host":"docs.internal.domain","level":"info","method":"GET","msg":"access","pages_https":false,"proto":"HTTP/1.1","referrer":"https://docs.internal.domain/auth?code=5819e8ad980c728a124bacc7862ea88b(...sanitized)\u0026state=jcUP_t0TzfHsqWF12PU-Sg%3D%3D","remote_addr":"172.17.0.1:52670","remote_ip":"172.17.0.1","status":302,"system":"http","time":"2024-04-19T15:19:51Z","ttfb_ms":0,"uri":"/auth?domain=http://docs.internal.domain\u0026state=jcUP_t0TzfHsqWF12PU-Sg==","user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","written_bytes":257}
Output of checks
root@gitlab:/# gitlab-rake gitlab:check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.35.0 ? ... OK (14.35.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... Checking gitlab@ttiltd.net
yes
Mailroom enabled? ... skipped
MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
Tables are truncated? ... skipped
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
(trimmed: all yes)
Redis version >= 6.2.14? ... yes
Ruby version >= 3.0.6 ? ... yes (3.1.4)
Git user has default SSH configuration? ... yes
Active users: ... 36
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled)
All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
Unknown