Multiple duplicates of "https:" in CPS causing nginx errors

Summary

CSP header in Omnibus includes multiple repeats of https: part. There are so many of them that in our setup sometimes default proxy_buffer_size in nginx - the one used for headers, 4kB by default - is reached and request fails with HTTP 502.

Steps to reproduce

  • Enable CSP in Omnibus Gitlab (gitlab_rails['content_security_policy'] = {'enabled' => true})
  • Try opening the log in page.
  • Check headers.

What is the current bug behavior?

In our case the CSP header ends with multiple repeats of https: - there are 143 unnecessary repeats at the end.

This causes nginx to sometimes return 502 (depending on other headers added).

By default, nginx uses a single memory page size for proxy_buffer_size which is 4kB in our case. I've seen total headers size from ~3,2kB up to ~4,2kB. There is nearly a 1kB just in those repeated https: entries.

It seems that error occurs when the following header appears (I don't know when does Gitlab add it) - exact content likely depends on Gitlab version:

link: </assets/application_utilities-2531e4e6ef42e4af0a1b836590e0b362055984d2fa233ae3c5b07d7c4a2761fd.css>; rel=preload; as=style; type=text/css,</assets/application-f79ed5a6b0dfecf39281aeefcdd5b15d7cc6d871a3ae20e60c40d6a718377704.css>; rel=preload; as=style; type=text/css,</assets/highlight/themes/white-0163ec1ff3033e0ebaf2e7700680941596e39d73535518445a42947430b7d452.css>; rel=preload; as=style; type=text/css

What is the expected correct behavior?

  • CSP header doesn't include repeats.
  • Troubleshooting of this case improved:
    • As it may be possible to hit this limit with long enough domains used in rules, it would be nice to have an error/warning reported somewhere if there is a risk of hitting the limit.
    • Maybe default values used by nginx should be increased? Although they're likely set to single page for performance reasons.
    • Add this case to troubleshooting docs and mention it in the CSP documentation.

Relevant logs and/or screenshots

CSP header value (anonymized):

base-uri 'self'; child-src https://www.google.com/recaptcha/ https://www.recaptcha.net/ https://content.googleapis.com https://content-compute.googleapis.com https://content-cloudbilling.googleapis.com https://content-cloudresourcemanager.googleapis.com https://www.googletagmanager.com/ns.html https://example.com/assets/ https://example.com/-/speedscope/index.html https://example.com/-/sandbox/ https://example.com/assets/ blob: data:; connect-src 'self' wss://example.com https://example.net https://cdn.cookielaw.org https://*.onetrust.com *.google-analytics.com *.analytics.google.com *.googletagmanager.com; default-src 'self'; font-src 'self'; frame-ancestors 'self'; frame-src https://www.google.com/recaptcha/ https://www.recaptcha.net/ https://content.googleapis.com https://content-compute.googleapis.com https://content-cloudbilling.googleapis.com https://content-cloudresourcemanager.googleapis.com https://www.googletagmanager.com/ns.html https://example.com/admin/ https://example.com/assets/ https://example.com/-/speedscope/index.html https://example.com/-/sandbox/; img-src 'self' data: blob: http: https: *.google-analytics.com *.googletagmanager.com; manifest-src 'self'; media-src 'self' data: blob: http: https:; object-src 'none'; report-uri https://example.net/api/project_id/security/?sentry_key=key&sentry_environment=production&sentry_release=gitlab-16.4.1-ee.0; script-src 'strict-dynamic' 'self' 'unsafe-inline' 'unsafe-eval' https://www.google.com/recaptcha/ https://www.recaptcha.net https://apis.google.com https://cdn.cookielaw.org https://*.onetrust.com https://cdn.bizible.com/scripts/bizible.js *.googletagmanager.com 'nonce-ppmUKI5kfPfzP8iJHjuUOA=='; style-src 'self' 'unsafe-inline'; worker-src https://example.com/assets/ blob: data:; form-action 'self' https: http: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https:

Nginx logs the following error when issue occurs:

upstream sent too big header while reading response header from upstream

Output of checks

Results of GitLab environment info

Domains anonymized.

Omnibus with the following CSP configuration:

CSP config 
gitlab_rails['content_security_policy'] = {
  'enabled' => true,
  'report_only' => false,
 'directives' => {
   'report_uri' => 'https://example.net/api/project_id/security/?sentry_key=key&sentry_environment=production&sentry_release=gitlab-16.4.1-ee.0',
 }
}
Expand for output related to GitLab environment info 
System information
System:         Debian 10
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   3.0.6p216
Gem Version:    3.4.19
Bundler Version:2.4.19
Rake Version:   13.0.6
Redis Version:  7.0.13
Sidekiq Version:6.5.7
Go Version:     unknown

GitLab information
Version:        16.4.1-ee
Revision:       229bc5f5985
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     13.11
URL:            https://example.com
HTTP Clone URL: https://example.com/some-group/some-project.git
SSH Clone URL:  git@example.com:some-group/some-project.git
Elasticsearch:  yes
Geo:            yes
Geo node:       Primary
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers: 

GitLab Shell
Version:        14.28.0
Repository storages:
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address:      unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version:      16.4.1
- default Git Version:  2.42.0

Results of GitLab application Check

Not relevant.

Possible fixes

Increase proxy buffers in the nginx config, e.g.

proxy_busy_buffers_size   512k;
proxy_buffers   8 512k;
proxy_buffer_size 256k;