Multiple duplicates of "https:" in CPS causing nginx errors
Summary
CSP header in Omnibus includes multiple repeats of https:
part. There are so many of them that in our setup sometimes default proxy_buffer_size
in nginx - the one used for headers, 4kB by default - is reached and request fails with HTTP 502.
Steps to reproduce
- Enable CSP in Omnibus Gitlab (
gitlab_rails['content_security_policy'] = {'enabled' => true}
) - Try opening the log in page.
- Check headers.
What is the current bug behavior?
In our case the CSP header ends with multiple repeats of https:
- there are 143 unnecessary repeats at the end.
This causes nginx to sometimes return 502 (depending on other headers added).
By default, nginx uses a single memory page size for proxy_buffer_size
which is 4kB in our case. I've seen total headers size from ~3,2kB up to ~4,2kB. There is nearly a 1kB just in those repeated https:
entries.
It seems that error occurs when the following header appears (I don't know when does Gitlab add it) - exact content likely depends on Gitlab version:
link: </assets/application_utilities-2531e4e6ef42e4af0a1b836590e0b362055984d2fa233ae3c5b07d7c4a2761fd.css>; rel=preload; as=style; type=text/css,</assets/application-f79ed5a6b0dfecf39281aeefcdd5b15d7cc6d871a3ae20e60c40d6a718377704.css>; rel=preload; as=style; type=text/css,</assets/highlight/themes/white-0163ec1ff3033e0ebaf2e7700680941596e39d73535518445a42947430b7d452.css>; rel=preload; as=style; type=text/css
What is the expected correct behavior?
- CSP header doesn't include repeats.
- Troubleshooting of this case improved:
- As it may be possible to hit this limit with long enough domains used in rules, it would be nice to have an error/warning reported somewhere if there is a risk of hitting the limit.
- Maybe default values used by nginx should be increased? Although they're likely set to single page for performance reasons.
- Add this case to troubleshooting docs and mention it in the CSP documentation.
Relevant logs and/or screenshots
CSP header value (anonymized):
base-uri 'self'; child-src https://www.google.com/recaptcha/ https://www.recaptcha.net/ https://content.googleapis.com https://content-compute.googleapis.com https://content-cloudbilling.googleapis.com https://content-cloudresourcemanager.googleapis.com https://www.googletagmanager.com/ns.html https://example.com/assets/ https://example.com/-/speedscope/index.html https://example.com/-/sandbox/ https://example.com/assets/ blob: data:; connect-src 'self' wss://example.com https://example.net https://cdn.cookielaw.org https://*.onetrust.com *.google-analytics.com *.analytics.google.com *.googletagmanager.com; default-src 'self'; font-src 'self'; frame-ancestors 'self'; frame-src https://www.google.com/recaptcha/ https://www.recaptcha.net/ https://content.googleapis.com https://content-compute.googleapis.com https://content-cloudbilling.googleapis.com https://content-cloudresourcemanager.googleapis.com https://www.googletagmanager.com/ns.html https://example.com/admin/ https://example.com/assets/ https://example.com/-/speedscope/index.html https://example.com/-/sandbox/; img-src 'self' data: blob: http: https: *.google-analytics.com *.googletagmanager.com; manifest-src 'self'; media-src 'self' data: blob: http: https:; object-src 'none'; report-uri https://example.net/api/project_id/security/?sentry_key=key&sentry_environment=production&sentry_release=gitlab-16.4.1-ee.0; script-src 'strict-dynamic' 'self' 'unsafe-inline' 'unsafe-eval' https://www.google.com/recaptcha/ https://www.recaptcha.net https://apis.google.com https://cdn.cookielaw.org https://*.onetrust.com https://cdn.bizible.com/scripts/bizible.js *.googletagmanager.com 'nonce-ppmUKI5kfPfzP8iJHjuUOA=='; style-src 'self' 'unsafe-inline'; worker-src https://example.com/assets/ blob: data:; form-action 'self' https: http: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https:
Nginx logs the following error when issue occurs:
upstream sent too big header while reading response header from upstream
Output of checks
Results of GitLab environment info
Domains anonymized.
Omnibus with the following CSP configuration:
CSP config
gitlab_rails['content_security_policy'] = {
'enabled' => true,
'report_only' => false,
'directives' => {
'report_uri' => 'https://example.net/api/project_id/security/?sentry_key=key&sentry_environment=production&sentry_release=gitlab-16.4.1-ee.0',
}
}
Expand for output related to GitLab environment info
System information System: Debian 10 Proxy: no Current User: git Using RVM: no Ruby Version: 3.0.6p216 Gem Version: 3.4.19 Bundler Version:2.4.19 Rake Version: 13.0.6 Redis Version: 7.0.13 Sidekiq Version:6.5.7 Go Version: unknown GitLab information Version: 16.4.1-ee Revision: 229bc5f5985 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 13.11 URL: https://example.com HTTP Clone URL: https://example.com/some-group/some-project.git SSH Clone URL: git@example.com:some-group/some-project.git Elasticsearch: yes Geo: yes Geo node: Primary Using LDAP: yes Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 14.28.0 Repository storages: - default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Gitaly - default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket - default Version: 16.4.1 - default Git Version: 2.42.0
Results of GitLab application Check
Not relevant.
Possible fixes
Increase proxy buffers in the nginx config, e.g.
proxy_busy_buffers_size 512k;
proxy_buffers 8 512k;
proxy_buffer_size 256k;