<!--- Please read this! Before opening a new issue, make sure to search for keywords in the issues filtered by the "regression" or "type::bug" label: - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=type::bug and verify the issue you're about to submit isn't a duplicate. ---> ### Summary CSP header in Omnibus includes multiple repeats of `https:` part. There are so many of them that in our setup sometimes default `proxy_buffer_size` in nginx - the one used for headers, 4kB by default - is reached and request fails with HTTP 502. ### Steps to reproduce * Enable CSP in Omnibus Gitlab (`gitlab_rails['content_security_policy'] = {'enabled' => true}`) * Try opening the log in page. * Check headers. ### What is the current *bug* behavior? In our case the CSP header ends with multiple repeats of `https:` - there are 143 unnecessary repeats at the end. This causes nginx to _sometimes_ return 502 (depending on other headers added). By default, nginx uses a single memory page size for `proxy_buffer_size` which is 4kB in our case. I've seen total headers size from ~3,2kB up to ~4,2kB. There is nearly a 1kB just in those repeated `https:` entries. It seems that error occurs when the following header appears (I don't know when does Gitlab add it) - exact content likely depends on Gitlab version: ``` link: </assets/application_utilities-2531e4e6ef42e4af0a1b836590e0b362055984d2fa233ae3c5b07d7c4a2761fd.css>; rel=preload; as=style; type=text/css,</assets/application-f79ed5a6b0dfecf39281aeefcdd5b15d7cc6d871a3ae20e60c40d6a718377704.css>; rel=preload; as=style; type=text/css,</assets/highlight/themes/white-0163ec1ff3033e0ebaf2e7700680941596e39d73535518445a42947430b7d452.css>; rel=preload; as=style; type=text/css ``` ### What is the expected *correct* behavior? * CSP header doesn't include repeats. * Troubleshooting of this case improved: * As it may be possible to hit this limit with long enough domains used in rules, it would be nice to have an error/warning reported somewhere if there is a risk of hitting the limit. * Maybe default values used by nginx should be increased? Although they're likely set to single page for performance reasons. * Add this case to troubleshooting docs and mention it in the [CSP documentation](https://docs.gitlab.com/omnibus/settings/configuration.html#set-a-content-security-policy). ### Relevant logs and/or screenshots CSP header value (anonymized): ``` base-uri 'self'; child-src https://www.google.com/recaptcha/ https://www.recaptcha.net/ https://content.googleapis.com https://content-compute.googleapis.com https://content-cloudbilling.googleapis.com https://content-cloudresourcemanager.googleapis.com https://www.googletagmanager.com/ns.html https://example.com/assets/ https://example.com/-/speedscope/index.html https://example.com/-/sandbox/ https://example.com/assets/ blob: data:; connect-src 'self' wss://example.com https://example.net https://cdn.cookielaw.org https://*.onetrust.com *.google-analytics.com *.analytics.google.com *.googletagmanager.com; default-src 'self'; font-src 'self'; frame-ancestors 'self'; frame-src https://www.google.com/recaptcha/ https://www.recaptcha.net/ https://content.googleapis.com https://content-compute.googleapis.com https://content-cloudbilling.googleapis.com https://content-cloudresourcemanager.googleapis.com https://www.googletagmanager.com/ns.html https://example.com/admin/ https://example.com/assets/ https://example.com/-/speedscope/index.html https://example.com/-/sandbox/; img-src 'self' data: blob: http: https: *.google-analytics.com *.googletagmanager.com; manifest-src 'self'; media-src 'self' data: blob: http: https:; object-src 'none'; report-uri https://example.net/api/project_id/security/?sentry_key=key&sentry_environment=production&sentry_release=gitlab-16.4.1-ee.0; script-src 'strict-dynamic' 'self' 'unsafe-inline' 'unsafe-eval' https://www.google.com/recaptcha/ https://www.recaptcha.net https://apis.google.com https://cdn.cookielaw.org https://*.onetrust.com https://cdn.bizible.com/scripts/bizible.js *.googletagmanager.com 'nonce-ppmUKI5kfPfzP8iJHjuUOA=='; style-src 'self' 'unsafe-inline'; worker-src https://example.com/assets/ blob: data:; form-action 'self' https: http: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: https: ``` Nginx logs the following error when issue occurs: ``` upstream sent too big header while reading response header from upstream ``` ### Output of checks <!-- If you are reporting a bug on GitLab.com, uncomment below --> <!-- This bug happens on GitLab.com --> <!-- /label ~"reproduced on GitLab.com" --> #### Results of GitLab environment info <!-- Input any relevant GitLab environment information if needed. --> Domains anonymized. Omnibus with the following CSP configuration: <details> <summary>CSP config</summary> ```ruby gitlab_rails['content_security_policy'] = { 'enabled' => true, 'report_only' => false, 'directives' => { 'report_uri' => 'https://example.net/api/project_id/security/?sentry_key=key&sentry_environment=production&sentry_release=gitlab-16.4.1-ee.0', } } ``` </details> <details> <summary>Expand for output related to GitLab environment info</summary> <pre> System information System: Debian 10 Proxy: no Current User: git Using RVM: no Ruby Version: 3.0.6p216 Gem Version: 3.4.19 Bundler Version:2.4.19 Rake Version: 13.0.6 Redis Version: 7.0.13 Sidekiq Version:6.5.7 Go Version: unknown GitLab information Version: 16.4.1-ee Revision: 229bc5f5985 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 13.11 URL: https://example.com HTTP Clone URL: https://example.com/some-group/some-project.git SSH Clone URL: git@example.com:some-group/some-project.git Elasticsearch: yes Geo: yes Geo node: Primary Using LDAP: yes Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 14.28.0 Repository storages: - default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Gitaly - default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket - default Version: 16.4.1 - default Git Version: 2.42.0 </pre> </details> #### Results of GitLab application Check <!-- Input any relevant GitLab application check information if needed. --> Not relevant. ### Possible fixes <!-- If you can, link to the line of code that might be responsible for the problem. --> Increase proxy buffers in the nginx config, e.g. ``` proxy_busy_buffers_size 512k; proxy_buffers 8 512k; proxy_buffer_size 256k; ```