A [customer](https://gitlab.zendesk.com/agent/tickets/663065) found the following issue. They're on GitLab 16.11.10, but that seems irrelevant. #### Summary of problem Users experience periodic stalls in loading pages with GitLab. This can happen on new tabs or mid-session and seems to affect users with many open tabs. Requests from Chrome show as Pending and the browser displays a spinner. If some of the tabs are closed then eventually some of the other tabs recover. #### Customer investigation > We believe this occurs because: > > Chrome is now using HTTP/2 for websocket communication with Gitlab Merge Request pages (such as group/project/-/merge_requests/4) – so far, we have only found Merge Request pages to use websockets. > Every Merge Request page will have one websocket connection to Gitlab. > While the Merge Request page is open, Gitlab automatically sends a ping to the browser via each websocket every 3 seconds to keep the websocket alive. > Since the websocket now runs over HTTP/2, they keep Chrome's HTTP/2 sessions alive as well. > However, Chrome only supports up to 6 TCP connections per domain (e.g. gitlab.example.com) before it prevents new sessions from being created. When the Chrome browser has fully consumed all 6 sessions for that domain, it would cause the Chrome browser to stall. Previously, we were using HTTP/1.1 websockets, which did not count towards that limit; by moving to HTTP/2 the websockets are run over HTTP/2 connections which do. > > The switch to websockets was enabled by a change to default settings during an HA Proxy upgrade. > > It is not clear at this point what are the conditions under which Chrome would create a new HTTP/2 session, but one of the conditions is likely when the maximum of 100 streams have been utilized over a single HTTP/2 session. #### Reproduction > Chrome has a timer that runs every 30 minutes that forces a re-verificaton of the HTTP certificate, if you make a request to the same site (e.g. gitlab.example.com) after the timer has kicked in. This verification will always initiate a new HTTP/2 session. > > Once this HTTP/2 session is created, HTTP requests to gitlab.example.com will start making use of this HTTP/2 session. If you did not load a page with websockets, the HTTP2 session will just expire after all the HTTP connections are completed. However for pages with websockets, that websocket connection will keep that HTTP/2 connection alive (via the Gitlab 3 second pings). This leads to HTTP/2 sessions beginning to accumulate. > > So you can easily reproduce this (after about 3 hours) by running a scheduled task on Windows Task scheduler that opens an MR page every 30 minutes, e.g.: > > "c:\Program Files (x86)\Google\Chrome\Application\chrome.exe" https://..../-/merge_requests/40 #### Workaround > We are temporarily forcing the browser back to HTTP 1.1 via our HA Proxy configuration. > > https://docs.haproxy.org/2.8/configuration.html#3.1-h2-workaround-bogus-websocket-clients > > We note that you are planning to make more use of websockets in the future. > > https://about.gitlab.com/blog/how-we-supercharged-gitlab-ci-statuses-with-websockets/ > > Please pass this on to the engineering team in case there's any preventative action that can be taken in the client code. #### Relevant links - [Customer ticket](https://gitlab.zendesk.com/agent/tickets/663065) (internal link) - [Issue in Chrome](https://issues.chromium.org/issues/405934874)