Skip to content

GitLab stalls indefinitely on requests in Chrome with many tabs open, recovers if tabs are closed

A customer found the following issue. They're on GitLab 16.11.10, but that seems irrelevant.

Summary of problem

Users experience periodic stalls in loading pages with GitLab. This can happen on new tabs or mid-session and seems to affect users with many open tabs.

Requests from Chrome show as Pending and the browser displays a spinner. If some of the tabs are closed then eventually some of the other tabs recover.

Customer investigation

We believe this occurs because:

Chrome is now using HTTP/2 for websocket communication with Gitlab Merge Request pages (such as group/project/-/merge_requests/4) – so far, we have only found Merge Request pages to use websockets. Every Merge Request page will have one websocket connection to Gitlab. While the Merge Request page is open, Gitlab automatically sends a ping to the browser via each websocket every 3 seconds to keep the websocket alive. Since the websocket now runs over HTTP/2, they keep Chrome's HTTP/2 sessions alive as well. However, Chrome only supports up to 6 TCP connections per domain (e.g. gitlab.example.com) before it prevents new sessions from being created. When the Chrome browser has fully consumed all 6 sessions for that domain, it would cause the Chrome browser to stall. Previously, we were using HTTP/1.1 websockets, which did not count towards that limit; by moving to HTTP/2 the websockets are run over HTTP/2 connections which do.

The switch to websockets was enabled by a change to default settings during an HA Proxy upgrade.

It is not clear at this point what are the conditions under which Chrome would create a new HTTP/2 session, but one of the conditions is likely when the maximum of 100 streams have been utilized over a single HTTP/2 session.

Reproduction

Chrome has a timer that runs every 30 minutes that forces a re-verificaton of the HTTP certificate, if you make a request to the same site (e.g. gitlab.example.com) after the timer has kicked in. This verification will always initiate a new HTTP/2 session.

Once this HTTP/2 session is created, HTTP requests to gitlab.example.com will start making use of this HTTP/2 session. If you did not load a page with websockets, the HTTP2 session will just expire after all the HTTP connections are completed. However for pages with websockets, that websocket connection will keep that HTTP/2 connection alive (via the Gitlab 3 second pings). This leads to HTTP/2 sessions beginning to accumulate.

So you can easily reproduce this (after about 3 hours) by running a scheduled task on Windows Task scheduler that opens an MR page every 30 minutes, e.g.:

"c:\Program Files (x86)\Google\Chrome\Application\chrome.exe" https://..../-/merge_requests/40

Workaround

We are temporarily forcing the browser back to HTTP 1.1 via our HA Proxy configuration.

https://docs.haproxy.org/2.8/configuration.html#3.1-h2-workaround-bogus-websocket-clients

We note that you are planning to make more use of websockets in the future.

https://about.gitlab.com/blog/how-we-supercharged-gitlab-ci-statuses-with-websockets/

Please pass this on to the engineering team in case there's any preventative action that can be taken in the client code.

Relevant links

Edited by 🤖 GitLab Bot 🤖