Investigate why Projects List API shows notably higher Rails memory usage under load
After a recent review of Ruby 2.6 to 2.7 memory usage of GitLab under load via our performance tests, a few endpoints stood out as having notably higher Rails memory usage.
Comparing between GitLab 13.5.0
and the latest nightly (13.9.0-pre 16bea7edbd9
) found that the Projects List API under load surprisingly is now consuming over 1 GB more memory:
13.5.0
:
13.9.0-pre 16bea7edbd9
:
The memory increase was first detected in a larger review of if Ruby 2.7 lead to increased memory usage comparing 13.5.0
and 13.8.0-pre 852ea7c0283
(full results here) and further tests were run today to confirm if that memory increase has maintained. Testing was done using our standard performance testing setup in lab conditions with GPT running against a 10k Reference Architecture at 200 RPS.
It should also be noted that for testing the Project's endpoint we have 2500 projects in our test environment and we test a mix of both forms of pagination (keyset and offset). Additionally this endpoint is known problem endpoint and has a general performance issue raised against it.
As shown above the endpoint today pushes each of the 3 GitLab Rails nodes to use around 13.5 GB of ram each where as back on 13.5.0
it only consumed around around 12 GB each. This is a notable increase of around 1.5 GB of Rails ram usage. As such, we thought it worth raising an issue to be investigated why this is the case. This is all the data we have to hand at the moment but access can be granted to our test environments to investigate further as required.