Investigate why Projects List API shows notably higher Rails memory usage under load

After a recent review of Ruby 2.6 to 2.7 memory usage of GitLab under load via our performance tests, a few endpoints stood out as having notably higher Rails memory usage.

Comparing between GitLab 13.5.0 and the latest nightly (13.9.0-pre 16bea7edbd9) found that the Projects List API under load surprisingly is now consuming over 1 GB more memory:

13.5.0:

13.9.0-pre 16bea7edbd9:

The memory increase was first detected in a larger review of if Ruby 2.7 lead to increased memory usage comparing 13.5.0 and 13.8.0-pre 852ea7c0283 (full results here) and further tests were run today to confirm if that memory increase has maintained. Testing was done using our standard performance testing setup in lab conditions with GPT running against a 10k Reference Architecture at 200 RPS.

It should also be noted that for testing the Project's endpoint we have 2500 projects in our test environment and we test a mix of both forms of pagination (keyset and offset). Additionally this endpoint is known problem endpoint and has a general performance issue raised against it.

As shown above the endpoint today pushes each of the 3 GitLab Rails nodes to use around 13.5 GB of ram each where as back on 13.5.0 it only consumed around around 12 GB each. This is a notable increase of around 1.5 GB of Rails ram usage. As such, we thought it worth raising an issue to be investigated why this is the case. This is all the data we have to hand at the moment but access can be granted to our test environments to investigate further as required.

added Category:Cloud Connector api bugperformance devopsplan groupproject management performance-refinement severity3 typebug labels

marked this issue as related to #321262 (closed)

marked this issue as related to #321271 (closed)

marked this issue as related to #211495 (closed)

mentioned in issue #321262 (closed)

added sectiondev label

mentioned in issue gitlab-org/quality/triage-reports#1987 (closed)

mentioned in issue gitlab-org/quality/triage-reports#2047 (closed)

mentioned in issue gitlab-org/quality/triage-reports#2143 (closed)

mentioned in issue gitlab-org/quality/triage-reports#2267 (closed)

mentioned in issue gitlab-org/quality/triage-reports#2338 (closed)

changed milestone to %14.1

changed milestone to %14.0

changed milestone to %14.1

added [deprecated] Accepting merge requests label

added Quality:performance-issues label

added backend label

@jlear this is marked as %14.1, is that still the intention at this point?

@sean_carroll I think it's unlikely we'll make it to this in %14.1 as we have #255449 (closed) and #326120 (closed) both as severity1. I'll move this to %14.2 for now, cc @gweaver

No problem, thanks for the update @jlear

changed milestone to %14.2

added to epic &6360

marked this issue as related to #334667 (closed)

changed milestone to %14.3

added missed:14.2 label

changed milestone to %14.4

added missed:14.3 label

changed milestone to %14.5

added missed:14.4 label

Moving to ~"group::workspace" due to project

added devopsmanage + 1 deleted label and removed devopsplan groupproject management labels

changed milestone to %14.6

added missed:14.5 label

changed milestone to %14.7

added missed:14.6 label

changed milestone to %14.8

added missed:14.7 label

changed milestone to %Backlog

changed epic to &8164

added organization-error-budget label

@mksionek This looks like a good candidate for %15.5 to continue making progress with Improve performance of Projects List API under ... (&8164). Could you work with the engineers on getting this reviewed and weighted?

This would require a lot of investigation, specifically that this is not an easy task.

@grantyoung can you please confirm is it still a problem?

Hey @mksionek. Looking at today's test results against 15.4.0-pre 569e2476b4f higher memory usage does still look to be occurring:

Obviously it's been a little while since 13.9 so we can't account for any other general memory increases but today there's still an increase from around 12.4GiB up to 14.4GiB while the endpoint is being hit. After the test has completed the memory returns to similar levels.

Wanted to cross post this response I gave to a similar question on the Single Issue Details endpoint - #321262 (comment 1083592124)

TL;DR Perhaps have a look to see if this endpoint is doing anything obvious memory wise? If not, there's maybe a bigger effort required here to review memory across the board.

@lohrc The issue here is unknown and may take some time/effort to discover. I propose we create an issue to investigate the issue first, and then hopefully spin out a second follow up issue to solve whatever that problem may be.

@alexpooley Could we just use this issue for the investigation, since it contains all the details? I have renamed the title to reflect that. We can then spin off follow-up issues based on the results. With that in mind, could you try to give this issue a weight just for the investigation part of the effort?

@lohrc Sure! Weight set to 2.

changed milestone to %15.5

mentioned in issue gitlab-org/manage/general-discussion#17545

added Deliverable label

added missed-deliverable label

changed title from Projects List API shows notably higher Rails memory usage under load to Investigate why Projects List API shows notably higher Rails memory usage under load

set weight to 2

added milestonep1 label

added workflowvalidation backlog label

added workflowrefinement label and removed workflowvalidation backlog label

mentioned in issue gitlab-org/manage/general-discussion#17553

changed milestone to %15.6

added missed:15.5 label

changed milestone to %15.7

added missed:15.6 label

mentioned in epic &7832

removed [deprecated] Accepting merge requests label

added priority2 label

added workflowready for development label and removed workflowrefinement label

changed milestone to %15.8

added missed:15.7 label

assigned to @abdwdd

Since this issue was floated with ~"group::application performance" on Slack, I have some early thoughts on this:

• This is most likely unrelated to the Ruby upgrade. If the runtime were responsible, we would have seen similar degradation in production, where we are much more likely to run into extreme corners of resource utilization. I also do not recall any changes to MRI that would explain this, plus, it appears to only happen for one endpoint. I highly suspect this is due to a change introduced in the application that occurred around the same time.
• In order to investigate, we cannot just look at process RSS (I had left a similar comment elsewhere); this is a very coarse metric and can be misleading, as the factors behind RSS growth are far too many to consider. It merely tells us that there might be an issue, but will never tell us what causes it.
• It would be useful to post here how these measurements were taken. @grantyoung If our team had an executable script to see under which conditions memory growth was observed, we could then drill into metric data or inspect the heap. It could, for example, be interesting to see if or how per-thread allocations and GC stats changed between runs.

@grantyoung I'm moving this to workflowblocked as we don't have an executable script.

We don't have any further insight available. This was raised based on a requested review by the Memory team (now ~"group::application performance") as a high level indicator based on machine memory usage.

Since this was raised the product has gone through multiple releases as well as teams and assignments being changed and I don't have capacity to deep dive into this further. If we're not concerned by this (as well as no problems reported by customers) then maybe it's best to just close? It's maybe best to review memory usage more generally with the upcoming Ruby 3 upgrade to see if we're happy with it.

I'm fine with closing this.

IIUC our team is trying to move away from this bottom-up sort of analysis without also understanding what the user impact is. For example, it is not clear to me in this case how the higher memory usage we started to see for this endpoint affects users. If it translates into higher latencies throughout (this could be due to knock-on effects such as an increase in memory kills due to OOMs), it could be cause for concern and a reason to investigate deeper. If not, then given the small size of our team there are likely more impactful things to work on.

I hope I am correctly reflecting what @rogerwoo and our team have been talking about lately -- we are still in the process of figuring some of these things out since Roger only joined us very recently and we also have a new EM joining (today, in fact! )

Ok agreed yeah, this area does require a wider approach. Will close thanks and apologies for the noise.

Not at all, thanks for investigating!

added workflowblocked label and removed workflowready for development label

changed milestone to %Backlog

closed

added devopsdata stores grouptenant scale sectioncore platform labels and removed devopsmanage sectiondev + 1 deleted label

added devopstenant scale grouporganizations sectioninfrastructure platforms labels and removed devopsdata stores grouptenant scale [DEPRECATED] sectioncore platform labels