Fix N+1 queries in Jira Development Panel API endpoint
What does this MR do?
There were three sources of N+1 queries here: the license check, the path and namespace information, and the root namespace.
The license check was the worst. We were checking the license information on each project individually. This meant we couldn't paginate in SQL, but did so in Ruby, so it was not only an N+1, it was loading too many records.
To fix this, we use the fact that this API endpoint can only return projects in a particular namespace. License checks end up at one of two places: for most instances, it's the instance's license itself. For GitLab.com, where individual namespaces have their own plan, it's the root namespace (subgroups can't have plans; they inherit their plan from the root).
This means that we only ever need a single check. If it passes, every project returned has the feature available. If it fails, we return a 404, like the other endpoints here. That way we can paginate in SQL, as we should.
The path and namespace information N+1 was simple to fix: just preload that information.
The final N+1 was on the root namespace, which we return as the owner
field for compatibility with GitHub. Again, this was always the same for
all items in the response, but we can't preload it easily because
different projects will be at different levels of the hierarchy.
Instead, we just calculate the root namespace once, and pass that as an
option to the entity. The entity uses that value if it's given, and
falls back to calculating it if it's not (in case this entity is used
elsewhere without that option).
For #29741 (closed).
Screenshots
Does this MR meet the acceptance criteria?
Conformity
Performance and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
Merge request reports
Activity
changed milestone to %12.4
I wasn't able to test this fully in a console as my usual trick, copying and pasting the new class definition, doesn't appear to work for Grape endpoints. I did some validation, though:
-
I checked the current results in the profiler. To do so, I had to override the user-agent check we do:
class Gitlab::Jira::Middleware; def self.jira_dvcs_connector?(*); true; end; end
-
I also sanity-checked a few examples to validate that we always have all licensed projects in the response, or all non-licensed projects. Here
user_from_kibana
andpath_from_kibana
are from https://log.gitlab.net/app/kibana#/doc/AWOSvARQwig0Nc2UGcr2/pubsub-rails-inf-gprd-2019.10.08-000014/doc?id=AW2ryOS_qojRxKGhNrn2&_g=h@1f9dccc[ gprd ] production> def check(username, path) [ gprd ] production> projects = Project.public_or_visible_to_user(User.find_by_username(username)).in_namespace(Group.find_by_full_path(path).self_and_descendants) [ gprd ] production> [projects.count, projects.select { |project| project.feature_available?(:jira_dev_panel_integration) }.count] [ gprd ] production> end => :check [ gprd ] production> check('smcgivern', 'issue-reproduce') => [1566, 1566] [ gprd ] production> check('smcgivern', 'gitlab-org') => [606, 606] [ gprd ] production> check('smcgivern', 'gitlab-com') => [1190, 1190] [ gprd ] production> check('smcgivern', 'gitlab-com/people-group') => [14, 14] [ gprd ] production> check('smcgivern', 'gl-technical-interviews/backend') => [499, 0] [ gprd ] production> check('smcgivern', 'gl-retrospectives') => [22, 22] [ gprd ] production> check(user_from_kibana, path_from_kibana) => [1824, 1824]
-
mentioned in issue #29741 (closed)
- Resolved by Thong Kuah
@engwan as you have the context here, and you're a trainee maintainer, could you review this?
assigned to @engwan and unassigned @smcgivern
1 Warning 6625a6d3: This commit’s subject line is acceptable, but please try to reduce it to 50 characters. Reviewer roulette
Changes that require review have been detected! A merge request is normally reviewed by both a reviewer and a maintainer in its primary category (e.g. frontend or backend), and by a maintainer in all other categories.
To spread load more evenly across eligible reviewers, Danger has randomly picked a candidate for each review slot. Feel free to override this selection if you think someone else would be better-suited, or the chosen person is unavailable.
Once you've decided who will review this merge request, mention them as you normally would! Danger does not (yet?) automatically notify them for you.
Category Reviewer Maintainer backend Alper Akgun ( @a_akgun
)Rémy Coutable ( @rymai
)Generated by
DangerEdited by 🤖 GitLab Bot 🤖added typefeature label
- Resolved by Heinrich Lee Yu
- Resolved by Sean McGivern
- Resolved by Thong Kuah
- Resolved by Heinrich Lee Yu
Thanks @smcgivern, looks good!
Do you think we need to add a spec for the unlicensed case for this endpoint? I see that we don't have existing specs for this and we changed the behavior a bit here by now returning a 404 instead of an empty list.
assigned to @smcgivern and unassigned @engwan
- Resolved by Sean McGivern
mentioned in issue #33741 (closed)
Thanks @engwan! Could you take another look please?
assigned to @engwan and unassigned @smcgivern
added 1 commit
- a6ec2bc4 - Fix N+1 queries in Jira Development Panel API endpoint
assigned to @smcgivern and unassigned @engwan
assigned to @tkuah and unassigned @smcgivern
Thanks for the quick response on this @smcgivern
- Resolved by Thong Kuah
License checks end up at one of two places: for most instances, it's the instance's license itself. For GitLab.com, where individual namespaces have their own plan, it's the root namespace (subgroups can't have plans; they inherit their plan from the root).
This is more for my understanding: is this fact documented somewhere ? (https://docs.gitlab.com/ee/development/licensed_feature_availability.html#restricting-features-scoped-by-namespaces-or-projects doesn't seem to mention root and also https://gitlab.com/gitlab-org/gitlab/blob/2b607f0916c880a6c947ebfe4f4ee32f43fba551/ee/app/models/ee/namespace.rb#L234-241 seems to indicate we check plans for the whole hierarchy.
I'm probably missing something :)
- Resolved by Sean McGivern
- Resolved by Thong Kuah
Thanks @smcgivern - I have a question in my mind, if that's OK then this MR is good to go.
assigned to @smcgivern and unassigned @tkuah
added 1 commit
- 6625a6d3 - Fix N+1 queries in Jira Development Panel API endpoint
assigned to @tkuah and unassigned @smcgivern
mentioned in commit 71568a69
Thanks @smcgivern @engwan LGTM
This merge request has been deployed to the GitLab.com environment
gstg
in GitLab auto-deploy version12.4.201910171510-ac38081b602.8a20b2fd697
.A list of all the deployed commits can be found here.
If this message is incorrect, please create an issue in the Release Tools project.added workflowstaging label
This merge request has been deployed to the GitLab.com environment
gprd-cny
in GitLab auto-deploy version12.4.201910171510-ac38081b602.8a20b2fd697
.A list of all the deployed commits can be found here.
If this message is incorrect, please create an issue in the Release Tools project.added workflowcanary label and removed workflowstaging label
This merge request has been deployed to the GitLab.com environment
gprd
in GitLab auto-deploy version12.4.201910171510-ac38081b602.8a20b2fd697
.A list of all the deployed commits can be found here.
If this message is incorrect, please create an issue in the Release Tools project.added workflowproduction label and removed workflowcanary label
mentioned in commit Dalahro/gitlab@4d1a3d31
mentioned in issue #37012 (closed)
mentioned in issue #195432 (closed)
mentioned in issue #196862 (closed)