Fix N+1 queries in Jira Development Panel API endpoint

What does this MR do?

There were three sources of N+1 queries here: the license check, the path and namespace information, and the root namespace.

The license check was the worst. We were checking the license information on each project individually. This meant we couldn't paginate in SQL, but did so in Ruby, so it was not only an N+1, it was loading too many records.

To fix this, we use the fact that this API endpoint can only return projects in a particular namespace. License checks end up at one of two places: for most instances, it's the instance's license itself. For GitLab.com, where individual namespaces have their own plan, it's the root namespace (subgroups can't have plans; they inherit their plan from the root).

This means that we only ever need a single check. If it passes, every project returned has the feature available. If it fails, we return a 404, like the other endpoints here. That way we can paginate in SQL, as we should.

The path and namespace information N+1 was simple to fix: just preload that information.

The final N+1 was on the root namespace, which we return as the owner field for compatibility with GitHub. Again, this was always the same for all items in the response, but we can't preload it easily because different projects will be at different levels of the hierarchy. Instead, we just calculate the root namespace once, and pass that as an option to the entity. The entity uses that value if it's given, and falls back to calculating it if it's not (in case this entity is used elsewhere without that option).

For #29741 (closed).

Screenshots

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• Separation of EE specific content

Performance and Testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.

changed milestone to %12.4

added GitLab.com Priority Mechanical Sympathy backend bugperformance customer devopsplan gitlab.com groupproject management infradev infrastructure priority3 production request severity3 + 1 deleted label

I wasn't able to test this fully in a console as my usual trick, copying and pasting the new class definition, doesn't appear to work for Grape endpoints. I did some validation, though:

1. I checked the current results in the profiler. To do so, I had to override the user-agent check we do: class Gitlab::Jira::Middleware; def self.jira_dvcs_connector?(*); true; end; end

2. I also sanity-checked a few examples to validate that we always have all licensed projects in the response, or all non-licensed projects. Here user_from_kibana and path_from_kibana are from https://log.gitlab.net/app/kibana#/doc/AWOSvARQwig0Nc2UGcr2/pubsub-rails-inf-gprd-2019.10.08-000014/doc?id=AW2ryOS_qojRxKGhNrn2&_g=h@1f9dccc

   [ gprd ] production> def check(username, path)
   [ gprd ] production> projects = Project.public_or_visible_to_user(User.find_by_username(username)).in_namespace(Group.find_by_full_path(path).self_and_descendants)
   [ gprd ] production> [projects.count, projects.select { |project| project.feature_available?(:jira_dev_panel_integration) }.count]
   [ gprd ] production> end
   => :check
   [ gprd ] production> check('smcgivern', 'issue-reproduce')
   => [1566, 1566]
   [ gprd ] production> check('smcgivern', 'gitlab-org')
   => [606, 606]
   [ gprd ] production> check('smcgivern', 'gitlab-com')
   => [1190, 1190]
   [ gprd ] production> check('smcgivern', 'gitlab-com/people-group')
   => [14, 14]
   [ gprd ] production> check('smcgivern', 'gl-technical-interviews/backend')
   => [499, 0]
   [ gprd ] production> check('smcgivern', 'gl-retrospectives')
   => [22, 22]
   [ gprd ] production> check(user_from_kibana, path_from_kibana)
   => [1824, 1824]

mentioned in issue #29741 (closed)

@engwan as you have the context here, and you're a trainee maintainer, could you review this?

assigned to @engwan and unassigned @smcgivern

	1 Warning
	6625a6d3: This commit’s subject line is acceptable, but please try to reduce it to 50 characters.

Reviewer roulette

Changes that require review have been detected! A merge request is normally reviewed by both a reviewer and a maintainer in its primary category (e.g. frontend or backend), and by a maintainer in all other categories.

To spread load more evenly across eligible reviewers, Danger has randomly picked a candidate for each review slot. Feel free to override this selection if you think someone else would be better-suited, or the chosen person is unavailable.

Once you've decided who will review this merge request, mention them as you normally would! Danger does not (yet?) automatically notify them for you.

Category	Reviewer	Maintainer
backend	Alper Akgun (@a_akgun)	Rémy Coutable (@rymai)

Generated by Danger

added typefeature label

Thanks @smcgivern, looks good!

Do you think we need to add a spec for the unlicensed case for this endpoint? I see that we don't have existing specs for this and we changed the behavior a bit here by now returning a 404 instead of an empty list.

assigned to @smcgivern and unassigned @engwan

mentioned in issue #33741 (closed)

Thanks @engwan! Could you take another look please?

assigned to @engwan and unassigned @smcgivern

added 1 commit

• a6ec2bc4 - Fix N+1 queries in Jira Development Panel API endpoint

Compare with previous version

approved this merge request