Skip to content

[RUN AS-IF-FOSS] Fix N+1 search scope=issues

Dmitry Gruzd requested to merge fix-search-issues-n-plus-1 into master

What does this MR do?

This MR fixes N+1 issues found in /search scope=issues

Bullet output

Non AGS search only

USE eager loading detected: Issue => [:project] Add to your finder: :includes => [:project]
  Issue => [:project]
  Add to your finder: :includes => [:project]
Call stack
  /home/dgruzd/projects/gdk-ee/gitlab/app/policies/issuable_policy.rb:4:in `block in <class:IssuablePolicy>'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/declarative_policy/base.rb:333:in `instance_eval'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/declarative_policy/base.rb:333:in `block in delegated_policies'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/declarative_policy/base.rb:332:in `transform_values'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/declarative_policy/base.rb:332:in `delegated_policies'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/declarative_policy/base.rb:289:in `runner'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/declarative_policy/base.rb:234:in `block in allowed?'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/declarative_policy/base.rb:234:in `all?'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/declarative_policy/base.rb:234:in `allowed?'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/declarative_policy/base.rb:226:in `can?'
  /home/dgruzd/projects/gdk-ee/gitlab/app/models/ability.rb:72:in `allowed?'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/metrics/instrumentation.rb:161:in `block in allowed?'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/metrics/method_call.rb:36:in `measure'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/metrics/instrumentation.rb:161:in `allowed?'
  /home/dgruzd/projects/gdk-ee/gitlab/app/services/search_service.rb:81:in `visible_result?'
  /home/dgruzd/projects/gdk-ee/gitlab/app/services/search_service.rb:85:in `block in redact_unauthorized_results'
  /home/dgruzd/projects/gdk-ee/gitlab/app/services/search_service.rb:85:in `reject'
  /home/dgruzd/projects/gdk-ee/gitlab/app/services/search_service.rb:85:in `redact_unauthorized_results'
  /home/dgruzd/projects/gdk-ee/gitlab/app/services/search_service.rb:65:in `search_objects'
  /home/dgruzd/projects/gdk-ee/gitlab/app/controllers/search_controller.rb:35:in `show'
  /home/dgruzd/projects/gdk-ee/gitlab/app/controllers/application_controller.rb:546:in `block in allow_gitaly_ref_name_caching'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/gitaly_client.rb:325:in `allow_ref_name_caching'
  /home/dgruzd/projects/gdk-ee/gitlab/app/controllers/application_controller.rb:545:in `allow_gitaly_ref_name_caching'
  /home/dgruzd/projects/gdk-ee/gitlab/ee/lib/gitlab/ip_address_state.rb:10:in `with'
  /home/dgruzd/projects/gdk-ee/gitlab/ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'
  /home/dgruzd/projects/gdk-ee/gitlab/app/controllers/application_controller.rb:491:in `set_current_admin'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/session.rb:11:in `with_session'
  /home/dgruzd/projects/gdk-ee/gitlab/app/controllers/application_controller.rb:482:in `set_session_storage'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/i18n.rb:55:in `with_locale'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/i18n.rb:61:in `with_user_locale'
  /home/dgruzd/projects/gdk-ee/gitlab/app/controllers/application_controller.rb:476:in `set_locale'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/error_tracking.rb:50:in `with_context'
  /home/dgruzd/projects/gdk-ee/gitlab/app/controllers/application_controller.rb:541:in `sentry_context'
  /home/dgruzd/projects/gdk-ee/gitlab/app/controllers/application_controller.rb:469:in `block in set_current_context'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/application_context.rb:52:in `block in use'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/application_context.rb:52:in `use'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/application_context.rb:20:in `with_context'
  /home/dgruzd/projects/gdk-ee/gitlab/app/controllers/application_controller.rb:462:in `set_current_context'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/metrics/elasticsearch_rack_middleware.rb:24:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/metrics/redis_rack_middleware.rb:22:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/rails_queue_duration.rb:29:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/metrics/transaction.rb:54:in `run'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/metrics/rack_middleware.rb:17:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/request_profiler/middleware.rb:17:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/query_limiting/middleware.rb:17:in `block in call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/query_limiting/transaction.rb:39:in `run'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/query_limiting/middleware.rb:16:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/ee/lib/gitlab/database/load_balancing/rack_middleware.rb:39:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/ee/lib/gitlab/jira/middleware.rb:19:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/go.rb:20:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/etag_caching/middleware.rb:13:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/multipart.rb:140:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/read_only/controller.rb:51:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/read_only.rb:18:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/basic_health_check.rb:25:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/request_context.rb:23:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/config/initializers/fix_local_cache_middleware.rb:9:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/static.rb:11:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/webpack/dev_server_middleware.rb:27:in `perform_request'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/metrics/requests_rack_middleware.rb:60:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/release_env.rb:12:in `call'

AGS and non AGS searches

USE eager loading detected: Issue => [:author] Add to your finder: :includes => [:author]
  Issue => [:author]
  Add to your finder: :includes => [:author]
Call stack
  /home/dgruzd/projects/gdk-ee/gitlab/app/models/concerns/cache_markdown_field.rb:38:in `banzai_render_context'
  /home/dgruzd/projects/gdk-ee/gitlab/app/models/issue.rb:359:in `banzai_render_context'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/metrics/instrumentation.rb:161:in `block in banzai_render_context'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/metrics/method_call.rb:36:in `measure'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/metrics/instrumentation.rb:161:in `banzai_render_context'
  /home/dgruzd/projects/gdk-ee/gitlab/app/helpers/markup_helper.rb:291:in `markdown_field_render_context'
  /home/dgruzd/projects/gdk-ee/gitlab/app/helpers/markup_helper.rb:285:in `render_markdown_field'
  /home/dgruzd/projects/gdk-ee/gitlab/app/helpers/markup_helper.rb:116:in `markdown_field'
  /home/dgruzd/projects/gdk-ee/gitlab/app/helpers/search_helper.rb:253:in `search_md_sanitize'
  /home/dgruzd/projects/gdk-ee/gitlab/app/views/search/results/_issue.html.haml:11:in `_app_views_search_results__issue_html_haml___482972679664846455_70257513216540'
  /home/dgruzd/projects/gdk-ee/gitlab/app/views/search/_results.html.haml:35:in `_app_views_search__results_html_haml___1308391796588172837_70257506706740'
  /home/dgruzd/projects/gdk-ee/gitlab/app/views/search/show.html.haml:14:in `_app_views_search_show_html_haml___261604582242372378_70257509698360'
  /home/dgruzd/projects/gdk-ee/gitlab/app/controllers/application_controller.rb:134:in `render'
  /home/dgruzd/projects/gdk-ee/gitlab/app/controllers/application_controller.rb:546:in `block in allow_gitaly_ref_name_caching'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/gitaly_client.rb:325:in `allow_ref_name_caching'
  /home/dgruzd/projects/gdk-ee/gitlab/app/controllers/application_controller.rb:545:in `allow_gitaly_ref_name_caching'
  /home/dgruzd/projects/gdk-ee/gitlab/ee/lib/gitlab/ip_address_state.rb:10:in `with'
  /home/dgruzd/projects/gdk-ee/gitlab/ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'
  /home/dgruzd/projects/gdk-ee/gitlab/app/controllers/application_controller.rb:491:in `set_current_admin'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/session.rb:11:in `with_session'
  /home/dgruzd/projects/gdk-ee/gitlab/app/controllers/application_controller.rb:482:in `set_session_storage'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/i18n.rb:55:in `with_locale'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/i18n.rb:61:in `with_user_locale'
  /home/dgruzd/projects/gdk-ee/gitlab/app/controllers/application_controller.rb:476:in `set_locale'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/error_tracking.rb:50:in `with_context'
  /home/dgruzd/projects/gdk-ee/gitlab/app/controllers/application_controller.rb:541:in `sentry_context'
  /home/dgruzd/projects/gdk-ee/gitlab/app/controllers/application_controller.rb:469:in `block in set_current_context'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/application_context.rb:52:in `block in use'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/application_context.rb:52:in `use'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/application_context.rb:20:in `with_context'
  /home/dgruzd/projects/gdk-ee/gitlab/app/controllers/application_controller.rb:462:in `set_current_context'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/metrics/elasticsearch_rack_middleware.rb:24:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/metrics/redis_rack_middleware.rb:22:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/rails_queue_duration.rb:29:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/metrics/transaction.rb:54:in `run'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/metrics/rack_middleware.rb:17:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/request_profiler/middleware.rb:17:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/query_limiting/middleware.rb:17:in `block in call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/query_limiting/transaction.rb:39:in `run'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/query_limiting/middleware.rb:16:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/ee/lib/gitlab/database/load_balancing/rack_middleware.rb:39:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/ee/lib/gitlab/jira/middleware.rb:19:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/go.rb:20:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/etag_caching/middleware.rb:13:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/multipart.rb:140:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/read_only/controller.rb:51:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/read_only.rb:18:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/basic_health_check.rb:25:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/request_context.rb:23:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/config/initializers/fix_local_cache_middleware.rb:9:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/static.rb:11:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/webpack/dev_server_middleware.rb:27:in `perform_request'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/metrics/requests_rack_middleware.rb:60:in `call'
  /home/dgruzd/projects/gdk-ee/gitlab/lib/gitlab/middleware/release_env.rb:12:in `call'

Databaselab

Before

author calls explain
SELECT "users".* FROM "users" WHERE "users"."id" = 1625112
project calls explain
SELECT "projects".* FROM "projects" WHERE "projects"."id" = 278964

After

author calls explain
SELECT users.* FROM users WHERE users.id IN (6049551, 1625112, 2672834)
project calls explain
SELECT "projects".* FROM "projects" WHERE "projects"."id" IN (7764, 14978605, 6282913, 13083, 278964)

Screenshots

Performance bar and find were used to see how many calls were made for the page load.

Before

Author calls (22 calls to SELECT "users.*" FROM "users"...) image
Project calls (21 calls to SELECT "projects.*" FROM "projects"...) image

After

Author calls (3 calls to SELECT "users.*" FROM "users"...) image
Project calls (2 calls to SELECT "projects.*" FROM "projects"...) image

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team
Edited by Terri Chu

Merge request reports