500 on /-/forks/new.json when user is part of thousands of groups

Summary

When a user (such as gitlab-qa) is part of thousands of groups and they try to fork a project, the call to /-/forks/new.json fails with a 500.

Link to Kibana: https://log.gprd.gitlab.net/goto/bd9f5c6f65618ca1b061c02caab4a5dd

Expand for Kibana logs

"ee/lib/gitlab/database/load_balancing/connection_proxy.rb:72:in `block in read_using_load_balancer'", "ee/lib/gitlab/database/load_balancing/load_balancer.rb:32:in `read'", "ee/lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `read_using_load_balancer'", "ee/lib/gitlab/database/load_balancing/connection_proxy.rb:44:in `select_all'", "app/models/group.rb:478:in `max_member_access_for_user'", "lib/gitlab/metrics/instrumentation.rb:160:in `block in max_member_access_for_user'", "lib/gitlab/metrics/method_call.rb:27:in `measure'", "lib/gitlab/metrics/instrumentation.rb:160:in `max_member_access_for_user'", "app/policies/group_policy.rb:228:in `lookup_access_level!'", "ee/app/policies/ee/group_policy.rb:317:in `lookup_access_level!'", "app/policies/group_policy.rb:224:in `access_level'", "app/policies/group_policy.rb:18:in `block in '", "lib/declarative_policy/condition.rb:23:in `instance_eval'", "lib/declarative_policy/condition.rb:23:in `compute'", "lib/declarative_policy/condition.rb:44:in `block in pass?'", "lib/declarative_policy/base.rb:303:in `cache'", "lib/declarative_policy/condition.rb:44:in `pass?'", "lib/declarative_policy/rule.rb:81:in `pass?'", "lib/declarative_policy/step.rb:81:in `pass?'", "lib/declarative_policy/runner.rb:92:in `block in run'", "lib/declarative_policy/runner.rb:180:in `block in steps_by_score'", "lib/declarative_policy/runner.rb:149:in `loop'", "lib/declarative_policy/runner.rb:149:in `steps_by_score'", "lib/declarative_policy/runner.rb:82:in `run'", "lib/declarative_policy/runner.rb:60:in `pass?'", "lib/declarative_policy/base.rb:234:in `block in allowed?'", "lib/declarative_policy/base.rb:234:in `all?'", "lib/declarative_policy/base.rb:234:in `allowed?'", "lib/declarative_policy/base.rb:226:in `can?'", "app/models/ability.rb:72:in `allowed?'", "lib/gitlab/metrics/instrumentation.rb:160:in `block in allowed?'", "lib/gitlab/metrics/method_call.rb:27:in `measure'", "lib/gitlab/metrics/instrumentation.rb:160:in `allowed?'", "app/models/user.rb:1091:in `can?'", "app/serializers/fork_namespace_entity.rb:34:in `block in '", "app/serializers/base_serializer.rb:16:in `represent'", "app/controllers/projects/forks_controller.rb:51:in `block (2 levels) in new'", "app/controllers/projects/forks_controller.rb:41:in `new'", "ee/lib/gitlab/ip_address_state.rb:10:in `with'", "ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'", "app/controllers/application_controller.rb:494:in `set_current_admin'", "lib/gitlab/session.rb:11:in `with_session'", "app/controllers/application_controller.rb:485:in `set_session_storage'", "lib/gitlab/i18n.rb:73:in `with_locale'", "lib/gitlab/i18n.rb:79:in `with_user_locale'", "app/controllers/application_controller.rb:479:in `set_locale'", "lib/gitlab/error_tracking.rb:52:in `with_context'", "app/controllers/application_controller.rb:544:in `sentry_context'", "app/controllers/application_controller.rb:472:in `block in set_current_context'", "lib/gitlab/application_context.rb:54:in `block in use'", "lib/gitlab/application_context.rb:54:in `use'", "lib/gitlab/application_context.rb:21:in `with_context'", "app/controllers/application_controller.rb:464:in `set_current_context'", "ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase'", "lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'", "lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'", "lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'", "lib/gitlab/metrics/transaction.rb:56:in `run'", "lib/gitlab/metrics/rack_middleware.rb:16:in `call'", "lib/gitlab/request_profiler/middleware.rb:17:in `call'", "ee/lib/gitlab/database/load_balancing/rack_middleware.rb:39:in `call'", "lib/gitlab/jira/middleware.rb:19:in `call'", "lib/gitlab/middleware/go.rb:20:in `call'", "lib/gitlab/etag_caching/middleware.rb:21:in `call'", "lib/gitlab/middleware/multipart.rb:234:in `call'", "lib/gitlab/middleware/read_only/controller.rb:50:in `call'", "lib/gitlab/middleware/read_only.rb:18:in `call'", "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'", "lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'", "lib/gitlab/middleware/basic_health_check.rb:25:in `call'", "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'", "lib/gitlab/middleware/request_context.rb:23:in `call'", "config/initializers/fix_local_cache_middleware.rb:9:in `call'", "lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'", "lib/gitlab/middleware/release_env.rb:12:in `call'"

Discovered by end-to-end test: qa/specs/features/ee/browser_ui/1_manage/group/prevent_forking_outside_group_spec.rb

Seen on Canary and Production - https://ops.gitlab.net/gitlab-org/quality/canary/-/jobs/2499510 & https://ops.gitlab.net/gitlab-org/quality/production/-/jobs/2499801

E2E test stack trace

Failures:
  1) Manage prevent forking outside group when disabled allows forking outside of group
     Failure/Error: fork_new.search_for_group(group_for_fork.path)
     
     QA::Support::Repeater::WaitExceededError:
       Page did not fully load. This could be due to an unending async request or loading icon.
     # ./qa/support/repeater.rb:44:in `repeat_until'
     # ./qa/support/waiter.rb:23:in `wait_until'
     # ./qa/support/wait_for_requests.rb:11:in `wait_for_requests'
     # ./qa/page/base.rb:107:in `find_element'
     # ./qa/support/page/logging.rb:38:in `find_element'
     # ./qa/page/project/fork/new.rb:21:in `search_for_group'
     # ./qa/specs/features/ee/browser_ui/1_manage/group/prevent_forking_outside_group_spec.rb:65:in `block in visit_project_and_search_group_for_fork'
     # ./qa/scenario/actable.rb:16:in `perform'
     # ./qa/specs/features/ee/browser_ui/1_manage/group/prevent_forking_outside_group_spec.rb:64:in `visit_project_and_search_group_for_fork'
     # ./qa/specs/features/ee/browser_ui/1_manage/group/prevent_forking_outside_group_spec.rb:25:in `block (4 levels) in <module:QA>'

Screenshot / HTML page

Proposal

We are looking for a Rate Limit or Application Limit solution here.