Improve performance for elasticsearch limited project/namespace checks

Summary

Incident: gitlab-com/gl-infra/production#7175 (closed)

We're seeing degraded performance with the queries which check for Elasticsearch limited namespaces and projects (#359043 (closed))

The method elasticsearch_limited_namespaces currently checks all namespaces which are also indexed. This is timing out in QA tests. The query returns all namespaces and descendants.

The QA namespace gitlab-qa-sandbox-group-2 has 867 descendant namespaces.

Example Project

https://log.gprd.gitlab.net/app/discover#/?_a=h@d719674&_g=h@431af32

exception backtrack

lib/gitlab/database/load_balancing/connection_proxy.rb:100:in `block in read_using_load_balancer', lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write', lib/gitlab/database/load_balancing/load_balancer.rb:179:in `retry_with_backoff', lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write', lib/gitlab/database/load_balancing/connection_proxy.rb:99:in `read_using_load_balancer', lib/gitlab/database/load_balancing/connection_proxy.rb:54:in `block (2 levels) in <class:ConnectionProxy>', ee/app/models/ee/application_setting.rb:217:in `block in elasticsearch_indexes_namespace?', ee/lib/gitlab/elastic/elasticsearch_enabled_cache.rb:35:in `block in fetch', lib/gitlab/redis/wrapper.rb:23:in `block in with', lib/gitlab/redis/wrapper.rb:23:in `with', ee/lib/gitlab/elastic/elasticsearch_enabled_cache.rb:29:in `fetch', ee/app/models/ee/application_setting.rb:216:in `elasticsearch_indexes_namespace?', lib/gitlab/current_settings.rb:32:in `method_missing', ee/app/models/ee/namespace.rb:430:in `use_elasticsearch?', ee/app/services/ee/projects/transfer_service.rb:41:in `update_elasticsearch_hooks', ee/app/services/ee/projects/transfer_service.rb:34:in `post_update_hooks', app/services/projects/transfer_service.rb:113:in `proceed_to_transfer', app/services/projects/transfer_service.rb:70:in `transfer', app/services/projects/transfer_service.rb:35:in `execute', app/controllers/projects_controller.rb:130:in `transfer', ee/lib/gitlab/ip_address_state.rb:10:in `with', ee/app/controllers/ee/application_controller.rb:45:in `set_current_ip_address', app/controllers/application_controller.rb:527:in `set_current_admin', lib/gitlab/session.rb:11:in `with_session', app/controllers/application_controller.rb:518:in `set_session_storage', lib/gitlab/i18n.rb:105:in `with_locale', lib/gitlab/i18n.rb:111:in `with_user_locale', app/controllers/application_controller.rb:512:in `set_locale', app/controllers/application_controller.rb:506:in `set_current_context', ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase', lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call', lib/gitlab/middleware/memory_report.rb:13:in `call', lib/gitlab/middleware/speedscope.rb:13:in `call', lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call', lib/gitlab/middleware/rails_queue_duration.rb:33:in `call', lib/gitlab/metrics/rack_middleware.rb:16:in `block in call', lib/gitlab/metrics/web_transaction.rb:46:in `run', lib/gitlab/metrics/rack_middleware.rb:16:in `call', lib/gitlab/jira/middleware.rb:19:in `call', lib/gitlab/middleware/go.rb:20:in `call', lib/gitlab/etag_caching/middleware.rb:21:in `call', lib/gitlab/middleware/query_analyzer.rb:11:in `block in call', lib/gitlab/database/query_analyzer.rb:37:in `within', lib/gitlab/middleware/query_analyzer.rb:11:in `call', lib/gitlab/middleware/multipart.rb:173:in `call', lib/gitlab/middleware/read_only/controller.rb:50:in `call', lib/gitlab/middleware/read_only.rb:18:in `call', lib/gitlab/middleware/same_site_cookies.rb:27:in `call', lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call', lib/gitlab/middleware/basic_health_check.rb:25:in `call', lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call', lib/gitlab/middleware/request_context.rb:21:in `call', lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call', config/initializers/fix_local_cache_middleware.rb:11:in `call', lib/gitlab/middleware/compressed_json.rb:26:in `call', lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call', lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call', lib/gitlab/metrics/requests_rack_middleware.rb:77:in `call', lib/gitlab/middleware/release_env.rb:13:in `call'

What is the current bug behavior?

Seen during project transfer, the QA tests timed out when running a transfer of a project which is being transfered to or from an Elasticsearch enabled namespace