Add feature flag to skip related_ids query for large namespaces

Dmitry Gruzdrequested to merge
1866-fix-namespace-search-term-limit into master

What does this MR do and why?

Fixes namespace-wide searches failing when namespaces have more than 65,536 projects.

When searching in very large namespaces, the related_ids_for_notes query generates an Elasticsearch terms query containing all project IDs in the namespace. For namespaces with more than 65k projects, this exceeds Elasticsearch's index.max_terms_count limit, causing all searches (issues, merge requests, notes) to fail.

This MR adds a feature flag search_skip_related_ids that allows skipping the related_ids_for_notes query for affected namespaces. The guard is implemented in the base SearchResults class, so it applies to both group-level and project-level searches. The flag is checked against the root ancestor of the searched namespace, so enabling it for a top-level namespace automatically applies to all subgroups and projects within that hierarchy.

Trade-off: The related_ids_for_notes query boosts issues/MRs that have matching notes in search results. Disabling it means slightly lower search relevancy, but searches will work instead of failing completely.

References

  • Related to ongoing TraversalIDs optimization work for search scopes

How to set up and validate locally

Prerequisites

  1. Ensure Elasticsearch is running in GDK
  2. Have a group with indexed projects

Testing the fix

  1. Enable the feature flag for a group:

    # In rails console
group = Group.find_by_full_path('your-group')
Feature.enable(:search_skip_related_ids, group)

  2. Perform searches at different levels:

    • Group search: http://localhost:3000/groups/your-group/-/search?search=test&scope=issues
    • Project search: http://localhost:3000/your-group/your-project/-/search?search=test&scope=issues
    • Verify both complete successfully

  3. Verify the query behavior for group searches:

    # In rails console
user = User.first
group = Group.find_by_full_path('your-group')

# With FF disabled (default) - calls Note.elastic_search
Feature.disable(:search_skip_related_ids, group)
results = Gitlab::Elastic::GroupSearchResults.new(user, 'test', [], group: group)
results.send(:scope_options, :merge_requests)[:related_ids]
# => Returns array of IDs (or empty if no matches)

# With FF enabled - skips Note.elastic_search
Feature.enable(:search_skip_related_ids, group)
results = Gitlab::Elastic::GroupSearchResults.new(user, 'test', [], group: group)
results.send(:scope_options, :merge_requests)[:related_ids]
# => Returns []

  4. Verify it works for project searches:

    # Enable for top-level group
group = Group.find_by_full_path('your-group')
Feature.enable(:search_skip_related_ids, group)

# Search in project - flag is inherited from root ancestor
project = Project.find_by_full_path('your-group/your-project')
results = Gitlab::Elastic::ProjectSearchResults.new(user, 'test', project: project)
results.send(:skip_related_ids_for_large_namespace?)
# => true

  5. Verify it works for subgroups:

    # Enable for top-level group
top_group = Group.find_by_full_path('your-group')
Feature.enable(:search_skip_related_ids, top_group)

# Search in subgroup - flag is inherited
subgroup = Group.find_by_full_path('your-group/subgroup')
results = Gitlab::Elastic::GroupSearchResults.new(user, 'test', [], group: subgroup)
results.send(:skip_related_ids_for_large_namespace?)
# => true

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Dmitry Gruzd

Merge request reports