Zoekt: Possible performance regression

Summary

We are observing long search times in Zoekt code search. A simple search query is taking over 1.3 seconds to complete, which is significantly above expected performance thresholds.

Reproduction

Example search URL: https://gitlab.com/search?group_id=9970&scope=blobs&search=ZOEKT_TARGETS_CACHE_EXPIRES_IN+case:yes&project_id=278964

Observed response time: 1304.347ms

Technical Details

Request Details

• Endpoint: POST /webserver/api/v2/search
• Timeout configured: 120s
• Target node: zoekt.gprd.gke.gitlab.net:8443/nodes/gitlab-gitlab-zoekt-35.gitlab-gitlab-zoekt.gitlab.svc.cluster.local

Request Payload

{
  "version": 2,
  "timeout": "120s",
  "num_context_lines": 1,
  "max_file_match_window": 0,
  "max_file_match_results": 5000,
  "max_line_match_window": 5000,
  "max_line_match_results": 0,
  "max_line_match_results_per_file": 5,
  "forward_to": [
    {
      "query": {
        "and": {
          "children": [
            {
              "query_string": {
                "query": "ZOEKT_TARGETS_CACHE_EXPIRES_IN case:yes"
              }
            },
            {
              "meta": {
                "key": "project_id",
                "value": "^(278964)$"
              }
            }
          ]
        }
      },
      "endpoint": "https://zoekt.gprd.gke.gitlab.net:8443/nodes/gitlab-gitlab-zoekt-35..."
    }
  ]
}

Call Stack

The request flows through:

1. GraphqlController#execute - GraphQL query execution
2. Search::Blob::BlobSearchResolver#resolve - Blob search resolver
3. SearchService#search_objects - Search service
4. Search::Zoekt::SearchResults#objects - Zoekt search results
5. Search::Zoekt::SearchResults#zoekt_search_and_wrap - Search wrapper with caching
6. Search::Zoekt::Cache#fetch - Cache layer
7. Gitlab::Search::Zoekt::Client#search - HTTP client to Zoekt
8. Gitlab::Search::Zoekt::Client#add_request_details - Request instrumentation

Relevant Code Paths

• ee/lib/gitlab/search/zoekt/client.rb - Zoekt HTTP client
• ee/lib/search/zoekt/search_results.rb - Search results processing
• ee/lib/search/zoekt/cache.rb - Caching layer

Investigation Areas

1. Network latency - Time spent in HTTP request to Zoekt node
2. Query complexity - The access control branch logic adds complexity to the query
3. Node selection - Load balancer picking potentially overloaded nodes
4. Result processing - Time spent parsing and processing results
5. Cache effectiveness - Is the cache (ZOEKT_TARGETS_CACHE_EXPIRES_IN = 10.minutes) being utilized effectively?

Related Issues

• #585434 (closed) - Investigate and fix GraphQL endpoint slowness in global search (mentions 15s SLI target for Zoekt)

Expected Behavior

Search queries should complete well under the 15-second SLI target, ideally in the low hundreds of milliseconds for simple project-scoped searches.

Environment

• Instance: GitLab.com (production)
• Search scope: Project-level (project_id: 278964)
• Group context: gitlab-org (group_id: 9970)