Search groups

Task 3: Search Groups Using Elasticsearch

Context

Replace the current database query (Api.groups(search)) with Elasticsearch queries to improve performance. This requires creating a query builder, search service, and updating the API/frontend.

Implementation Details

1. Create Group Query Builder

File: ee/lib/search/elastic/group_query_builder.rb

Follow pattern from work_item_query_builder.rb:

module Search
  module Elastic
    class GroupQueryBuilder
      def initialize(search_term, current_user, params = {})
        @search_term = search_term
        @current_user = current_user
        @params = params
      end

      def build
        {
          query: query_hash,
          sort: sort_order,
          size: @params[:per_page] || 20
        }
      end

      private

      def query_hash
        {
          bool: {
            must: must_clauses,
            filter: filter_clauses
          }
        }
      end

      def must_clauses
        return [] if @search_term.blank?

        [{
          multi_match: {
            query: @search_term,
            fields: ['name^3', 'full_name^2', 'path^2', 'description'],
            type: 'best_fields',
            fuzziness: 'AUTO'
          }
        }]
      end

      def filter_clauses
        [
          visibility_filter,
          traversal_filter
        ].compact
      end

      def visibility_filter
        # Filter based on user permissions
        # Public groups: visibility_level = 20
        # Internal groups: visibility_level = 10 (if user logged in)
        # Private groups: only if user has access
      end

      def traversal_filter
        # Filter by parent if specified
        return unless @params[:parent_id]
        { term: { parent_id: @params[:parent_id] } }
      end

      def sort_order
        case @params[:order_by]
        when 'name'
          [{ 'name.keyword' => { order: 'asc' } }]
        when 'similarity'
          ['_score']
        else
          [{ updated_at: { order: 'desc' } }]
        end
      end
    end
  end
end

2. Create Group Search Results Class

File: ee/lib/search/elastic/group_search_results.rb

module Search
  module Elastic
    class GroupSearchResults
      attr_reader :current_user, :query, :params

      def initialize(current_user, query, params = {})
        @current_user = current_user
        @query = query
        @params = params
      end

      def objects(scope = nil)
        return Group.none unless use_elasticsearch?

        # Build ES query
        es_query = GroupQueryBuilder.new(query, current_user, params).build
        
        # Execute search
        response = client.search(
          index: index_name,
          body: es_query
        )

        # Hydrate results from database
        group_ids = response.dig('hits', 'hits')&.map { |hit| hit['_source']['id'] } || []
        Group.id_in(group_ids).preload(:route)
      end

      def formatted_count
        objects.count
      end

      private

      def use_elasticsearch?
        ::Feature.enabled?(:elasticsearch_group_search, current_user) &&
          ::Gitlab::CurrentSettings.elasticsearch_search?
      end

      def index_name
        Search::Elastic::References::Group.index
      end

      def client
        Group.__elasticsearch__.client
      end
    end
  end
end

3. Update Groups API

File: lib/api/groups.rb (or create ee/lib/api/ee/groups.rb override)

Modify the groups search endpoint:

get do
  if use_elasticsearch_for_groups?
    results = ::Search::Elastic::GroupSearchResults.new(
      current_user,
      params[:search],
      params
    ).objects
    
    present paginate(results), with: Entities::Group
  else
    # Existing database search
    groups = GroupsFinder.new(current_user, params).execute
    present paginate(groups), with: Entities::Group
  end
end

helpers do
  def use_elasticsearch_for_groups?
    ::Feature.enabled?(:elasticsearch_group_search, current_user) &&
      ::Gitlab::CurrentSettings.elasticsearch_search?
  end
end

4. Update Frontend API Client

File: app/assets/javascripts/api/groups_api.js

The existing Api.groups() call should work without changes if the backend API is updated. The frontend just calls the API endpoint.

5. Add Feature Flag

File: config/feature_flags/development/elasticsearch_group_search.yml

---
name: elasticsearch_group_search
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/XXXXXX
rollout_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/XXXXXX
milestone: 'X.Y'
type: development
group: group::global search
default_enabled: false

6. Visibility and Permissions

The query builder must respect:

  • Public groups (visibility_level: 20) - visible to everyone
  • Internal groups (visibility_level: 10) - visible to logged-in users
  • Private groups (visibility_level: 0) - visible only to members
  • Use traversal_ids for hierarchy-based filtering

Files to Create/Modify

  • ee/lib/search/elastic/group_query_builder.rb (new)
  • ee/lib/search/elastic/group_search_results.rb (new)
  • lib/api/groups.rb or ee/lib/api/ee/groups.rb (modify)
  • config/feature_flags/development/elasticsearch_group_search.yml (new)

Testing

  • ee/spec/lib/search/elastic/group_query_builder_spec.rb
    • Test text search on name, path, description
    • Test visibility filtering
    • Test parent/hierarchy filtering
    • Test sort orders
  • ee/spec/lib/search/elastic/group_search_results_spec.rb
    • Test result hydration
    • Test empty results
    • Test permission filtering
  • ee/spec/requests/api/groups_spec.rb (modify)
    • Test ES-powered search
    • Test fallback to database search
    • Test feature flag toggle

Validation Steps

  1. Enable feature flag:

    Feature.enable(:elasticsearch_group_search)

  2. Test search via console:

    results = Search::Elastic::GroupSearchResults.new(
  User.first,
  'gitlab',
  {}
).objects

  3. Test via API:

    curl -H "Authorization: Bearer TOKEN" \
  "http://localhost:3000/api/v4/groups?search=gitlab"

  4. Verify ES query:

    # Enable ES query logging
curl -s "localhost:9200/gitlab-development-groups/_search?pretty" \
  -d '{ "query": { "multi_match": { "query": "gitlab", "fields": ["name^3", "full_name^2"] }}}'

Performance Testing

  • Benchmark database query vs ES query
  • Test with 10k, 100k, 1M groups
  • Measure response time improvement
  • Test concurrent searches

Acceptance Criteria

  • Group query builder constructs proper ES queries
  • Search results class executes ES searches and hydrates AR objects
  • API endpoint uses ES when feature flag enabled
  • Search respects visibility levels and user permissions
  • Feature flag controls ES vs DB search
  • Fuzzy matching works correctly
  • Sort options (similarity, name, date) work
  • Performance improvement measured and documented
  • Fallback to database search when ES unavailable
Edited by 🤖 GitLab Bot 🤖